samba crashed: very critical. [2016/07/30 07:00:47.601811, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:47.607554, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:47.612709, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:47.617773, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:47.703414, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:47.708867, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:47.731963, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:50.494388, 1] ../source3/lib/messages.c:976(mess_parent_dgm_cleanup_done) messaging dgm cleanup job ended with NT_STATUS_NO_MEMORY [2016/07/30 07:00:57.180022, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:58.086243, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:58.093154, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:58.099855, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:58.106786, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:58.188456, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory [2016/07/30 07:00:58.196342, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) ps -auxw ... ...... ...... root 58944 0.0 0.2 699252 18388 ? S 07:49 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 58945 0.0 0.1 693020 15724 ? S 07:49 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 58946 0.0 0.2 699252 18388 ? S 07:49 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 58948 0.0 0.2 699252 18388 ? S 07:50 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 58950 0.0 0.2 699252 18388 ? S 07:50 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 58953 0.0 0.1 693020 15720 ? S 07:50 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 58954 0.0 0.2 699252 18548 ? S 07:50 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59082 0.0 0.2 699252 18648 ? S 07:51 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59084 0.0 0.2 699252 18648 ? S 07:51 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59085 0.0 0.1 693020 15740 ? S 07:51 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59086 0.0 0.2 699252 18648 ? S 07:51 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59087 0.0 0.1 693020 15748 ? S 07:52 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59088 0.0 0.2 699252 18648 ? S 07:52 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59090 0.0 0.2 699252 18648 ? S 07:52 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59091 0.0 0.1 693020 15732 ? S 07:52 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59092 0.0 0.2 699252 18648 ? S 07:52 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59093 0.0 0.2 699252 18648 ? S 07:52 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59095 0.0 0.2 699252 18648 ? S 07:53 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59096 0.0 0.1 693020 15752 ? S 07:53 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59098 0.0 0.1 693020 15732 ? S 07:53 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59099 0.0 0.2 699252 18644 ? S 07:53 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59227 0.0 0.2 699252 18652 ? S 07:54 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59229 0.0 0.2 699252 18652 ? S 07:54 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59230 0.0 0.1 693020 15760 ? S 07:54 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59231 0.0 0.2 699252 18652 ? S 07:54 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59232 0.0 0.2 699252 18652 ? S 07:55 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59233 0.0 0.1 693020 15748 ? S 07:55 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59234 0.0 0.2 699252 18652 ? S 07:55 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59236 0.0 0.1 693020 15736 ? S 07:55 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59238 0.0 0.2 699252 18656 ? S 07:55 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59239 0.0 0.2 699252 18652 ? S 07:55 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59240 0.0 0.2 699252 18656 ? S 07:56 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59241 0.0 0.2 699252 18656 ? S 07:56 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59243 0.0 0.1 693020 15744 ? S 07:56 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59244 0.0 0.2 699252 18652 ? S 07:56 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59245 0.0 0.1 693020 15752 ? S 07:56 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59246 0.0 0.2 699252 18664 ? S 07:56 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59247 0.0 0.2 699252 18664 ? S 07:56 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59374 0.0 0.0 107896 592 ? S 07:57 0:00 sleep 180 root 59376 0.0 0.2 699252 20204 ? S 07:57 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59377 0.0 0.1 693020 16276 ? S 07:57 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59378 0.0 0.2 699252 20216 ? S 07:57 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59380 0.0 0.1 693020 16268 ? S 07:57 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59381 0.0 0.2 699252 20220 ? S 07:58 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59384 0.0 0.2 699252 20200 ? S 07:58 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59385 0.6 0.0 157240 1932 pts/1 R+ 07:58 0:00 ps -auxw root 59386 0.0 0.2 699252 20208 ? S 07:58 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59388 0.0 0.1 693020 16292 ? S 07:58 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59389 0.0 0.2 699252 20196 ? S 07:58 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59390 0.0 0.2 699252 20224 ? S 07:59 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59391 0.0 0.2 699252 20216 ? S 07:59 0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground root 59392 0.0 0.0 107896 620 ? S 07:59 0:00 sleep 60 .. ..... ....
Can you please confirm: smbd -V for each PID of a samba and smbd process: smbcontrol pool-usage <samba pid> Then confirm what all the connections keeping the smbd processes alive are with: netstat -avp Finally, details about your AD domain and any other info on what may be causing this issue. Thanks!
(In reply to Andrew Bartlett from comment #1) [root@ ~]# smbd -V Version 4.5.0rc1 netstat -avp [root@mems ~]# netstat -avp Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 localhost:netbios-ns 0.0.0.0:* LISTEN 906/smbd tcp 0 0 mems:netbios-ns 0.0.0.0:* LISTEN 906/smbd tcp 0 0 localhost:netbios-dgm 0.0.0.0:* LISTEN 906/smbd tcp 0 0 mems:netbios-dgm 0.0.0.0:* LISTEN 906/smbd tcp 0 0 mems:nameserver 0.0.0.0:* LISTEN 900/samba tcp 0 0 localhost:nameserver 0.0.0.0:* LISTEN 900/samba tcp 0 0 localhost:netbios-ssn 0.0.0.0:* LISTEN 906/smbd tcp 0 0 mems:netbios-ssn 0.0.0.0:* LISTEN 906/smbd tcp 0 0 localhost:dyna-access 0.0.0.0:* LISTEN 1058/clamd tcp 0 0 mems:kpasswd 0.0.0.0:* LISTEN 903/samba tcp 0 0 localhost:kpasswd 0.0.0.0:* LISTEN 903/samba tcp 0 0 mems:domain 0.0.0.0:* LISTEN 1223/named tcp 0 0 localhost:domain 0.0.0.0:* LISTEN 1223/named tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN 796/sshd tcp 0 0 0.0.0.0:ipp 0.0.0.0:* LISTEN 791/cupsd tcp 0 0 mems:kerberos 0.0.0.0:* LISTEN 903/samba tcp 0 0 localhost:kerberos 0.0.0.0:* LISTEN 903/samba tcp 0 0 localhost:rndc 0.0.0.0:* LISTEN 1223/named tcp 0 0 mems:ldaps 0.0.0.0:* LISTEN 901/samba tcp 0 0 localhost:ldaps 0.0.0.0:* LISTEN 901/samba tcp 0 0 localhost:microsoft-ds 0.0.0.0:* LISTEN 906/smbd tcp 0 0 mems:microsoft-ds 0.0.0.0:* LISTEN 906/smbd tcp 0 0 mems:1024 0.0.0.0:* LISTEN 898/samba tcp 0 0 localhost:1024 0.0.0.0:* LISTEN 898/samba tcp 0 0 mems:msft-gc 0.0.0.0:* LISTEN 901/samba tcp 0 0 localhost:msft-gc 0.0.0.0:* LISTEN 901/samba tcp 0 0 mems:msft-gc-ssl 0.0.0.0:* LISTEN 901/samba tcp 0 0 mems:ldap 0.0.0.0:* LISTEN 901/samba tcp 0 0 localhost:msft-gc-ssl 0.0.0.0:* LISTEN 901/samba tcp 0 0 localhost:ldap 0.0.0.0:* LISTEN 901/samba tcp 0 0 mems:epmap 0.0.0.0:* LISTEN 898/samba tcp 0 0 localhost:epmap 0.0.0.0:* LISTEN 898/samba tcp 0 0 mems:ldap 10.0.20.15:35386 ESTABLISHED 901/samba tcp 0 0 mems:ssh 10.0.20.108:60984 ESTABLISHED 3604/sshd: root@pts tcp 0 0 mems:microsoft-ds 10.0.20.15:47574 ESTABLISHED 1882/smbd tcp 0 0 mems:microsoft-ds 10.0.20.81:57063 ESTABLISHED 6628/smbd tcp 0 0 mems:microsoft-ds 10.0.20.62:60845 ESTABLISHED 6629/smbd tcp 0 0 mems:1024 10.0.20.15:37172 ESTABLISHED 898/samba tcp 0 0 mems:55552 10.0.20.9:iscsi-target ESTABLISHED 830/iscsid tcp6 0 0 localhost:netbios-ns [::]:* LISTEN 906/smbd tcp6 0 0 localhost:netbios-dgm [::]:* LISTEN 906/smbd tcp6 0 0 localhost:netbios-ssn [::]:* LISTEN 906/smbd tcp6 0 0 [::]:http [::]:* LISTEN 1057/httpd tcp6 0 0 localhost:kpasswd [::]:* LISTEN 903/samba tcp6 0 0 [::]:ftp [::]:* LISTEN 820/vsftpd tcp6 0 0 [::]:ssh [::]:* LISTEN 796/sshd tcp6 0 0 [::]:ipp [::]:* LISTEN 791/cupsd tcp6 0 0 localhost:kerberos [::]:* LISTEN 903/samba tcp6 0 0 [::]:https [::]:* LISTEN 1057/httpd tcp6 0 0 localhost:ldaps [::]:* LISTEN 901/samba tcp6 0 0 localhost:microsoft-ds [::]:* LISTEN 906/smbd tcp6 0 0 localhost:1024 [::]:* LISTEN 898/samba tcp6 0 0 localhost:msft-gc [::]:* LISTEN 901/samba tcp6 0 0 localhost:msft-gc-ssl [::]:* LISTEN 901/samba tcp6 0 0 localhost:ldap [::]:* LISTEN 901/samba tcp6 0 0 localhost:epmap [::]:* LISTEN 898/samba netstat: no support for `AF INET (sctp)' on this system. netstat: no support for `AF INET (sctp)' on this system. udp 0 0 0.0.0.0:55789 0.0.0.0:* 534/rsyslogd udp 0 0 0.0.0.0:59935 0.0.0.0:* 545/avahi-daemon: r udp 0 0 mems:domain 0.0.0.0:* 1223/named udp 0 0 localhost:domain 0.0.0.0:* 1223/named udp 0 0 mems:kerberos 0.0.0.0:* 903/samba udp 0 0 localhost:kerberos 0.0.0.0:* 903/samba udp 0 0 mems:ntp 0.0.0.0:* 5166/ntpd udp 0 0 localhost:ntp 0.0.0.0:* 5166/ntpd udp 0 0 0.0.0.0:ntp 0.0.0.0:* 5166/ntpd udp 0 0 mems:netbios-ns 0.0.0.0:* 899/samba udp 0 0 10.0.20.255:netbios-ns 0.0.0.0:* 899/samba udp 0 0 localhost:netbios-ns 0.0.0.0:* 899/samba udp 0 0 127.255.255.:netbios-ns 0.0.0.0:* 899/samba udp 132736 0 10.0.20.255:netbios-ns 0.0.0.0:* 892/nmbd udp 0 0 mems:netbios-ns 0.0.0.0:* 892/nmbd udp 132736 0 0.0.0.0:netbios-ns 0.0.0.0:* 892/nmbd udp 0 0 mems:netbios-dgm 0.0.0.0:* 899/samba udp 0 0 10.0.20.255:netbios-dgm 0.0.0.0:* 899/samba udp 0 0 localhost:netbios-dgm 0.0.0.0:* 899/samba udp 0 0 127.255.255:netbios-dgm 0.0.0.0:* 899/samba udp 117504 0 10.0.20.255:netbios-dgm 0.0.0.0:* 892/nmbd udp 0 0 mems:netbios-dgm 0.0.0.0:* 892/nmbd udp 119680 0 0.0.0.0:netbios-dgm 0.0.0.0:* 892/nmbd udp 0 0 mems:ldap 0.0.0.0:* 902/samba udp 0 0 localhost:ldap 0.0.0.0:* 902/samba udp 0 0 mems:kpasswd 0.0.0.0:* 903/samba udp 0 0 localhost:kpasswd 0.0.0.0:* 903/samba udp 0 0 0.0.0.0:mdns 0.0.0.0:* 545/avahi-daemon: r udp6 0 0 localhost:kerberos [::]:* 903/samba udp6 0 0 fe80::20c:29ff:fe1d:ntp [::]:* 5166/ntpd udp6 0 0 localhost:ntp [::]:* 5166/ntpd udp6 0 0 [::]:ntp [::]:* 5166/ntpd udp6 0 0 localhost:ldap [::]:* 902/samba udp6 0 0 localhost:kpasswd [::]:* 903/samba Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node PID/Program name Path unix 2 [ ] DGRAM 67753 6288/smbd /usr/local/samba/private/msg.sock/6288 unix 2 [ ] DGRAM 61455 5637/smbd /usr/local/samba/private/msg.sock/5637 unix 2 [ ] DGRAM 60979 5798/smbd /usr/local/samba/private/msg.sock/5798 unix 2 [ ] DGRAM 60781 5641/smbd /usr/local/samba/private/msg.sock/5641 unix 2 [ ] DGRAM 61550 5644/smbd /usr/local/samba/private/msg.sock/5644 unix 2 [ ] DGRAM 15760 892/nmbd /usr/local/samba/private/msg.sock/892 unix 2 [ ACC ] STREAM LISTENING 14836 898/samba /usr/local/samba/var/run/ncalrpc/np/browser unix 2 [ ] DGRAM 14804 898/samba /usr/local/samba/private/msg.sock/898 unix 2 [ ] DGRAM 14805 901/samba /usr/local/samba/private/msg.sock/901 unix 2 [ ] DGRAM 14806 902/samba /usr/local/samba/private/msg.sock/902 unix 2 [ ] DGRAM 14819 903/samba /usr/local/samba/private/msg.sock/903 unix 2 [ ] DGRAM 14252 897/samba /usr/local/samba/private/msg.sock/897 unix 2 [ ACC ] STREAM LISTENING 14837 898/samba /usr/local/samba/var/run/ncalrpc/np/unixinfo unix 2 [ ] DGRAM 14829 905/samba /usr/local/samba/private/msg.sock/905 unix 2 [ ACC ] STREAM LISTENING 14838 898/samba /usr/local/samba/var/run/ncalrpc/np/protected_storage unix 2 [ ] DGRAM 14833 910/samba /usr/local/samba/private/msg.sock/910 unix 2 [ ACC ] STREAM LISTENING 14869 898/samba /usr/local/samba/var/run/ncalrpc/np/epmapper unix 2 [ ] DGRAM 14258 904/samba /usr/local/samba/private/msg.sock/904 unix 2 [ ] DGRAM 15904 908/winbindd /usr/local/samba/private/msg.sock/908 unix 2 [ ] DGRAM 15834 899/samba /usr/local/samba/private/msg.sock/899 unix 2 [ ] DGRAM 16870 900/samba /usr/local/samba/private/msg.sock/900 unix 2 [ ] DGRAM 14259 907/samba /usr/local/samba/private/msg.sock/907 unix 2 [ ] DGRAM 14260 909/samba /usr/local/samba/private/msg.sock/909 unix 2 [ ] DGRAM 15916 906/smbd /usr/local/samba/private/msg.sock/906 unix 2 [ ] DGRAM 14274 917/smbd /usr/local/samba/private/msg.sock/917 unix 2 [ ] DGRAM 14892 918/smbd /usr/local/samba/private/msg.sock/918 unix 2 [ ACC ] STREAM LISTENING 14261 907/samba /usr/local/samba/var/lib/ntp_signd/socket unix 2 [ ACC ] STREAM LISTENING 15826 892/nmbd /usr/local/samba/var/run/nmbd/unexpected unix 2 [ ] DGRAM 15941 919/winbindd /usr/local/samba/private/msg.sock/919 unix 2 [ ] DGRAM 14275 920/smbd /usr/local/samba/private/msg.sock/920 unix 2 [ ] DGRAM 62147 5953/smbd /usr/local/samba/private/msg.sock/5953 unix 2 [ ACC ] STREAM LISTENING 15938 908/winbindd /usr/local/samba/var/locks/winbindd_privileged/pipe unix 2 [ ] DGRAM 17471 1051/winbindd /usr/local/samba/private/msg.sock/1051 unix 2 [ ACC ] STREAM LISTENING 14919 901/samba /usr/local/samba/private/ldap_priv/ldapi unix 2 [ ] DGRAM 9765 1/systemd /run/systemd/shutdownd unix 2 [ ] DGRAM 27424 1882/smbd /usr/local/samba/private/msg.sock/1882 unix 2 [ ] DGRAM 39 1/systemd /run/systemd/notify unix 2 [ ] DGRAM 61553 5645/smbd /usr/local/samba/private/msg.sock/5645 unix 2 [ ] DGRAM 60755 5640/smbd /usr/local/samba/private/msg.sock/5640 unix 2 [ ] DGRAM 60830 5649/smbd /usr/local/samba/private/msg.sock/5649 unix 2 [ ] DGRAM 61524 5643/smbd /usr/local/samba/private/msg.sock/5643 unix 2 [ ] DGRAM 61560 5650/smbd /usr/local/samba/private/msg.sock/5650 unix 2 [ ] DGRAM 61414 6090/smbd /usr/local/samba/private/msg.sock/6090 unix 2 [ ] DGRAM 60949 5780/smbd /usr/local/samba/private/msg.sock/5780 unix 2 [ ] DGRAM 56726 5115/winbindd /usr/local/samba/private/msg.sock/5115 unix 2 [ ] DGRAM 62482 6105/smbd /usr/local/samba/private/msg.sock/6105 unix 2 [ ] DGRAM 60964 5793/smbd /usr/local/samba/private/msg.sock/5793 unix 2 [ ACC ] STREAM LISTENING 14834 898/samba /usr/local/samba/var/run/ncalrpc/np/dnsserver unix 2 [ ] DGRAM 60957 5787/smbd /usr/local/samba/private/msg.sock/5787 unix 2 [ ACC ] STREAM LISTENING 14835 898/samba /usr/local/samba/var/run/ncalrpc/np/ntsvcs unix 2 [ ACC ] STREAM LISTENING 14384 1/systemd /var/run/dbus/system_bus_socket unix 2 [ ] DGRAM 62149 5955/smbd /usr/local/samba/private/msg.sock/5955 unix 2 [ ] DGRAM 60961 5790/smbd /usr/local/samba/private/msg.sock/5790 unix 2 [ ACC ] STREAM LISTENING 14839 898/samba /usr/local/samba/var/run/ncalrpc/np/lsass unix 2 [ ] DGRAM 67806 6310/smbd /usr/local/samba/private/msg.sock/6310 unix 2 [ ACC ] STREAM LISTENING 14840 898/samba /usr/local/samba/var/run/ncalrpc/np/lsarpc unix 2 [ ] DGRAM 60984 5804/smbd /usr/local/samba/private/msg.sock/5804 unix 2 [ ACC ] STREAM LISTENING 14841 898/samba /usr/local/samba/var/run/ncalrpc/np/netlogon unix 2 [ ] DGRAM 60987 5805/smbd /usr/local/samba/private/msg.sock/5805 unix 2 [ ACC ] STREAM LISTENING 14842 898/samba /usr/local/samba/var/run/ncalrpc/np/samr unix 2 [ ACC ] STREAM LISTENING 14389 1/systemd /var/run/cups/cups.sock unix 2 [ ] DGRAM 63000 6458/smbd /usr/local/samba/private/msg.sock/6458 unix 2 [ ACC ] STREAM LISTENING 14843 898/samba /usr/local/samba/var/run/ncalrpc/np/rpcecho unix 2 [ ] DGRAM 61224 5937/smbd /usr/local/samba/private/msg.sock/5937 unix 2 [ ] DGRAM 62145 5952/smbd /usr/local/samba/private/msg.sock/5952 unix 2 [ ] DGRAM 61228 5941/smbd /usr/local/samba/private/msg.sock/5941 unix 2 [ ACC ] STREAM LISTENING 57 1/systemd /run/systemd/journal/stdout unix 2 [ ] DGRAM 61231 5943/smbd /usr/local/samba/private/msg.sock/5943 unix 2 [ ACC ] STREAM LISTENING 14857 898/samba /usr/local/samba/var/run/ncalrpc/np/wkssvc unix 2 [ ] DGRAM 62573 6143/smbd /usr/local/samba/private/msg.sock/6143 unix 5 [ ] DGRAM 60 1/systemd /run/systemd/journal/socket unix 2 [ ] DGRAM 62484 6106/smbd /usr/local/samba/private/msg.sock/6106 unix 2 [ ] DGRAM 62332 6091/smbd /usr/local/samba/private/msg.sock/6091 unix 24 [ ] DGRAM 62 1/systemd /dev/log unix 2 [ ACC ] STREAM LISTENING 15681 869/python2 /var/run/fail2ban/fail2ban.sock unix 2 [ ACC ] STREAM LISTENING 14387 1/systemd @ISCSID_UIP_ABSTRACT_NAMESPACE unix 2 [ ACC ] STREAM LISTENING 20559 1058/clamd /var/run/clamd.scan/clamd.sock unix 2 [ ] DGRAM 62416 6141/smbd /usr/local/samba/private/msg.sock/6141 unix 2 [ ACC ] STREAM LISTENING 14844 898/samba /usr/local/samba/var/run/ncalrpc/DEFAULT unix 2 [ ACC ] STREAM LISTENING 14858 898/samba /usr/local/samba/var/run/ncalrpc/EPMAPPER unix 2 [ ACC ] STREAM LISTENING 14918 901/samba /usr/local/samba/private/ldapi unix 2 [ ACC ] STREAM LISTENING 1918 1/systemd /run/systemd/private unix 2 [ ] DGRAM 14240 896/samba /usr/local/samba/private/msg.sock/896 unix 2 [ ] DGRAM 62355 6107/smbd /usr/local/samba/private/msg.sock/6107 unix 2 [ ACC ] SEQPACKET LISTENING 1941 1/systemd /run/udev/control unix 2 [ ACC ] STREAM LISTENING 13721 537/NetworkManager /var/run/NetworkManager/private unix 2 [ ] DGRAM 62759 6298/smbd /usr/local/samba/private/msg.sock/6298 unix 2 [ ACC ] STREAM LISTENING 12703 1/systemd /var/run/avahi-daemon/socket unix 2 [ ACC ] STREAM LISTENING 15937 908/winbindd /usr/local/samba/var/run/winbindd/pipe unix 2 [ ] DGRAM 67804 6308/smbd /usr/local/samba/private/msg.sock/6308 unix 2 [ ] DGRAM 67766 6301/smbd /usr/local/samba/private/msg.sock/6301 unix 2 [ ] DGRAM 62778 6307/smbd /usr/local/samba/private/msg.sock/6307 unix 2 [ ] DGRAM 63022 6471/smbd /usr/local/samba/private/msg.sock/6471 unix 2 [ ] DGRAM 62785 6312/smbd /usr/local/samba/private/msg.sock/6312 unix 2 [ ] DGRAM 68085 6448/smbd /usr/local/samba/private/msg.sock/6448 unix 2 [ ] DGRAM 68096 6462/smbd /usr/local/samba/private/msg.sock/6462 unix 2 [ ] DGRAM 62989 6452/smbd /usr/local/samba/private/msg.sock/6452 unix 2 [ ] DGRAM 69762 6609/smbd /usr/local/samba/private/msg.sock/6609 unix 2 [ ] DGRAM 68100 6464/smbd /usr/local/samba/private/msg.sock/6464 unix 2 [ ] DGRAM 63019 6469/smbd /usr/local/samba/private/msg.sock/6469 unix 2 [ ] DGRAM 63015 6466/smbd /usr/local/samba/private/msg.sock/6466 unix 2 [ ] DGRAM 68142 6467/smbd /usr/local/samba/private/msg.sock/6467 unix 2 [ ] DGRAM 68600 6476/smbd /usr/local/samba/private/msg.sock/6476 unix 2 [ ] DGRAM 68605 6478/smbd /usr/local/samba/private/msg.sock/6478 unix 2 [ ] DGRAM 63034 6479/smbd /usr/local/samba/private/msg.sock/6479 unix 2 [ ] DGRAM 69776 6611/smbd /usr/local/samba/private/msg.sock/6611 unix 2 [ ] DGRAM 69798 6620/smbd /usr/local/samba/private/msg.sock/6620 unix 2 [ ] DGRAM 63332 6614/smbd /usr/local/samba/private/msg.sock/6614 unix 2 [ ] DGRAM 63334 6622/smbd /usr/local/samba/private/msg.sock/6622 unix 2 [ ] DGRAM 63343 6628/smbd /usr/local/samba/private/msg.sock/6628 unix 2 [ ] DGRAM 69801 6629/smbd /usr/local/samba/private/msg.sock/6629 unix 2 [ ACC ] STREAM LISTENING 14388 1/systemd @ISCSIADM_ABSTRACT_NAMESPACE unix 2 [ ACC ] STREAM LISTENING 16843 791/cupsd /var/run/cups/cups.sock unix 2 [ ] DGRAM 62753 6292/smbd /usr/local/samba/private/msg.sock/6292 unix 3 [ ] STREAM CONNECTED 872 527/audispd unix 3 [ ] STREAM CONNECTED 13805 538/dbus-daemon /var/run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 12845 538/dbus-daemon /var/run/dbus/system_bus_socket unix 2 [ ] STREAM CONNECTED 40938 3243/httpd unix 2 [ ] DGRAM 15254 1058/clamd unix 3 [ ] STREAM CONNECTED 50908 1882/smbd unix 2 [ ] STREAM CONNECTED 15274 1070/httpd unix 3 [ ] STREAM CONNECTED 915 536/systemd-logind unix 3 [ ] STREAM CONNECTED 14563 556/wpa_supplicant unix 3 [ ] STREAM CONNECTED 17188 1057/httpd unix 3 [ ] DGRAM 13400 411/systemd-udevd unix 2 [ ] DGRAM 61474 5637/smbd unix 2 [ ] STREAM CONNECTED 60715 536/systemd-logind unix 3 [ ] STREAM CONNECTED 16108 908/winbindd unix 3 [ ] STREAM CONNECTED 16846 791/cupsd unix 3 [ ] STREAM CONNECTED 18715 1198/crond unix 2 [ ] DGRAM 933 545/avahi-daemon: r unix 3 [ ] STREAM CONNECTED 57347 908/winbindd unix 3 [ ] STREAM CONNECTED 13938 791/cupsd unix 3 [ ] STREAM CONNECTED 12900 538/dbus-daemon /var/run/dbus/system_bus_socket unix 2 [ ] STREAM CONNECTED 60790 5643/smbd unix 3 [ ] STREAM CONNECTED 48652 898/samba /usr/local/samba/var/run/ncalrpc/np/samr unix 2 [ ] STREAM CONNECTING 0 - /usr/local/samba/var/run/winbindd/pipe unix 3 [ ] STREAM CONNECTED 14436 387/systemd-journal /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 14479 537/NetworkManager unix 2 [ ] DGRAM 949 537/NetworkManager unix 3 [ ] STREAM CONNECTED 14939 906/smbd unix 3 [ ] STREAM CONNECTED 14551 387/systemd-journal /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 17519 387/systemd-journal /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 60835 908/winbindd /usr/local/samba/var/locks/winbindd_privileged/pipe unix 2 [ ] STREAM CONNECTED 60784 5641/smbd unix 2 [ ] DGRAM 12760 538/dbus-daemon unix 3 [ ] STREAM CONNECTED 15685 538/dbus-daemon /var/run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 15940 908/winbindd unix 3 [ ] STREAM CONNECTED 937 549/avahi-daemon: c unix 2 [ ] STREAM CONNECTED 60736 5637/smbd unix 2 [ ] DGRAM 27437 1882/smbd unix 3 [ ] STREAM CONNECTED 12745 1/systemd unix 3 [ ] STREAM CONNECTED 13724 537/NetworkManager unix 2 [ ] STREAM CONNECTING 0 - /usr/local/samba/var/run/winbindd/pipe unix 2 [ ] DGRAM 58792 5641/smbd unix 2 [ ] STREAM CONNECTED 15252 1058/clamd unix 3 [ ] STREAM CONNECTED 16148 387/systemd-journal /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 936 545/avahi-daemon: r unix 2 [ ] DGRAM 59923 5643/smbd unix 3 [ ] STREAM CONNECTED 13263 387/systemd-journal /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTING 0 - /usr/local/samba/var/run/winbindd/pipe unix 3 [ ] STREAM CONNECTED 16107 1051/winbindd unix 3 [ ] STREAM CONNECTED 13723 537/NetworkManager unix 2 [ ] DGRAM 14099 830/iscsid unix 2 [ ] STREAM CONNECTED 54844 4971/CROND unix 3 [ ] STREAM CONNECTED 27951 898/samba /usr/local/samba/var/run/ncalrpc/np/netlogon unix 3 [ ] STREAM CONNECTED 14502 387/systemd-journal /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 20499 387/systemd-journal /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 15268 1068/httpd unix 3 [ ] STREAM CONNECTED 14628 387/systemd-journal /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 12758 538/dbus-daemon /var/run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 14481 387/systemd-journal /run/systemd/journal/stdout unix 2 [ ] DGRAM 825 510/auditd unix 3 [ ] STREAM CONNECTED 16202 1058/clamd unix 3 [ ] STREAM CONNECTED 12756 538/dbus-daemon unix 2 [ ] STREAM CONNECTED 58754 5640/smbd unix 3 [ ] STREAM CONNECTED 13622 536/systemd-logind unix 2 [ ] DGRAM 19705 1198/crond unix 2 [ ] DGRAM 13391 411/systemd-udevd unix 2 [ ] STREAM CONNECTING 0 - /usr/local/samba/var/run/winbindd/pipe unix 2 [ ] DGRAM 59881 5639/smbd unix 2 [ ] STREAM CONNECTING 0 - /usr/local/samba/var/run/winbindd/pipe unix 3 [ ] STREAM CONNECTED 871 532/sedispatch unix 3 [ ] STREAM CONNECTED 13669 546/irqbalance unix 3 [ ] STREAM CONNECTED 13803 557/polkitd unix 3 [ ] STREAM CONNECTED 12755 538/dbus-daemon unix 2 [ ] STREAM CONNECTED 15276 1071/httpd unix 3 [ ] STREAM CONNECTED 823 527/audispd unix 3 [ ] STREAM CONNECTED 9940 387/systemd-journal /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 16830 387/systemd-journal /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 55177 5166/ntpd unix 3 [ ] STREAM CONNECTED 13984 796/sshd unix 2 [ ] DGRAM 46636 3604/sshd: root@pts unix 3 [ ] STREAM CONNECTED 12807 387/systemd-journal /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 12757 538/dbus-daemon /var/run/dbus/system_bus_socket unix 2 [ ] DGRAM 16862 829/iscsid unix 2 [ ] DGRAM 866 527/audispd unix 3 [ ] STREAM CONNECTED 15939 919/winbindd unix 2 [ ] DGRAM 9881 1/systemd unix 3 [ ] STREAM CONNECTED 13216 387/systemd-journal /run/systemd/journal/stdout unix 3 [ ] STREAM CONNECTED 15705 791/cupsd unix 2 [ ] DGRAM 13616 536/systemd-logind unix 3 [ ] STREAM CONNECTED 16833 538/dbus-daemon /var/run/dbus/system_bus_socket unix 2 [ ] STREAM CONNECTED 59880 5639/smbd unix 2 [ ] STREAM CONNECTED 27426 1882/smbd unix 3 [ ] STREAM CONNECTED 14480 387/systemd-journal /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTING 0 - /usr/local/samba/var/run/winbindd/pipe unix 2 [ ] DGRAM 58777 5640/smbd unix 2 [ ] DGRAM 14086 823/xinetd unix 2 [ ] STREAM CONNECTED 17523 1069/httpd unix 3 [ ] STREAM CONNECTED 14215 798/python unix 3 [ ] STREAM CONNECTED 824 510/auditd unix 2 [ ] STREAM CONNECTED 43074 3249/httpd unix 2 [ ] STREAM CONNECTED 42298 3231/httpd unix 3 [ ] STREAM CONNECTED 13261 798/python unix 2 [ ] STREAM CONNECTED 15278 1067/httpd unix 3 [ ] DGRAM 13401 411/systemd-udevd unix 3 [ ] STREAM CONNECTED 13804 538/dbus-daemon /var/run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 57346 5115/winbindd unix 3 [ ] STREAM CONNECTED 15754 888/colord unix 3 [ ] STREAM CONNECTED 941 545/avahi-daemon: r unix 3 [ ] STREAM CONNECTED 16847 538/dbus-daemon /var/run/dbus/system_bus_socket unix 2 [ ] DGRAM 57466 5166/ntpd unix 2 [ ] STREAM CONNECTED 40952 3248/httpd unix 3 [ ] STREAM CONNECTED 14688 387/systemd-journal /run/systemd/journal/stdout unix 2 [ ] STREAM CONNECTED 46626 3604/sshd: root@pts unix 2 [ ] STREAM CONNECTED 19923 1223/named unix 3 [ ] STREAM CONNECTED 13791 556/wpa_supplicant unix 2 [ ] DGRAM 1009 557/polkitd unix 2 [ ] STREAM CONNECTED 43071 3247/httpd unix 3 [ ] STREAM CONNECTED 14544 538/dbus-daemon /var/run/dbus/system_bus_socket unix 2 [ ] DGRAM 16229 1057/httpd unix 3 [ ] STREAM CONNECTED 12806 545/avahi-daemon: r unix 3 [ ] STREAM CONNECTED 14278 538/dbus-daemon /var/run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 14030 807/sh unix 3 [ ] STREAM CONNECTED 13610 538/dbus-daemon unix 3 [ ] STREAM CONNECTED 9939 411/systemd-udevd unix 2 [ ] DGRAM 17636 1223/named unix 3 [ ] STREAM CONNECTED 15706 538/dbus-daemon /var/run/dbus/system_bus_socket unix 3 [ ] STREAM CONNECTED 25471 1882/smbd unix 3 [ ] STREAM CONNECTED 16832 888/colord unix 2 [ ] DGRAM 43035 1066/logger netstat: no support for `AF IPX' on this system. netstat: no support for `AF AX25' on this system. netstat: no support for `AF X25' on this system. netstat: no support for `AF NETROM' on this system. smbcontrol pool-usage 898 Can't find pid for destination 'pool-usage' [root@~]# [global] ldap server require strong auth = no allow dcerpc auth level connect = yes client ipc signing = default client ipc max protocol = default client ipc min protocol = default ntlm auth = No lanman auth = No raw NTLMv2 auth = No client NTLMv2 auth = Yes client lanman auth = Yes server max protocol = SMB3_11 server min protocol = LANMAN1 client max protocol = SMB3_11 client min protocol = CORE nsupdate command = /usr/bin/nsupdate -g # netbios aliases = XXXXX log level = 3 passdb:3 auth:10 winbind:2 log file = /var/log/samba/log.%m max log size = 0 logging = file allow dns updates = nonsecure and secure dns proxy = No dns forwarder = restrict anonymous = 0 usershare allow guests = No security = USER allow trusted domains = Yes bind interfaces only = Yes obey pam restrictions = Yes interfaces = lo ens192 auth methods = server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl, winbindd, ntp_signd, kcc, dnsupdate dcerpc endpoint servers = epmapper, wkssvc, rpcecho, samr, netlogon, lsarpc, drsuapi, dssetup, unixinfo, browser, eventlog6, backupkey, remote, dnsserver dos charset = CP850 unix charset = UTF-8 workgroup = TEST realm = TEST.LOCAL netbios name = XXXXX server string = XXXXX Samba Server hosts allow = ALL 127.0.0.1 guest ok = No guest account = nobody map to guest = Bad User guest only = No config backend = file server role = active directory domain controller server role check:inhibit = yes encrypt passwords = Yes old password allowed period = 120 password server = XXXXX.TEST.local smb passwd file = /usr/local/samba/private/smbpasswd private dir = /usr/local/samba/private passdb backend = tdbsam algorithmic rid base = 1000 pam password change = Yes passwd chat = *New*password* %n\n *ReType*new*password* %n\n*passwd:*all*authentication*tokens*updated*successfully* passwd chat debug = No passwd chat timeout = 2 username level = 0 unix password sync = Yes client plaintext auth = Yes dedicated keytab file = /etc/krb5.keytab kerberos method = system keytab map untrusted to domain = Yes debug timestamp = Yes debug prefix timestamp = No debug hires timestamp = Yes debug pid = No debug uid = No debug class = No enable core files = Yes smb ports = 445 137 138 139 large readwrite = Yes client schannel = Auto server schannel = Auto unicode = Yes min receivefile size = 16384 read raw = Yes write raw = Yes disable netbios = No reset on zero vc = No log writeable files on exit = Yes defer sharing violations = Yes nt pipe support = Yes nt status support = Yes max mux = 50 max xmit = 32768 name resolve order = wins bcast hosts max ttl = 259200 max wins ttl = 518400 min wins ttl = 21600 time server = Yes unix extensions = No # client signing = required # server signing = required client signing = mandatory server signing = mandatory client use spnego = Yes client ldap sasl wrapping = sign enable asu support = No cldap port = 389 krb5 port = 88 kpasswd port = 464 web port = 901 rpc big endian = No deadtime = 0 getwd cache = Yes keepalive = 300 lpq cache time = 30 smbd profiling level = off spotlight = no tls priority = NORMAL:-VERS-SSL3.0 max smbd processes = 0 max disk size = 0 max open files = 65535 socket options = IPTOS_LOWDELAY TCP_NODELAY SO_KEEPALIVE use mmap = Yes hostname lookups = No name cache timeout = 3600 clustering = No ctdb timeout = 0 ctdb locktime warn threshold = 0 smb2 max read = 8388608 smb2 max write = 8388608 smb2 max trans = 8388608 smb2 max credits = 8192 mangling method = hash mangle prefix = 1 max stat cache size = 256 stat cache = No machine password timeout = 0 username map cache time = 0 logon script = logon path = logon drive = logon home = domain logons = Yes init logon delay = 100 os level = 255 lm announce = Auto lm interval = 60 preferred master = Yes local master = Yes domain master = Yes browse list = Yes enhanced browsing = No wins proxy = Yes wins support = Yes smb2 leases = Yes ldap admin dn = ldap delete dn = No ldap group suffix = ldap idmap suffix = ldap machine suffix = ldap passwd sync = no ldap replication sleep = 1000 ldap suffix = ldap ssl = start tls ldap ssl ads = No ldap deref = auto ldap follow referral = Auto ldap timeout = 15 ldap connection timeout = 2 ldap page size = 1024 ldap user suffix = ldap debug level = 0 ldap debug threshold = 10 lock directory = /usr/local/samba/var/lock state directory = /usr/local/samba/var/locks cache directory = /usr/local/samba/var/cache pid directory = /usr/local/samba/var/run ntp signd socket directory = /usr/local/samba/var/lib/ntp_signd utmp = No nmbd bind explicit broadcast = Yes homedir map = auto.home afs token lifetime = 604800 afs share = No afs username map = NIS homedir = Yes registry shares = Yes usershare max shares = 0 usershare owner only = No usershare path = /usr/local/samba/var/locks/usershares allow insecure wide links = No async smb echo handler = No host msdfs = No msdfs shuffle referrals = No passdb expand explicit = No idmap cache time = 604800 idmap negative cache time = 120 template homedir = /home/%D/%U template shell = /bin/bash winbind cache time = 3600 winbind reconnect delay = 30 winbind request timeout = 60 winbind max clients = 2000 winbind enum users = Yes winbind enum groups = Yes winbind use default domain = Yes winbind trusted domains only = No winbind nested groups = Yes winbind expand groups = 10 winbind nss info = rfc2307 winbind refresh tickets = Yes winbind offline logon = Yes winbind normalize names = Yes winbind rpc only = Yes create krb5 conf = Yes ncalrpc dir = /usr/local/samba/var/run/ncalrpc winbind max domain connections = 1 winbindd socket directory = /usr/local/samba/var/run/winbindd winbindd privileged socket directory = /usr/local/samba/var/lib/winbindd_privileged winbind sealed pipes = No neutralize nt4 emulation = Yes winbind sealed pipes:TEST = Yes require strong key:TEST = Yes reject md5 servers = No require strong key = No # require strong key = Yes multicast dns register = Yes samba kcc command = /usr/local/samba/sbin/samba_kcc spn update command = /usr/local/samba/sbin/samba_spnupdate share backend = classic allow nt4 crypto = Yes reject md5 clients = No tls enabled = Yes tls keyfile = tls/key.pem tls certfile = tls/cert.pem tls cafile = tls/ca.pem tls crlfile = tls dh params file = spoolssd:prefork_child_min_life = 60 spoolssd:prefork_max_allowed_clients = 200 spoolssd:prefork_spawn_rate = 5 spoolssd:prefork_max_children = 75 spoolssd:prefork_min_children = 5 rpc_server:tcpip = no rpc_daemon:spoolssd = fork rpc_server:default = external rpc_server:spoolss = external rpc_server:svcctl = embedded rpc_server:srvsvc = embedded rpc_server:eventlog = embedded rpc_server:ntsvcs = embedded rpc_server:winreg = embedded dsdb:schema update allowed = yes sdb:schema update allowed = yes idmap config TEST : range = 2000000-9999999 idmap config TEST : default = yes idmap config TEST : backend = ad idmap config TEST : readonly = no idmap config TEST : schema_mode = rfc2307 idmap config TEST : cache time = 3600 idmap config * : default = yes idmap config * : readonly = no idmap config * : schema_mode = rfc2307 idmap config * : backend = tdb idmap config * : range = 2000000-9999999 idmap_ldb:use rfc2307 = yes idmap config all : readonly = yes idmap config all : default = yes idmap config all : backend = tdb dbwrap_tdb_mutexes:* = yes prefork children:smb = 4 registry:hkey_users = hku.ldb registry:hkey_local_machine = hklm.ldb kccsrv:samba_kcc = true read only = Yes ##acl ayarlari### acl group control = Yes acl map full control = Yes acl allow execute always = Yes force unknown acl user = Yes inherit permissions = Yes inherit acls = Yes inherit owner = No ##inherit owner No olmalidir. hangi kullanici ile dosya kaydettigini belirtir map acl inherit = Yes nt acl support = Yes acl:search = no acl_xattr:ignore system acl = yes profile acls = Yes ##acl ayarlari sonu administrative share = No allocation roundup size = 1048576 aio read size = 16384 aio write size = 16384 aio max threads = 100 ea support = Yes smb encrypt = default durable handles = Yes block size = 1024 change notify = Yes directory name cache size = 100 kernel change notify = No max connections = 65535 strict allocate = Yes strict rename = No strict sync = No sync always = No use sendfile = Yes write cache size = 0 default case = lower case sensitive = No preserve case = Yes short preserve case = Yes mangling char = ~ hide dot files = Yes hide special files = No hide unreadable = No hide unwriteable files = No delete veto files = No map archive = Yes map hidden = Yes map system = Yes map readonly = yes mangled names = Yes store dos attributes = Yes dmapi support = No browseable = Yes access based share enum = No blocking locks = Yes csc policy = disable lock spin time = 200 oplock break wait time = 0 fake oplocks = No kernel oplocks = Yes kernel share modes = Yes locking = Yes oplocks = Yes level2 oplocks = Yes oplock contention limit = 2 posix locking = Yes strict locking = Auto dfree cache time = 0 preexec close = No root preexec close = No available = Yes fstype = NTFS wide links = Yes follow symlinks = Yes delete readonly = No dos filemode = Yes dos filetimes = Yes dos filetime resolution = Yes fake directory create times = No msdfs root = No ntvfs handler = unixuid, default vfs objects = acl_xattr full_audit full_audit:prefix = IP=%I|USER=%u|MACHINE=%m|VOLUME=%S full_audit:failure = connect disconnect full_audit:success = connect disconnect opendir mkdir rmdir closedir open close read pread write pwrite sendfile rename unlink chmod fchmod chown fchown chdir ftruncate lock symlink readlink link mknod full_audit:TEST = local5 full_audit:priority = notice #printer satirlari# load printers = No use client driver = No show add printer wizard = Yes printcap cache time = 0 printcap name = cups cups encrypt = No cups connection timeout = 60 disable spoolss = No min print space = 0 max reported print jobs = 0 max print jobs = 1000 print notify backchannel = No printing = cups cups options = raw default devmode = Yes force printername = Yes printjob username = %U spoolss: architecture = Windows x64 [homes] comment = Home Directories path = /strg/homes/%U valid users = @"Domain Users" admin users = @"Domain Admins" read only = No browseable = No writable = Yes create mask = 0644 force create mode = 0660 force directory mode = 0770 directory mask = 0755 veto files = /*.encrypted/*.ecc/*.ccc/ hide files = /Recycle Bin/ vfs objects = acl_xattr full_audit recycle recycle:mode = KEEP_DIRECTORIES|VERSION|TOUCH recycle:noversions = *.tmp|*.temp|*.dat|*.ini recycle:excludedir = /Recycle Bin recycle:exclude = *.tmp|*.temp|*.o|*.obj|~$*|*.??|*.log|*.trace|*.TMP|*.ASV|*.$$$|*.asv recycle:versions = yes recycle:touch_mtime = yes recycle:touch = yes recycle:maxsize = 0 recycle:minsize = 0 recycle:keeptree = yes recycle:repository = Recycle Bin [profiles] comment = Network Profiles Share path = /strg/profiles directory mask = 0755 create mask = 0644 force create mode = 0660 force directory mode = 0770 profile acls = Yes guest ok = No read only = No #browseable izni bir defalik kurulum esnasinda permision ayarlari esnasinda acilir.sonra kapatilir. browseable = No write ok = Yes [netlogon] comment = Network Netlogon Share path = /usr/local/samba/var/locks/sysvol/TEST.local/scripts read only = No guest ok = No write ok = Yes browseable = No [sysvol] path = /usr/local/samba/var/locks/sysvol read only = No browseable = No write ok = Yes [printers] comment = All Printers path = /var/spool/samba write list = root administrator writeable = No printable = Yes print ok = Yes guest ok = Yes public = Yes browseable = Yes acl_xattr:ignore system acl = yes [print$] comment = Printer Drivers path = /strg/printer_drivers invalid users = qwerty valid users = @"Domain Users" admin users = @"Domain Admins" write list = root administrator writeable = Yes read only = Yes browseable = Yes guest ok = Yes create mask = 0660 create mask = 0644 force create mode = 0660 force directory mode = 0770 directory mask = 0755 acl_xattr:ignore system acl = yes
Sorry, please use against all the stuck processes: smbcontrol <pid> pool-usage also attach with gdb -p <pid> and get a 'bt full'. Take care that both may expose sensitive system details, but as this is an rc release I'm assuming this is a test system. Thanks!
BTW, can you confirm the last version that this worked on? I'm keen to know if this is a regression from Samba 4.4 (it seems clear it is), so we can mark it as such. Thanks!
(In reply to Andrew Bartlett from comment #4) thanks for your cooperation. I updated on live system,a problem occured during updating and users were used the computers. Therefore, take this update back to 4.4.5 version. I have no problem about this 4.4.5 version now. Kind regards. Barış
If you are able to safely reproduce this somehow, and can get me that 'bt full' from a stuck smbd process, that would help narrow this down a lot. Thanks,
Created attachment 12356 [details] netstat
Created attachment 12357 [details] smbcontrol
Version 4.5.0rc2 critical problem. server very slow. ssh vs.. command dont respond.
â samba.service - Samba Daemon Loaded: loaded (/usr/lib/systemd/system/samba.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2016-08-10 19:13:49 EEST; 22min ago Process: 879 ExecStart=/usr/local/samba/sbin/samba --daemon $SAMBAOPTIONS (code=exited, status=0/SUCCESS) Main PID: 883 (samba) CGroup: /system.slice/samba.service ââ 883 /usr/local/samba/sbin/samba --daemon ââ 927 /usr/local/samba/sbin/samba --daemon ââ 928 /usr/local/samba/sbin/samba --daemon ââ 929 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ 930 /usr/local/samba/sbin/samba --daemon ââ 931 /usr/local/samba/sbin/samba --daemon ââ 932 /usr/local/samba/sbin/samba --daemon ââ 933 /usr/local/samba/sbin/samba --daemon ââ 934 /usr/local/samba/sbin/samba --daemon ââ 935 /usr/local/samba/sbin/samba --daemon ââ 936 /usr/local/samba/sbin/samba --daemon ââ 937 /usr/local/samba/sbin/samba --daemon ââ 938 /usr/local/samba/sbin/samba --daemon ââ 939 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground ââ 940 /usr/local/samba/sbin/samba --daemon ââ 946 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground ââ 947 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ 948 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ 949 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1015 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground ââ1374 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1377 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1382 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1383 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1384 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1385 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1387 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1388 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1389 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1391 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1393 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1523 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1525 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1526 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1527 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1529 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1530 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1535 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1542 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1543 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1544 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1545 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1550 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1551 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1553 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1554 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1555 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1556 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1557 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1558 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1559 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1561 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1692 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1693 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1695 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1697 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1698 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1700 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1701 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1703 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1708 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1710 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1712 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1713 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1714 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1716 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1718 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1720 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1721 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1722 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1724 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1862 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1868 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1869 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1870 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1873 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1874 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1876 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1877 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1878 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1879 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1884 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1886 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1887 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1888 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1901 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1902 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1904 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1906 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1909 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1913 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1915 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1917 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1919 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1922 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2057 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2061 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2062 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2063 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2065 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2082 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2086 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2088 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2090 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2093 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2095 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2096 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2097 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2110 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2117 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2118 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2119 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2125 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2128 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2145 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2164 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2277 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2278 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2281 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2282 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2283 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2302 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2305 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2306 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2307 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2313 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2314 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2316 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2318 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2319 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2327 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2330 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2331 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2333 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2334 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2336 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2338 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2339 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2340 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2344 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2474 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2476 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2477 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2479 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2480 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2483 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2487 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2493 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2494 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2495 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2500 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2502 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2504 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2505 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2507 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2509 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
my problem is solved. problem this line >> obey pam restrictions = Yes samba 4.4.x (all version) obey pam restrictions = Yes >>> running >> no problem samba 4.5.X obey pam restrictions = No >> running >> no problem ?? best regards
Created attachment 12364 [details] smbcontrol_last_error :(( problem resuming. doesnt solved.
So, to summarise: This isn't a case that the process is blocked somewhere deep in the stack, because smbcontrol talloc reports (which don't really show anything interesting as far as I can tell) will only return when the process is in the main tevent loop. To me this means that the process is not detecting the end of file from the dropped socket, or that something else is keeping the process alive because it isn't for a specific client. Does the problem go away if you stop using printing, and set 'disable spoolss = yes'? Team: Do we know what this process is? Could it be a helper process of some kind? How could we tell? Thanks,
(In reply to Andrew Bartlett from comment #13) If this is on Linux, what does /proc/<pid> say about it ? Specifically what open file descriptors does it have ?
(In reply to baris tombul from comment #0) [2016/07/30 07:00:47.601811, 0] ../source3/smbd/server.c:1025(smbd_accept_connection) smbd_accept_connection: fork() failed: Cannot allocate memory Indicated that it's the parent smbd Can you paste the content of /usr/lib/systemd/system/samba.service and the output of 'smbstatus' There's not much we can do if fork(2) fails with ENOMEM. The number of processes and/or memory could be limited by a ulimit or cgroups. As this a not a generic problem I'll remove the regression flag. This should not block the 4.5.0 release.
cat /usr/lib/systemd/system/samba.service [Unit] Description=Samba Daemon Wants=nss-lookup.target Wants=named-setup-rndc.service Before=nss-lookup.target After=syslog.target network.target nmb.service [Service] PermissionsStartOnly=true Type=forking Environment=KRB5CCNAME=/usr/local/samba/var/run/krb5cc_samba EnvironmentFile=-/etc/sysconfig/samba PIDFile=/usr/local/samba/var/run/samba.pid LimitNOFILE=16384 ExecStart=/usr/local/samba/sbin/samba --daemon $SAMBAOPTIONS ExecReload=/bin/sh -c '/usr/local/samba/sbin/samba reload > /dev/null 2>&1 || /bin/kill -HUP $MAINPID || /bin/kill -HUP -a smbd || /bin/kill -HUP -a samba || /bin/kill -HUP -a winbindd' ExecStop=/bin/sh -c '/usr/local/samba/sbin/samba stop > /dev/null 2>&1 || /bin/kill -TERM $MAINPID || /bin/kill -TERM -a smbd || /bin/kill -TERM -a samba || /bin/kill -TERM -a winbindd' ExecStop=/bin/sh -c '/usr/bin/rm -f /usr/local/samba/var/run/smbd.pid' ExecStop=/bin/sh -c '/usr/bin/rm -f /usr/local/samba/var/run/samba.pid' ExecStop=/bin/sh -c '/usr/bin/rm -f /usr/local/samba/var/run/winbindd.pid' LimitCORE=infinity PrivateTmp=true Restart=on-failure [Install] WantedBy=multi-user.target ------------ service samba status Redirecting to /bin/systemctl status samba.service â samba.service - Samba Daemon Loaded: loaded (/usr/lib/systemd/system/samba.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2016-09-06 14:54:30 EEST; 1h 48min ago Process: 913 ExecStart=/usr/local/samba/sbin/samba --daemon $SAMBAOPTIONS (code=exited, status=0/SUCCESS) Main PID: 915 (samba) CGroup: /system.slice/samba.service ââ 915 /usr/local/samba/sbin/samba --daemon ââ 960 /usr/local/samba/sbin/samba --daemon ââ 961 /usr/local/samba/sbin/samba --daemon ââ 962 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ 963 /usr/local/samba/sbin/samba --daemon ââ 964 /usr/local/samba/sbin/samba --daemon ââ 965 /usr/local/samba/sbin/samba --daemon ââ 966 /usr/local/samba/sbin/samba --daemon ââ 967 /usr/local/samba/sbin/samba --daemon ââ 968 /usr/local/samba/sbin/samba --daemon ââ 969 /usr/local/samba/sbin/samba --daemon ââ 970 /usr/local/samba/sbin/samba --daemon ââ 971 /usr/local/samba/sbin/samba --daemon ââ 972 /usr/local/samba/sbin/samba --daemon ââ 973 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground ââ 978 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ 979 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ 980 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground ââ 982 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ1247 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground ââ1266 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ2569 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ4955 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ8067 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground ââ8068 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
smbstatus Samba version 4.5.0rc4-GIT-51a6036 PID Username Group Machine Protocol Version Encryption Signing ---------------------------------------------------------------------------------------------------------------------------------------- 8321 nobody 3000004 10.0.23.13 (ipv4:10.0.23.13:61117) SMB2_10 - - 8323 FACILITY\tdb04$ FACILITY\domain_computers 10.0.23.13 (ipv4:10.0.23.13:61119) SMB2_10 - HMAC-SHA256 8322 nobody 3000004 tdb04 (ipv4:10.0.23.13:61118) NT1 - - Service pid Machine Connected at Encryption Signing --------------------------------------------------------------------------------------------- IPC$ 8322 tdb04 Tue Sep 6 04:45:29 PM 2016 EEST - - IPC$ 8323 10.0.23.13 Tue Sep 6 04:45:29 PM 2016 EEST - HMAC-SHA256 IPC$ 8321 10.0.23.13 Tue Sep 6 04:45:29 PM 2016 EEST - - No locked files [root@mems ~]# smbstatus Samba version 4.5.0rc4-GIT-51a6036 PID Username Group Machine Protocol Version Encryption Signing ---------------------------------------------------------------------------------------------------------------------------------------- 8333 FACILITY\biomems07$ FACILITY\domain_computers 10.0.10.55 (ipv4:10.0.10.55:58201) SMB2_10 - HMAC-SHA256 Service pid Machine Connected at Encryption Signing --------------------------------------------------------------------------------------------- IPC$ 8333 10.0.10.55 Tue Sep 6 04:45:42 PM 2016 EEST - HMAC-SHA256
Created attachment 12459 [details] /var/log/messages I have same issue with latest samba 4.5.0 on CentOS 7.2.1511. Kernel: Linux smb.dc.serviceand.ru 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I tried for days. I've found the problem. ""dns_lookup_kdc "" nano /etc/krb5.conf edit: dns_lookup_kdc = false my new krb5.conf: [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [kdc] check-ticket-addresses = false [libdefaults] default_realm = TEST.LOCAL dns_lookup_realm = false dns_lookup_kdc = false rdns = false proxiable = true forwardable = true allow_weak_crypto = true noaddresses = true ticket_lifetime = 24h renew_lifetime = 7d [appdefaults] pam = { debug = false ticket_lifetime = 36000 renew_lifefime = 36000 forwardable = true krb4_convert = false } [domain_realm] facility.local = TEST.LOCAL .facility.local = TEST.LOCAL [realms] FACILITY.LOCAL = { kdc = xxx.test.local admin_server = XXX.TEST.LOCAL default_domain = TEST.LOCAL }
So this looks then like a memory leak inside the krb5 libraries, as dns_lookup_kdc = false is only relevent in krb5.conf. What version of kerberos are you using ?
(In reply to Jeremy Allison from comment #20) rpm -qa | grep krb krb5-libs-1.13.2-12.el7_2.x86_64 krb5-devel-1.13.2-12.el7_2.x86_64 krb5-workstation-1.13.2-12.el7_2.x86_64 krb5-pkinit-1.13.2-12.el7_2.x86_64 pam_krb5-2.4.8-4.el7.x86_64
*** Bug 12267 has been marked as a duplicate of this bug. ***
I'm having the same issue on 10 servers in 1 domain. It started after upgrading from 4.4.5 to 4.5.0. About 1 or 2 crashing servers each day. I have now changed the dns_lookup_kdc setting on all 10 servers. Will let you know if this solves the problem.
Some thoughts: If you don't have a KDC listed in your krb5.conf for TEST.LOCAL, then setting 'dns_lookup_kdc=false' is most likely to just break replication. Of course, that might be the clue. However we do need to work out what kind of process the apparently un-attached children are? Specifically, we need the open file descriptors that Jeremy asked for. Also, to isolate the issue, we need to eliminate printing: Does 'disable spoolss = yes' help?
Furthermore: Are the 'stuck' processes consuming CPU? If so, can you please get me a flamegraph of that per: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html Are they consuming ram? if so, can you confirm the talloc tree provided earlier is from a stuck smbd process? What else can we be told about them? We know they are not network-connected as they don't show up in netstat. Can we please get a level 10 log? (mark as private if need be)
Looks like you're right - "samba-tool drs showrepl" was showing thousands of failures on each server. After re-enabling dns_lookup_kdc and restarting samba, it's back to "0 consecutive failure(s)". This is a live environment with 10 servers and there's only 1 or 2 servers crashing each day. As you can imagine, when one does crash, there's quite some pressure from users to restart the thing a.s.a.p. So, to summarize, i should set log level to 10 and then when a crash occurs, before restarting samba, collect this info: - open file descriptors - flamegraph - talloc tree from a stuck smbd process - log files - smbcontrol <pid> pool-usage - gdb -p <pid> -> 'bt full' ...correct? Anything else? Should i recompile with debugging symbols?
Presumably this problem builds over time, then the server overloads? It should be enough to get this kind of info from a server pre-failure, when it is starting to show the extra tasks without matching client connections. I'm most interested in what these extra smbd processes are, if you see them. smbcontrol can turn up the log level per-process if level 10 is too much to hold across the whole system, but at this point I need as many clues as possible. Getting log files per-pid may be helpful with %p in the log file option. Also run 'samba-tool processes' so we at least get the names for some of the other tasks, as context. While a patch for master has just been accepted to squash this message, I'm still deeply suspicious that it is something like a cleanup failing, and so the task smbd not exiting: Sep 13 15:58:31 smb smbd[3160]: [2016/09/13 15:58:31.530724, 0] ../source3/smbd/smbd_cleanupd.c:172(smbd_cleanupd_process_exited) Sep 13 15:58:31 smb smbd[3160]: smbd_cleanupd_process_exited: got 0 cleanup events, expected at least 1 Finally, yes this will all be much better with debug symbols.
Also /proc/[pid]/comm for each stuck (smbd) process
my full stable smb.conf: [global] enumports command = /usr/local/bin/show-ports.sh ntlm auth = Yes lanman auth = No raw NTLMv2 auth = No client NTLMv2 auth = No client lanman auth = No idmap_ldb:use rfc2307 = Yes algorithmic rid base = 1000 kerberos method = secrets and keytab dedicated keytab file = /etc/krb5.keytab winbind max clients = 2000 winbindd:use external pipes = true winbind cache time = 300 winbind reconnect delay = 30 winbind request timeout = 60 winbind max domain connections = 1 winbindd socket directory = /usr/local/samba/var/run/winbindd winbindd privileged socket directory = /usr/local/samba/var/lib/winbindd_privileged winbind enum users = Yes winbind enum groups = Yes winbind use default domain = Yes winbind trusted domains only = No winbind nested groups = Yes winbind expand groups = 10 winbind nss info = rfc2307 winbind refresh tickets = Yes winbind offline logon = Yes winbind normalize names = Yes winbind sealed pipes = Yes winbind rpc only = Yes wins proxy = Yes wins support = Yes obey pam restrictions = No ldap server require strong auth = no server max protocol = SMB3 server min protocol = LANMAN1 server multi channel support = No client max protocol = default client min protocol = CORE restrict anonymous = 0 security = USER bind interfaces only = Yes interfaces = lo ens192 auth methods = server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl, winbindd, ntp_signd, kcc, dnsupdate dcerpc endpoint servers = epmapper, wkssvc, rpcecho, samr, netlogon, lsarpc, drsuapi, dssetup, unixinfo, browser, eventlog6, backupkey, remote, dnsserver dos charset = CP850 unix charset = UTF-8 workgroup = TEST realm = TEST.LOCAL netbios name = XXX netbios scope = server string = Test Samba Server hosts allow = ALL 127.0.0.1 guest ok = No server role = active directory domain controller server role check:inhibit = yes log level = 3 passdb:3 auth:10 winbind:2 log file = /var/log/samba/log.%m rndc command = /usr/sbin/rndc max log size = 0 set primary group script = logging = file allow dns updates = nonsecure and secure dns update command = /usr/local/samba/sbin/samba_dnsupdate pam password change = Yes smb ports = 445 139 nbt port = 137 kpasswd port = 464 krb5 port = 88 web port = 901 nbt port = 137 dgram port = 138 cldap port = 389 socket options = IPTOS_LOWDELAY TCP_NODELAY SO_KEEPALIVE domain logons = Yes os level = 255 preferred master = Yes local master = Yes domain master = Yes load printers = No use client driver = No show add printer wizard = Yes printcap cache time = 0 printcap name = cups cups encrypt = No cups connection timeout = 60 disable spoolss = No min print space = 0 max reported print jobs = 0 max print jobs = 1000 print notify backchannel = No printing = cups cups options = raw default devmode = Yes force printername = Yes printjob username = %U lpq cache time = 30 spoolss: architecture = Windows x64 debug timestamp = Yes debug prefix timestamp = No debug hires timestamp = Yes debug pid = No debug uid = No debug class = No timestamp logs = Yes require strong key = Yes allow dcerpc auth level connect = No client ipc signing = default client ipc max protocol = default client ipc min protocol = default nsupdate command = /usr/bin/nsupdate -g dns proxy = No allow trusted domains = Yes guest account = nobody map to guest = Bad User guest only = No config backend = file encrypt passwords = Yes smb passwd file = /usr/local/samba/private/smbpasswd private dir = /usr/local/samba/private passdb expand explicit = No passdb backend = tdbsam passwd chat debug = No passwd chat timeout = 2 passwd program = /usr/local/samba/bin/smbpasswd %u passwd chat = *New*password* %n\n *ReType*new*password* %n\n*passwd:*all*authentication*tokens*updated*successfully* password server = xxx.test.local old password allowed period = 120 unix password sync = Yes client plaintext auth = No map untrusted to domain = Yes enable core files = Yes large readwrite = Yes unicode = Yes read raw = Yes write raw = Yes disable netbios = No reset on zero vc = No log writeable files on exit = No defer sharing violations = Yes nt pipe support = Yes nt status support = Yes max mux = 50 max xmit = 32768 name resolve order = lmhosts wins host bcast max ttl = 259200 max wins ttl = 518400 min wins ttl = 21600 min receivefile size = 16384 time server = Yes time server = No unix extensions = Yes server signing = mandatory client signing = mandatory client schannel = Auto server schannel = Auto client use spnego = Yes client ldap sasl wrapping = sign enable asu support = No rpc big endian = No deadtime = 0 getwd cache = Yes keepalive = 300 smbd profiling level = off spotlight = No max smbd processes = 0 max disk size = 0 max open files = 65535 use mmap = Yes hostname lookups = No name cache timeout = 3600 clustering = No ctdb timeout = 0 ctdb locktime warn threshold = 0 smb2 max read = 8388608 smb2 max write = 8388608 smb2 max trans = 8388608 smb2 max credits = 8192 mangling method = hash2 mangle prefix = 1 max stat cache size = 256 stat cache = Yes machine password timeout = 604800 username map cache time = 0 username level = 0 init logon delay = 100 lm announce = Auto lm interval = 60 browse list = Yes enhanced browsing = Yes smb2 leases = Yes lock directory = /usr/local/samba/var/lock state directory = /usr/local/samba/var/locks cache directory = /usr/local/samba/var/cache pid directory = /usr/local/samba/var/run ntp signd socket directory = /usr/local/samba/var/lib/ntp_signd utmp = No nmbd bind explicit broadcast = Yes homedir map = auto.home afs token lifetime = 604800 afs share = No NIS homedir = No registry shares = No usershare allow guests = No usershare max shares = 0 usershare owner only = Yes usershare path = /usr/local/samba/var/locks/usershares async smb echo handler = No template homedir = /home/%D/%U template shell = /bin/bash create krb5 conf = Yes ncalrpc dir = /usr/local/samba/var/run/ncalrpc neutralize nt4 emulation = No reject md5 servers = No reject md5 clients = No set quota command = multicast dns register = Yes samba kcc command = /usr/local/samba/sbin/samba_kcc spn update command = /usr/local/samba/sbin/samba_spnupdate share backend = classic allow nt4 crypto = No tls enabled = Yes tls keyfile = tls/key.pem tls certfile = tls/cert.pem tls cafile = tls/ca.pem tls crlfile = tls dh params file = tls verify peer = as_strict_as_possible tls priority = NORMAL:-VERS-SSL3.0 rpc_server:spoolss = external rpc_daemon:spoolssd = fork spoolssd:prefork_child_min_life = 60 spoolssd:prefork_max_allowed_clients = 200 spoolssd:prefork_spawn_rate = 5 spoolssd:prefork_max_children = 75 spoolssd:prefork_min_children = 5 acl group control = No acl map full control = Yes acl allow execute always = No force unknown acl user = No inherit permissions = Yes inherit acls = Yes inherit owner = No map acl inherit = Yes nt acl support = Yes profile acls = No administrative share = No allocation roundup size = 1048576 aio read size = 16384 aio write size = 16384 aio max threads = 100 ea support = No smb encrypt = default durable handles = Yes block size = 1024 change notify = Yes directory name cache size = 100 kernel change notify = Yes max connections = 0 strict allocate = No strict rename = No strict sync = No sync always = No use sendfile = No write cache size = 0 default case = lower case sensitive = No preserve case = Yes short preserve case = Yes mangling char = ~ hide dot files = Yes hide special files = No hide unreadable = No hide unwriteable files = No delete veto files = No map archive = No map hidden = No map system = No map readonly = No mangled names = Yes mangling char = ~ store dos attributes = Yes dmapi support = No browseable = Yes access based share enum = No blocking locks = Yes csc policy = manual lock spin time = 200 oplock break wait time = 0 fake oplocks = No kernel oplocks = No kernel share modes = Yes locking = Yes oplocks = Yes level2 oplocks = Yes oplock contention limit = 2 posix locking = Yes strict locking = Auto dfree cache time = 0 preexec close = No root preexec close = No available = Yes fstype = NTFS wide links = No allow insecure wide links = No follow symlinks = Yes delete readonly = No dos filemode = No dos filetimes = Yes dos filetime resolution = No fake directory create times = No host msdfs = Yes msdfs root = No msdfs shuffle referrals = No ntvfs handler = unixuid, default vfs objects = dfs_samba4 acl_xattr full_audit:prefix = IP=%I|USER=%u|MACHINE=%m|VOLUME=%S full_audit:failure = connect disconnect full_audit:success = connect disconnect opendir mkdir rmdir closedir open close read pread write pwrite sendfile rename unlink chmod fchmod chown fchown chdir ftruncate lock symlink readlink link mknod full_audit:facility = local5 full_audit:priority = notice [profiles] comment = Network Profiles Share path = /mnt/storage/profiles profile acls = Yes browseable = No create mask = 0644 force create mode = 0660 force directory mode = 0770 read only = No [netlogon] comment = Network Netlogon Share path = /usr/local/samba/var/locks/sysvol/facility.local/scripts read only = No guest ok = No write ok = Yes browseable = No [sysvol] path = /usr/local/samba/var/locks/sysvol read only = No browseable = No write ok = Yes [printers] comment = All Printers path = /var/spool/samba create mask = 0700 browseable = yes guest ok = no printable = yes create mode=0700 write list = administrator "@Domain Admins" [print$] comment = Printer Drivers path = /mnt/storage/printer_drivers invalid users = qwerty valid users = @"Domain Users" admin users = @"Domain Admins" write list = root administrator writeable = Yes read only = Yes browseable = Yes guest ok = Yes create mask = 0660 create mask = 0644 force create mode = 0660 force directory mode = 0770 directory mask = 0755 acl_xattr:ignore system acl = yes
Any idea how i can identify the 'extra' processes without shutting down samba and looking what's left? ;) One of the servers just had more than 50 running smbd processes, while the other ones were all < 20. So i stopped samba using /etc/init.d/sernet-samba-ad (a leftover from we were using the free sernet packages). I now have 54 "stuck" smbd processes. "samba-tool processes" shows an empty list. One of the processes is running as user 3001439; the other ones as root. The processes are not consuming CPU: top shows 0.0% and the perf command example from top the flamegraph site says 'The perf.data file has no samples!'. /proc/[pid]/comm on all of them contains 'smbd'. "smbcontrol <pid> pool-usage" hangs for a while and then reports "No replies received". i'll attach: - "ls -l /proc/[pid]/fd" output per process - netstat -avp output - log files (smb.conf currently has no log level specified) - gdb bt full output (sorry, no debug symbols yet) I'm going to increase the log levels and compile with debug symbols. These are primary schools and they are closed for the weekend now, so i'll leave this specific server in this state for now, in case someone comes up with other idea on how to get more information out of it.
Created attachment 12468 [details] netstat -avp output on server 'juliana'
Created attachment 12469 [details] gdb "bt full" output of stuck smbd processes on server 'juliana'
Andrew, you mentioned "mark as private if need be". How do i do this? The log files and open file descriptors contain some private data like user and file names.
Created attachment 12470 [details] revert gencache of remote arch Can you try this patch? I've only checked that it doesn't cause outright failure in a testenv, but I think it might help. It reverts the addition of a gencache get and set during logon and logoff. The backtraces clearly show that we are blocked trying to operate on gencache.tdb, mostly adding or removing this cache item. This won't be the real bug, but it might buy you time. There is at least one process locked up in gencache_stablise(), but that won't be as likely to swamp the system. I'll include next a patch to remove mutexes from gencache.tdb so we can try and find who is holdling the lock on the whole DB.
Created attachment 12471 [details] remove muxtex suppot from gencache This larger hammer may allow us to use OS tools to work out who is currently holding the lock
I don't think the logs will show much, as once we get stuck in a blocking log, we don't do anything until we get it, including log :-) If you don't see a line like this at the bottom of the add an attachment screen: Privacy: Make attachment and comment private (visible only to members of the samba-devel group) Then just mail it to me and I'll look at it and upload with that ticked (as I have more privileges)
Re-assigning as file server bug, as I'm pretty sure the issue is not AD DC specific (as most of the AD DC can't even call gencache, and it is smbd processes blocked from accepting connections or terminating).
(In reply to Sander Plas from comment #30) BTW, I consider a process stuck if it has no corresponding entry in smbstatus, and/or no active connection to a client. There are helper processes that are like that, but we can quickly discount them, the backtraces you gave clearly show the issue being in the main client-connected process.
I get this with your second patch: [1693/3898] Compiling source3/lib/server_contexts.c ../source3/lib/gencache.c: In function ‘gencache_init’: ../source3/lib/gencache.c:126:11: error: too few arguments to function ‘tdb_wrap_open’ open_flags, 0644); ^ In file included from ../source3/lib/gencache.c:28:0: ../lib/tdb_wrap/tdb_wrap.h:39:18: note: declared here struct tdb_wrap *tdb_wrap_open(TALLOC_CTX *mem_ctx, ^ Waf: Leaving directory `/root/samba-4.5.0/bin' Build failed: -> task failed (err #1): {task: cc gencache.c -> gencache_58.o} make: *** [all] Error 1
Created attachment 12472 [details] remove mutex support from gencache Sorry about the bung patch. This one is what I built with, but I gave you the wrong file.
Ok, all servers except the one with the 54 stuck smbd processes are now running with debug symbols, your 2 patches and log level 10. I don't expect anything to happen soon as there won't be many users until monday. I can't seem to get the per-process logging to work though - this: log file = /var/log/samba/log.%p just results in a log file called "/var/log/samba/log.%p". Tried %d too, same result. Am i doing something wrong?
(In reply to Sander Plas from comment #41) Sorry, use %d to get the PID in the log file. It will still produce a %d for the 'samba' processes, but the smbd tasks will use it, and that is where it matters. If you are able to help us by running with both patches on some servers, only the gencache patch on others and only the mutex patch on others still, that may be more informative, as I strongly suspect that problem will 'go away' enough with the both that we never really understand it. Naturally you also have a production env to run, so make your judgements accordingly. I'll upload your logs as private in a moment.
As a quick look this looks like a deadlock problem. 12831.bt has: #2 0x00007f86b2984b42 in __pthread_mutex_lock_full (mutex=0x7f86b2fc80a8) at ../nptl/pthread_mutex_lock.c:258 result = <optimized out> oldval = <optimized out> id = 12831 __PRETTY_FUNCTION__ = "__pthread_mutex_lock_full" #3 0x00007f86ac810f04 in allrecord_mutex_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #4 0x00007f86ac8110fb in tdb_mutex_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #5 0x00007f86ac805e23 in fcntl_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #6 0x00007f86ac805fb4 in tdb_brlock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #7 0x00007f86ac8064ef in tdb_nest_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #8 0x00007f86ac80673b in tdb_lock_list () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #9 0x00007f86ac8067d8 in tdb_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #10 0x00007f86ac80353e in tdb_find_lock_hash () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #11 0x00007f86ac8038c6 in tdb_parse_record () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #12 0x00007f86afc8c62d in gencache_set_data_blob () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #13 0x00007f86afc8d687 in gencache_set () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #14 0x00007f86afc8cb5d in gencache_parse () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #15 0x00007f86afc8c7bc in gencache_del () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #16 0x00007f86afc71175 in remote_arch_cache_delete () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #17 0x00007f86b22c1ff4 in smbd_smb2_logoff_shutdown_done () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #18 0x00007f86b1703377 in _tevent_req_notify_callback () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #19 0x00007f86b170344a in tevent_req_finish () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #20 0x00007f86b170356f in tevent_req_trigger () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #21 0x00007f86b170278c in tevent_common_loop_immediate () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #22 0x00007f86afc8e10f in run_events_poll () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #23 0x00007f86afc8e783 in s3_event_loop_once () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #24 0x00007f86b17018d3 in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #25 0x00007f86b1701b1d in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #26 0x00007f86b1701be8 in _tevent_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #27 0x00007f86b229eaa3 in smbd_process () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #28 0x00007f86b2dcf0be in smbd_accept_connection () No symbol table info available. #29 0x00007f86afc8e608 in run_events_poll () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #30 0x00007f86afc8e894 in s3_event_loop_once () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #31 0x00007f86b17018d3 in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #32 0x00007f86b1701b1d in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #33 0x00007f86b1701be8 in _tevent_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #34 0x00007f86b2dcfe35 in smbd_parent_loop () all the others seem to be (note the difference in stack trace, the below is doing a chainlock, the above is not: #2 0x00007f86b2984b42 in __pthread_mutex_lock_full (mutex=0x7f86b2fc80a8) at ../nptl/pthread_mutex_lock.c:258 result = <optimized out> oldval = <optimized out> id = 26202 __PRETTY_FUNCTION__ = "__pthread_mutex_lock_full" #3 0x00007f86ac810f04 in allrecord_mutex_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #4 0x00007f86ac8110fb in tdb_mutex_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #5 0x00007f86ac805e23 in fcntl_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #6 0x00007f86ac805fb4 in tdb_brlock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #7 0x00007f86ac8064ef in tdb_nest_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #8 0x00007f86ac80673b in tdb_lock_list () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #9 0x00007f86ac8067d8 in tdb_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #10 0x00007f86ac8071d3 in tdb_chainlock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1 No symbol table info available. #11 0x00007f86afc8ca8e in gencache_parse () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #12 0x00007f86afc70f5e in remote_arch_cache_get () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #13 0x00007f86afc710df in remote_arch_cache_update () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #14 0x00007f86b22bcf26 in smbd_smb2_request_process_negprot () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #15 0x00007f86b22b7e79 in smbd_smb2_request_dispatch () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #16 0x00007f86b22bb118 in smbd_smb2_process_negprot () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #17 0x00007f86b2298b9a in process_smb () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #18 0x00007f86b229a22f in smbd_server_connection_read_handler () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #19 0x00007f86b229a310 in smbd_server_connection_handler () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #20 0x00007f86afc8e608 in run_events_poll () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #21 0x00007f86afc8e894 in s3_event_loop_once () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #22 0x00007f86b17018d3 in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #23 0x00007f86b1701b1d in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #24 0x00007f86b1701be8 in _tevent_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #25 0x00007f86b229eaa3 in smbd_process () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so No symbol table info available. #26 0x00007f86b2dcf0be in smbd_accept_connection () No symbol table info available. #27 0x00007f86afc8e608 in run_events_poll () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #28 0x00007f86afc8e894 in s3_event_loop_once () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0 No symbol table info available. #29 0x00007f86b17018d3 in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #30 0x00007f86b1701b1d in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #31 0x00007f86b1701be8 in _tevent_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0 No symbol table info available. #32 0x00007f86b2dcfe35 in smbd_parent_loop () No symbol table info available. #33 0x00007f86b2dd1800 in main ()
Question: Are you running the new gpgme code for password sync? I see a core file in there. Potentially if mutexes are not as robust as we expect, a crash happens while holding the mutex. Core was generated by `/usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground'. Program terminated with signal SIGABRT, Aborted. #0 0x00007fa9303e4c37 in ?? () Can you: - set max log size = 0 (so we don't rotate the log) - get me the gdb backtrace from the core file Finally, a tip: when sending large logs, 'tar --xz -cf logs.tar.xz logs/' makes really, really tiny files because it finds all the duplicates much better than zip. I'll upload the private logs in a moment.
The new gpgme code needs to be enabled explicitly, right? In that case, no. This is what i use; all the explicit paths are there to make it compatible with the old sernet packages. ./configure --sbindir=/usr/sbin --bindir=/usr/bin --with-configdir=/etc/samba --with-logfilebase=/var/log/samba --libdir=/usr/lib/x86_64-linux-gnu --with-modulesdir=/usr/lib/x86_64-linux-gnu/samba --with-lockdir=/var/cache/samba --with-statedir=/var/lib/samba --with-cachedir=/var/cache/samba --with-piddir=/var/run/samba --with-privatedir=/var/lib/samba/private --enable-debug I'm now running 3 servers with just the first patch, 3 with just the other and 4 with both. Can i delete logs for pid's that are no longer running? Some of the servers are low on disk space.
(In reply to Andrew Bartlett from comment #44) The core is from 2016-07-03, so it must be from a previous version of samba. The oldest smbd binary i can find in our backups of this server is from 2016-07-08. "bt full" (same output with the smbd binary from the backup and the current one) : (gdb) bt full Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7ffef3733548: #0 0x00007fa9303e4c37 in ?? () No symbol table info available. Cannot access memory at address 0x7ffef3733548
(In reply to Sander Plas from comment #46) Yes, I think it is reasonable to delete logs of PIDs that exit successfully. I also think Jeremy is onto something regarding a deadlock - I've not tried to get my head into tdb locking enough to understand why we can't make progress from here, but I know this: Many Samba Team members will be in a lab at SNIA SDC next Monday US time, and you now have the attention of those who can sit together and fix this. I really appreciate your assistance with getting us the debug info we need, we simply were not able to make progress without it.
(In reply to Andrew Bartlett from comment #44) The SIGABRT in the core file could be from: https://bugzilla.redhat.com/show_bug.cgi?id=1375973
This morning at 8 (Netherlands time) we were called by 4 of the schools who said that they were having the problem. Strangely, none of them had the typical "stuck smbd processes" symptom. One of the servers, running both patches, was just completely unresponsive. They are going to powercycle it. Another one, only running "patch 1" aka the gencache patch, did not have many smbd processes running and all smbd and samba processes exited without any problems using the init.d script. The third server (vondel), only had a stuck samba (not smbd) process. Apparently this one was already stuck for days since saturday when i recompiled & restarted samba on this server. I only checked for stuck smbd processes then. GDB complains about the deleted binary. This is another one of those "Python Exception <class 'gdb.MemoryError'> Cannot access memory at address" cases. The last server (cray-dep), running both patches, neither had any stuck smbd processes, but it did have one stuck samba process. I'll attach logs for vondel and cray-dep.
Created attachment 12478 [details] cray-dep backtrace
Created attachment 12479 [details] cray-dep netstat -avp
Created attachment 12480 [details] cray-dep file descriptors
Created attachment 12481 [details] cray-dep samba-tool processes
Created attachment 12482 [details] vondel backtrace
Created attachment 12483 [details] vondel netstat -avp
Created attachment 12484 [details] vondel file descriptors
Created attachment 12485 [details] vondel samba-tool processes
Another crash on server 'juliana'. looks like a 'classic' one with 32 stuck smbd processes. This one is still running without any patches, but with log level 10 and logging per pid.
Created attachment 12486 [details] netstat -avp output on server 'juliana' ; 2nd crash
Created attachment 12487 [details] gdb "bt full" output of stuck smbd processes on server 'juliana' ; 2nd crash
(In reply to Sander Plas from comment #50) > The last server (cray-dep), running both patches, neither had any stuck smbd > processes, but it did have one stuck samba process. This server has a segfault in a samba process. Can you set: panic action = path/to/gdb_backtrace %d This is in selftest/gdb_backtrace in the source tree. As for some reason your build doesn't include the internal backtrace handler (and gdb is better anyway). Other than that, if you can get me level 10 logs on juliana? The backtrace on cray-dep is truly weird: A failure getting a lock to allocate memory? Is there any chance of a full backtrace, as this one is stuck at only the first page. I'm also quite concerned about the impact of all this on production, and I'm wondering if the 'stuck samba process' thing is a different bug entirely. Is that process busy, or has it consumed a lot of memory?
Ok, added "panic action" setting to all servers. I'm sorry for the incomplete backtrace on cray-dep. Unfortunately i already killed the process, so no chance to get a full backtrace. The process was not consuming a lot of cpu or memory. The juliana logs are huge - still 14 GB after removing all the non needed stuff. The tar.xz job is running, but it will probably take a while.
Two days I've been using the "Version 4.5.1-GIT-dbbf8dc" version. I encounter problems.
Another stuck 'samba' process. this one is on the server 'stopoz-pdc', which is a central server in a data center. It has a remote desktop server as a client. samba-tool processes = empty /proc/31194/comm = samba smbcontrol 31194 pool-usage = No replies received I'll attach more information. I've sent the private juliana logs to Andrew and will send the stopoz-pdc ones too. I don't know how soon he will be able to upload them though, so if another team member wants to have them, just let me know and i'll email them.
Created attachment 12490 [details] stopoz-pdc samba process backtrace
Created attachment 12491 [details] stopoz-pdc samba process file descriptors
(In reply to Sander Plas from comment #66) This looks just like a busy process that is trying to print a lot of replication traffic out into the logs. Unless it is really stuck in exactly that lock for a long time, I'm just going to assume that it is the load of level 10 and some not-so-efficient debugging code. It is probably time to turn the log level back down, I think we have what we need. The folks at SNIA SDC will be getting into the lab soon, so I hope they can make some sense of things. The backtraces are the most important part. Thanks!
Created attachment 12494 [details] Patch
Comment on attachment 12494 [details] Patch 5 minutes looking at it in the plugfest, Volker found the lock order violation :-). Pushing to master now - will create backports for the bug. Longer answer - inside gencache_set() we take a chainlock on the record, but inside gencache_set_data_blob() called from gencache_set() on every 100 writes we call gencache_stabilize(). Inside gencache_stabilize() the old code called : tdb_lockall() - needs to be tdb_lockall_nonblock() to prevent lock ordering violations (we alredy hold the chainlock). Checking into the back traces from the entire set of processes Volker found one process inside gencache_stabilize() - which gave it away !
Wow, that's great! I'm going to try the patch and let you know the results. THANK YOU Volker, Jeremy and Andrew!
Created attachment 12499 [details] git-am fix for 4.5.next, 4.4.next, 4.3.next. OK, here's the cherry-pick from what went into master. Applies cleanly to all supported branches !
Comment on attachment 12499 [details] git-am fix for 4.5.next, 4.4.next, 4.3.next. This patch is good, but before we mark the bug fixed, I'd prefer the bug reporter to test.
thanks for your attention. I'm trying all the samba-test(git pull) suite regularly. now i 'm testing the patch. If there are problems , I will inform you.. # samba -V Version 4.5.1-GIT-dbbf8dc Kind Regards...
24 hours without a single crash now. This is looking really good, although i must admit that we did have one day without any crashes last week too. So, if it stays up for another full day i'd be pretty confident that the bug really is fixed.
samba -V Version 4.5.1-GIT-dbbf8dc This patch ' after the body was samba_dnsupdate --verbose --all-names dns_tkey_gssnegotiate: TKEY is unacceptable dns_tkey_gssnegotiate: TKEY is unacceptable dns_tkey_gssnegotiate: TKEY is unacceptable
(In reply to baris tombul from comment #76) You do not have my previous message . sorry. false flag.
Another day without any crashes. I think the issue has been solved!
Confirmation from bug reporter. Karolin please push to 4.5.next, 4.4.next, 4.3.next. Thanks ! Jeremy.
(In reply to Jeremy Allison from comment #79) Pushed to autobuild-v4-{5,4,3}-test.
(In reply to Karolin Seeger from comment #80) Pushed to all branches. Closing out bug report. Thanks!
Giving this a title was can find again. The issue is traced to a deadlock in gencache_stabilize()