Bug 12045 - gencache: Bail out of stabilize if we can not get the allrecord lock.
Summary: gencache: Bail out of stabilize if we can not get the allrecord lock.
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.5.0rc1
Hardware: x64 Linux
: P5 critical (vote)
Target Milestone: 4.5
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
: 12267 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-07-30 07:10 UTC by baris tombul
Modified: 2017-06-01 20:15 UTC (History)
8 users (show)

See Also:


Attachments
netstat (35.80 KB, text/plain)
2016-08-10 16:31 UTC, baris tombul
no flags Details
smbcontrol (2.95 MB, text/plain)
2016-08-10 16:33 UTC, baris tombul
no flags Details
smbcontrol_last_error (3.66 MB, text/plain)
2016-08-12 06:19 UTC, baris tombul
no flags Details
/var/log/messages (102.12 KB, text/plain)
2016-09-14 07:47 UTC, Dmitry Vlasov
no flags Details
netstat -avp output on server 'juliana' (23.16 KB, text/plain)
2016-09-16 18:04 UTC, Sander Plas
no flags Details
gdb "bt full" output of stuck smbd processes on server 'juliana' (217.32 KB, application/zip)
2016-09-16 18:06 UTC, Sander Plas
no flags Details
revert gencache of remote arch (2.62 KB, patch)
2016-09-16 18:28 UTC, Andrew Bartlett
no flags Details
remove muxtex suppot from gencache (744 bytes, patch)
2016-09-16 18:31 UTC, Andrew Bartlett
no flags Details
remove mutex support from gencache (840 bytes, patch)
2016-09-16 19:05 UTC, Andrew Bartlett
no flags Details
cray-dep backtrace (4.01 KB, text/plain)
2016-09-19 07:23 UTC, Sander Plas
no flags Details
cray-dep netstat -avp (18.50 KB, text/plain)
2016-09-19 07:23 UTC, Sander Plas
no flags Details
cray-dep file descriptors (3.89 KB, text/plain)
2016-09-19 07:24 UTC, Sander Plas
no flags Details
cray-dep samba-tool processes (771 bytes, text/plain)
2016-09-19 07:24 UTC, Sander Plas
no flags Details
vondel backtrace (188 bytes, text/plain)
2016-09-19 07:25 UTC, Sander Plas
no flags Details
vondel netstat -avp (32.63 KB, text/plain)
2016-09-19 07:25 UTC, Sander Plas
no flags Details
vondel file descriptors (3.50 KB, text/plain)
2016-09-19 07:25 UTC, Sander Plas
no flags Details
vondel samba-tool processes (1.11 KB, text/plain)
2016-09-19 07:26 UTC, Sander Plas
no flags Details
netstat -avp output on server 'juliana' ; 2nd crash (20.81 KB, text/plain)
2016-09-19 09:33 UTC, Sander Plas
no flags Details
gdb "bt full" output of stuck smbd processes on server 'juliana' ; 2nd crash (6.42 KB, application/x-xz)
2016-09-19 09:34 UTC, Sander Plas
no flags Details
stopoz-pdc samba process backtrace (60.71 KB, text/plain)
2016-09-19 14:00 UTC, Sander Plas
no flags Details
stopoz-pdc samba process file descriptors (4.00 KB, text/plain)
2016-09-19 14:01 UTC, Sander Plas
no flags Details
Patch (847 bytes, patch)
2016-09-19 21:33 UTC, Volker Lendecke
jra: review+
Details
git-am fix for 4.5.next, 4.4.next, 4.3.next. (1.06 KB, patch)
2016-09-20 15:55 UTC, Jeremy Allison
vl: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description baris tombul 2016-07-30 07:10:58 UTC
samba crashed:
very critical.

[2016/07/30 07:00:47.601811,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:47.607554,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory            
[2016/07/30 07:00:47.612709,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:47.617773,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:47.703414,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:47.708867,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:47.731963,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:50.494388,  1] ../source3/lib/messages.c:976(mess_parent_dgm_cleanup_done)
  messaging dgm cleanup job ended with NT_STATUS_NO_MEMORY
[2016/07/30 07:00:57.180022,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory            
[2016/07/30 07:00:58.086243,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:58.093154,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:58.099855,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:58.106786,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:58.188456,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory
[2016/07/30 07:00:58.196342,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)





ps -auxw 

...
......
......
root     58944  0.0  0.2 699252 18388 ?        S    07:49   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     58945  0.0  0.1 693020 15724 ?        S    07:49   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground    
root     58946  0.0  0.2 699252 18388 ?        S    07:49   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     58948  0.0  0.2 699252 18388 ?        S    07:50   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     58950  0.0  0.2 699252 18388 ?        S    07:50   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     58953  0.0  0.1 693020 15720 ?        S    07:50   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     58954  0.0  0.2 699252 18548 ?        S    07:50   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59082  0.0  0.2 699252 18648 ?        S    07:51   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59084  0.0  0.2 699252 18648 ?        S    07:51   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59085  0.0  0.1 693020 15740 ?        S    07:51   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59086  0.0  0.2 699252 18648 ?        S    07:51   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59087  0.0  0.1 693020 15748 ?        S    07:52   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59088  0.0  0.2 699252 18648 ?        S    07:52   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59090  0.0  0.2 699252 18648 ?        S    07:52   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59091  0.0  0.1 693020 15732 ?        S    07:52   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59092  0.0  0.2 699252 18648 ?        S    07:52   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59093  0.0  0.2 699252 18648 ?        S    07:52   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59095  0.0  0.2 699252 18648 ?        S    07:53   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59096  0.0  0.1 693020 15752 ?        S    07:53   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59098  0.0  0.1 693020 15732 ?        S    07:53   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59099  0.0  0.2 699252 18644 ?        S    07:53   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59227  0.0  0.2 699252 18652 ?        S    07:54   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59229  0.0  0.2 699252 18652 ?        S    07:54   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59230  0.0  0.1 693020 15760 ?        S    07:54   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59231  0.0  0.2 699252 18652 ?        S    07:54   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59232  0.0  0.2 699252 18652 ?        S    07:55   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59233  0.0  0.1 693020 15748 ?        S    07:55   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59234  0.0  0.2 699252 18652 ?        S    07:55   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59236  0.0  0.1 693020 15736 ?        S    07:55   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59238  0.0  0.2 699252 18656 ?        S    07:55   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59239  0.0  0.2 699252 18652 ?        S    07:55   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59240  0.0  0.2 699252 18656 ?        S    07:56   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59241  0.0  0.2 699252 18656 ?        S    07:56   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59243  0.0  0.1 693020 15744 ?        S    07:56   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59244  0.0  0.2 699252 18652 ?        S    07:56   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59245  0.0  0.1 693020 15752 ?        S    07:56   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59246  0.0  0.2 699252 18664 ?        S    07:56   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59247  0.0  0.2 699252 18664 ?        S    07:56   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59374  0.0  0.0 107896   592 ?        S    07:57   0:00 sleep 180
root     59376  0.0  0.2 699252 20204 ?        S    07:57   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59377  0.0  0.1 693020 16276 ?        S    07:57   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59378  0.0  0.2 699252 20216 ?        S    07:57   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59380  0.0  0.1 693020 16268 ?        S    07:57   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59381  0.0  0.2 699252 20220 ?        S    07:58   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59384  0.0  0.2 699252 20200 ?        S    07:58   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59385  0.6  0.0 157240  1932 pts/1    R+   07:58   0:00 ps -auxw
root     59386  0.0  0.2 699252 20208 ?        S    07:58   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59388  0.0  0.1 693020 16292 ?        S    07:58   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59389  0.0  0.2 699252 20196 ?        S    07:58   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59390  0.0  0.2 699252 20224 ?        S    07:59   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59391  0.0  0.2 699252 20216 ?        S    07:59   0:00 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     59392  0.0  0.0 107896   620 ?        S    07:59   0:00 sleep 60
..
.....
....
Comment 1 Andrew Bartlett 2016-07-30 09:08:26 UTC
Can you please confirm:

smbd -V 

for each PID of a samba and smbd process:

smbcontrol pool-usage <samba pid>

Then confirm what all the connections keeping the smbd processes alive are with:
netstat -avp

Finally, details about your AD domain and any other info on what may be causing this issue.

Thanks!
Comment 2 baris tombul 2016-07-30 09:32:06 UTC
(In reply to Andrew Bartlett from comment #1)

[root@ ~]# smbd -V
Version 4.5.0rc1


netstat -avp


[root@mems ~]# netstat -avp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 localhost:netbios-ns    0.0.0.0:*               LISTEN      906/smbd            
tcp        0      0 mems:netbios-ns         0.0.0.0:*               LISTEN      906/smbd            
tcp        0      0 localhost:netbios-dgm   0.0.0.0:*               LISTEN      906/smbd            
tcp        0      0 mems:netbios-dgm        0.0.0.0:*               LISTEN      906/smbd            
tcp        0      0 mems:nameserver         0.0.0.0:*               LISTEN      900/samba           
tcp        0      0 localhost:nameserver    0.0.0.0:*               LISTEN      900/samba           
tcp        0      0 localhost:netbios-ssn   0.0.0.0:*               LISTEN      906/smbd            
tcp        0      0 mems:netbios-ssn        0.0.0.0:*               LISTEN      906/smbd            
tcp        0      0 localhost:dyna-access   0.0.0.0:*               LISTEN      1058/clamd          
tcp        0      0 mems:kpasswd            0.0.0.0:*               LISTEN      903/samba           
tcp        0      0 localhost:kpasswd       0.0.0.0:*               LISTEN      903/samba           
tcp        0      0 mems:domain             0.0.0.0:*               LISTEN      1223/named          
tcp        0      0 localhost:domain        0.0.0.0:*               LISTEN      1223/named          
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN      796/sshd            
tcp        0      0 0.0.0.0:ipp             0.0.0.0:*               LISTEN      791/cupsd           
tcp        0      0 mems:kerberos           0.0.0.0:*               LISTEN      903/samba           
tcp        0      0 localhost:kerberos      0.0.0.0:*               LISTEN      903/samba           
tcp        0      0 localhost:rndc          0.0.0.0:*               LISTEN      1223/named          
tcp        0      0 mems:ldaps              0.0.0.0:*               LISTEN      901/samba           
tcp        0      0 localhost:ldaps         0.0.0.0:*               LISTEN      901/samba           
tcp        0      0 localhost:microsoft-ds  0.0.0.0:*               LISTEN      906/smbd            
tcp        0      0 mems:microsoft-ds       0.0.0.0:*               LISTEN      906/smbd            
tcp        0      0 mems:1024               0.0.0.0:*               LISTEN      898/samba           
tcp        0      0 localhost:1024          0.0.0.0:*               LISTEN      898/samba           
tcp        0      0 mems:msft-gc            0.0.0.0:*               LISTEN      901/samba           
tcp        0      0 localhost:msft-gc       0.0.0.0:*               LISTEN      901/samba           
tcp        0      0 mems:msft-gc-ssl        0.0.0.0:*               LISTEN      901/samba           
tcp        0      0 mems:ldap               0.0.0.0:*               LISTEN      901/samba           
tcp        0      0 localhost:msft-gc-ssl   0.0.0.0:*               LISTEN      901/samba           
tcp        0      0 localhost:ldap          0.0.0.0:*               LISTEN      901/samba           
tcp        0      0 mems:epmap              0.0.0.0:*               LISTEN      898/samba           
tcp        0      0 localhost:epmap         0.0.0.0:*               LISTEN      898/samba           
tcp        0      0 mems:ldap               10.0.20.15:35386        ESTABLISHED 901/samba           
tcp        0      0 mems:ssh                10.0.20.108:60984       ESTABLISHED 3604/sshd: root@pts 
tcp        0      0 mems:microsoft-ds       10.0.20.15:47574        ESTABLISHED 1882/smbd           
tcp        0      0 mems:microsoft-ds       10.0.20.81:57063        ESTABLISHED 6628/smbd           
tcp        0      0 mems:microsoft-ds       10.0.20.62:60845        ESTABLISHED 6629/smbd           
tcp        0      0 mems:1024               10.0.20.15:37172        ESTABLISHED 898/samba           
tcp        0      0 mems:55552              10.0.20.9:iscsi-target  ESTABLISHED 830/iscsid          
tcp6       0      0 localhost:netbios-ns    [::]:*                  LISTEN      906/smbd            
tcp6       0      0 localhost:netbios-dgm   [::]:*                  LISTEN      906/smbd            
tcp6       0      0 localhost:netbios-ssn   [::]:*                  LISTEN      906/smbd            
tcp6       0      0 [::]:http               [::]:*                  LISTEN      1057/httpd          
tcp6       0      0 localhost:kpasswd       [::]:*                  LISTEN      903/samba           
tcp6       0      0 [::]:ftp                [::]:*                  LISTEN      820/vsftpd          
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN      796/sshd            
tcp6       0      0 [::]:ipp                [::]:*                  LISTEN      791/cupsd           
tcp6       0      0 localhost:kerberos      [::]:*                  LISTEN      903/samba           
tcp6       0      0 [::]:https              [::]:*                  LISTEN      1057/httpd          
tcp6       0      0 localhost:ldaps         [::]:*                  LISTEN      901/samba           
tcp6       0      0 localhost:microsoft-ds  [::]:*                  LISTEN      906/smbd            
tcp6       0      0 localhost:1024          [::]:*                  LISTEN      898/samba           
tcp6       0      0 localhost:msft-gc       [::]:*                  LISTEN      901/samba           
tcp6       0      0 localhost:msft-gc-ssl   [::]:*                  LISTEN      901/samba           
tcp6       0      0 localhost:ldap          [::]:*                  LISTEN      901/samba           
tcp6       0      0 localhost:epmap         [::]:*                  LISTEN      898/samba           
netstat: no support for `AF INET (sctp)' on this system.
netstat: no support for `AF INET (sctp)' on this system.
udp        0      0 0.0.0.0:55789           0.0.0.0:*                           534/rsyslogd        
udp        0      0 0.0.0.0:59935           0.0.0.0:*                           545/avahi-daemon: r 
udp        0      0 mems:domain             0.0.0.0:*                           1223/named          
udp        0      0 localhost:domain        0.0.0.0:*                           1223/named          
udp        0      0 mems:kerberos           0.0.0.0:*                           903/samba           
udp        0      0 localhost:kerberos      0.0.0.0:*                           903/samba           
udp        0      0 mems:ntp                0.0.0.0:*                           5166/ntpd           
udp        0      0 localhost:ntp           0.0.0.0:*                           5166/ntpd           
udp        0      0 0.0.0.0:ntp             0.0.0.0:*                           5166/ntpd           
udp        0      0 mems:netbios-ns         0.0.0.0:*                           899/samba           
udp        0      0 10.0.20.255:netbios-ns  0.0.0.0:*                           899/samba           
udp        0      0 localhost:netbios-ns    0.0.0.0:*                           899/samba           
udp        0      0 127.255.255.:netbios-ns 0.0.0.0:*                           899/samba           
udp   132736      0 10.0.20.255:netbios-ns  0.0.0.0:*                           892/nmbd            
udp        0      0 mems:netbios-ns         0.0.0.0:*                           892/nmbd            
udp   132736      0 0.0.0.0:netbios-ns      0.0.0.0:*                           892/nmbd            
udp        0      0 mems:netbios-dgm        0.0.0.0:*                           899/samba           
udp        0      0 10.0.20.255:netbios-dgm 0.0.0.0:*                           899/samba           
udp        0      0 localhost:netbios-dgm   0.0.0.0:*                           899/samba           
udp        0      0 127.255.255:netbios-dgm 0.0.0.0:*                           899/samba           
udp   117504      0 10.0.20.255:netbios-dgm 0.0.0.0:*                           892/nmbd            
udp        0      0 mems:netbios-dgm        0.0.0.0:*                           892/nmbd            
udp   119680      0 0.0.0.0:netbios-dgm     0.0.0.0:*                           892/nmbd            
udp        0      0 mems:ldap               0.0.0.0:*                           902/samba           
udp        0      0 localhost:ldap          0.0.0.0:*                           902/samba           
udp        0      0 mems:kpasswd            0.0.0.0:*                           903/samba           
udp        0      0 localhost:kpasswd       0.0.0.0:*                           903/samba           
udp        0      0 0.0.0.0:mdns            0.0.0.0:*                           545/avahi-daemon: r 
udp6       0      0 localhost:kerberos      [::]:*                              903/samba           
udp6       0      0 fe80::20c:29ff:fe1d:ntp [::]:*                              5166/ntpd           
udp6       0      0 localhost:ntp           [::]:*                              5166/ntpd           
udp6       0      0 [::]:ntp                [::]:*                              5166/ntpd           
udp6       0      0 localhost:ldap          [::]:*                              902/samba           
udp6       0      0 localhost:kpasswd       [::]:*                              903/samba           
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ]         DGRAM                    67753    6288/smbd            /usr/local/samba/private/msg.sock/6288
unix  2      [ ]         DGRAM                    61455    5637/smbd            /usr/local/samba/private/msg.sock/5637
unix  2      [ ]         DGRAM                    60979    5798/smbd            /usr/local/samba/private/msg.sock/5798
unix  2      [ ]         DGRAM                    60781    5641/smbd            /usr/local/samba/private/msg.sock/5641
unix  2      [ ]         DGRAM                    61550    5644/smbd            /usr/local/samba/private/msg.sock/5644
unix  2      [ ]         DGRAM                    15760    892/nmbd             /usr/local/samba/private/msg.sock/892
unix  2      [ ACC ]     STREAM     LISTENING     14836    898/samba            /usr/local/samba/var/run/ncalrpc/np/browser
unix  2      [ ]         DGRAM                    14804    898/samba            /usr/local/samba/private/msg.sock/898
unix  2      [ ]         DGRAM                    14805    901/samba            /usr/local/samba/private/msg.sock/901
unix  2      [ ]         DGRAM                    14806    902/samba            /usr/local/samba/private/msg.sock/902
unix  2      [ ]         DGRAM                    14819    903/samba            /usr/local/samba/private/msg.sock/903
unix  2      [ ]         DGRAM                    14252    897/samba            /usr/local/samba/private/msg.sock/897
unix  2      [ ACC ]     STREAM     LISTENING     14837    898/samba            /usr/local/samba/var/run/ncalrpc/np/unixinfo
unix  2      [ ]         DGRAM                    14829    905/samba            /usr/local/samba/private/msg.sock/905
unix  2      [ ACC ]     STREAM     LISTENING     14838    898/samba            /usr/local/samba/var/run/ncalrpc/np/protected_storage
unix  2      [ ]         DGRAM                    14833    910/samba            /usr/local/samba/private/msg.sock/910
unix  2      [ ACC ]     STREAM     LISTENING     14869    898/samba            /usr/local/samba/var/run/ncalrpc/np/epmapper
unix  2      [ ]         DGRAM                    14258    904/samba            /usr/local/samba/private/msg.sock/904
unix  2      [ ]         DGRAM                    15904    908/winbindd         /usr/local/samba/private/msg.sock/908
unix  2      [ ]         DGRAM                    15834    899/samba            /usr/local/samba/private/msg.sock/899
unix  2      [ ]         DGRAM                    16870    900/samba            /usr/local/samba/private/msg.sock/900
unix  2      [ ]         DGRAM                    14259    907/samba            /usr/local/samba/private/msg.sock/907
unix  2      [ ]         DGRAM                    14260    909/samba            /usr/local/samba/private/msg.sock/909
unix  2      [ ]         DGRAM                    15916    906/smbd             /usr/local/samba/private/msg.sock/906
unix  2      [ ]         DGRAM                    14274    917/smbd             /usr/local/samba/private/msg.sock/917
unix  2      [ ]         DGRAM                    14892    918/smbd             /usr/local/samba/private/msg.sock/918
unix  2      [ ACC ]     STREAM     LISTENING     14261    907/samba            /usr/local/samba/var/lib/ntp_signd/socket
unix  2      [ ACC ]     STREAM     LISTENING     15826    892/nmbd             /usr/local/samba/var/run/nmbd/unexpected
unix  2      [ ]         DGRAM                    15941    919/winbindd         /usr/local/samba/private/msg.sock/919
unix  2      [ ]         DGRAM                    14275    920/smbd             /usr/local/samba/private/msg.sock/920
unix  2      [ ]         DGRAM                    62147    5953/smbd            /usr/local/samba/private/msg.sock/5953
unix  2      [ ACC ]     STREAM     LISTENING     15938    908/winbindd         /usr/local/samba/var/locks/winbindd_privileged/pipe
unix  2      [ ]         DGRAM                    17471    1051/winbindd        /usr/local/samba/private/msg.sock/1051
unix  2      [ ACC ]     STREAM     LISTENING     14919    901/samba            /usr/local/samba/private/ldap_priv/ldapi
unix  2      [ ]         DGRAM                    9765     1/systemd            /run/systemd/shutdownd
unix  2      [ ]         DGRAM                    27424    1882/smbd            /usr/local/samba/private/msg.sock/1882
unix  2      [ ]         DGRAM                    39       1/systemd            /run/systemd/notify
unix  2      [ ]         DGRAM                    61553    5645/smbd            /usr/local/samba/private/msg.sock/5645
unix  2      [ ]         DGRAM                    60755    5640/smbd            /usr/local/samba/private/msg.sock/5640
unix  2      [ ]         DGRAM                    60830    5649/smbd            /usr/local/samba/private/msg.sock/5649
unix  2      [ ]         DGRAM                    61524    5643/smbd            /usr/local/samba/private/msg.sock/5643
unix  2      [ ]         DGRAM                    61560    5650/smbd            /usr/local/samba/private/msg.sock/5650
unix  2      [ ]         DGRAM                    61414    6090/smbd            /usr/local/samba/private/msg.sock/6090
unix  2      [ ]         DGRAM                    60949    5780/smbd            /usr/local/samba/private/msg.sock/5780
unix  2      [ ]         DGRAM                    56726    5115/winbindd        /usr/local/samba/private/msg.sock/5115
unix  2      [ ]         DGRAM                    62482    6105/smbd            /usr/local/samba/private/msg.sock/6105
unix  2      [ ]         DGRAM                    60964    5793/smbd            /usr/local/samba/private/msg.sock/5793
unix  2      [ ACC ]     STREAM     LISTENING     14834    898/samba            /usr/local/samba/var/run/ncalrpc/np/dnsserver
unix  2      [ ]         DGRAM                    60957    5787/smbd            /usr/local/samba/private/msg.sock/5787
unix  2      [ ACC ]     STREAM     LISTENING     14835    898/samba            /usr/local/samba/var/run/ncalrpc/np/ntsvcs
unix  2      [ ACC ]     STREAM     LISTENING     14384    1/systemd            /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    62149    5955/smbd            /usr/local/samba/private/msg.sock/5955
unix  2      [ ]         DGRAM                    60961    5790/smbd            /usr/local/samba/private/msg.sock/5790
unix  2      [ ACC ]     STREAM     LISTENING     14839    898/samba            /usr/local/samba/var/run/ncalrpc/np/lsass
unix  2      [ ]         DGRAM                    67806    6310/smbd            /usr/local/samba/private/msg.sock/6310
unix  2      [ ACC ]     STREAM     LISTENING     14840    898/samba            /usr/local/samba/var/run/ncalrpc/np/lsarpc
unix  2      [ ]         DGRAM                    60984    5804/smbd            /usr/local/samba/private/msg.sock/5804
unix  2      [ ACC ]     STREAM     LISTENING     14841    898/samba            /usr/local/samba/var/run/ncalrpc/np/netlogon
unix  2      [ ]         DGRAM                    60987    5805/smbd            /usr/local/samba/private/msg.sock/5805
unix  2      [ ACC ]     STREAM     LISTENING     14842    898/samba            /usr/local/samba/var/run/ncalrpc/np/samr
unix  2      [ ACC ]     STREAM     LISTENING     14389    1/systemd            /var/run/cups/cups.sock
unix  2      [ ]         DGRAM                    63000    6458/smbd            /usr/local/samba/private/msg.sock/6458
unix  2      [ ACC ]     STREAM     LISTENING     14843    898/samba            /usr/local/samba/var/run/ncalrpc/np/rpcecho
unix  2      [ ]         DGRAM                    61224    5937/smbd            /usr/local/samba/private/msg.sock/5937
unix  2      [ ]         DGRAM                    62145    5952/smbd            /usr/local/samba/private/msg.sock/5952
unix  2      [ ]         DGRAM                    61228    5941/smbd            /usr/local/samba/private/msg.sock/5941
unix  2      [ ACC ]     STREAM     LISTENING     57       1/systemd            /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    61231    5943/smbd            /usr/local/samba/private/msg.sock/5943
unix  2      [ ACC ]     STREAM     LISTENING     14857    898/samba            /usr/local/samba/var/run/ncalrpc/np/wkssvc
unix  2      [ ]         DGRAM                    62573    6143/smbd            /usr/local/samba/private/msg.sock/6143
unix  5      [ ]         DGRAM                    60       1/systemd            /run/systemd/journal/socket
unix  2      [ ]         DGRAM                    62484    6106/smbd            /usr/local/samba/private/msg.sock/6106
unix  2      [ ]         DGRAM                    62332    6091/smbd            /usr/local/samba/private/msg.sock/6091
unix  24     [ ]         DGRAM                    62       1/systemd            /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     15681    869/python2          /var/run/fail2ban/fail2ban.sock
unix  2      [ ACC ]     STREAM     LISTENING     14387    1/systemd            @ISCSID_UIP_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     STREAM     LISTENING     20559    1058/clamd           /var/run/clamd.scan/clamd.sock
unix  2      [ ]         DGRAM                    62416    6141/smbd            /usr/local/samba/private/msg.sock/6141
unix  2      [ ACC ]     STREAM     LISTENING     14844    898/samba            /usr/local/samba/var/run/ncalrpc/DEFAULT
unix  2      [ ACC ]     STREAM     LISTENING     14858    898/samba            /usr/local/samba/var/run/ncalrpc/EPMAPPER
unix  2      [ ACC ]     STREAM     LISTENING     14918    901/samba            /usr/local/samba/private/ldapi
unix  2      [ ACC ]     STREAM     LISTENING     1918     1/systemd            /run/systemd/private
unix  2      [ ]         DGRAM                    14240    896/samba            /usr/local/samba/private/msg.sock/896
unix  2      [ ]         DGRAM                    62355    6107/smbd            /usr/local/samba/private/msg.sock/6107
unix  2      [ ACC ]     SEQPACKET  LISTENING     1941     1/systemd            /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     13721    537/NetworkManager   /var/run/NetworkManager/private
unix  2      [ ]         DGRAM                    62759    6298/smbd            /usr/local/samba/private/msg.sock/6298
unix  2      [ ACC ]     STREAM     LISTENING     12703    1/systemd            /var/run/avahi-daemon/socket
unix  2      [ ACC ]     STREAM     LISTENING     15937    908/winbindd         /usr/local/samba/var/run/winbindd/pipe
unix  2      [ ]         DGRAM                    67804    6308/smbd            /usr/local/samba/private/msg.sock/6308
unix  2      [ ]         DGRAM                    67766    6301/smbd            /usr/local/samba/private/msg.sock/6301
unix  2      [ ]         DGRAM                    62778    6307/smbd            /usr/local/samba/private/msg.sock/6307
unix  2      [ ]         DGRAM                    63022    6471/smbd            /usr/local/samba/private/msg.sock/6471
unix  2      [ ]         DGRAM                    62785    6312/smbd            /usr/local/samba/private/msg.sock/6312
unix  2      [ ]         DGRAM                    68085    6448/smbd            /usr/local/samba/private/msg.sock/6448
unix  2      [ ]         DGRAM                    68096    6462/smbd            /usr/local/samba/private/msg.sock/6462
unix  2      [ ]         DGRAM                    62989    6452/smbd            /usr/local/samba/private/msg.sock/6452
unix  2      [ ]         DGRAM                    69762    6609/smbd            /usr/local/samba/private/msg.sock/6609
unix  2      [ ]         DGRAM                    68100    6464/smbd            /usr/local/samba/private/msg.sock/6464
unix  2      [ ]         DGRAM                    63019    6469/smbd            /usr/local/samba/private/msg.sock/6469
unix  2      [ ]         DGRAM                    63015    6466/smbd            /usr/local/samba/private/msg.sock/6466
unix  2      [ ]         DGRAM                    68142    6467/smbd            /usr/local/samba/private/msg.sock/6467
unix  2      [ ]         DGRAM                    68600    6476/smbd            /usr/local/samba/private/msg.sock/6476
unix  2      [ ]         DGRAM                    68605    6478/smbd            /usr/local/samba/private/msg.sock/6478
unix  2      [ ]         DGRAM                    63034    6479/smbd            /usr/local/samba/private/msg.sock/6479
unix  2      [ ]         DGRAM                    69776    6611/smbd            /usr/local/samba/private/msg.sock/6611
unix  2      [ ]         DGRAM                    69798    6620/smbd            /usr/local/samba/private/msg.sock/6620
unix  2      [ ]         DGRAM                    63332    6614/smbd            /usr/local/samba/private/msg.sock/6614
unix  2      [ ]         DGRAM                    63334    6622/smbd            /usr/local/samba/private/msg.sock/6622
unix  2      [ ]         DGRAM                    63343    6628/smbd            /usr/local/samba/private/msg.sock/6628
unix  2      [ ]         DGRAM                    69801    6629/smbd            /usr/local/samba/private/msg.sock/6629
unix  2      [ ACC ]     STREAM     LISTENING     14388    1/systemd            @ISCSIADM_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     STREAM     LISTENING     16843    791/cupsd            /var/run/cups/cups.sock
unix  2      [ ]         DGRAM                    62753    6292/smbd            /usr/local/samba/private/msg.sock/6292
unix  3      [ ]         STREAM     CONNECTED     872      527/audispd          
unix  3      [ ]         STREAM     CONNECTED     13805    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     12845    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  2      [ ]         STREAM     CONNECTED     40938    3243/httpd           
unix  2      [ ]         DGRAM                    15254    1058/clamd           
unix  3      [ ]         STREAM     CONNECTED     50908    1882/smbd            
unix  2      [ ]         STREAM     CONNECTED     15274    1070/httpd           
unix  3      [ ]         STREAM     CONNECTED     915      536/systemd-logind   
unix  3      [ ]         STREAM     CONNECTED     14563    556/wpa_supplicant   
unix  3      [ ]         STREAM     CONNECTED     17188    1057/httpd           
unix  3      [ ]         DGRAM                    13400    411/systemd-udevd    
unix  2      [ ]         DGRAM                    61474    5637/smbd            
unix  2      [ ]         STREAM     CONNECTED     60715    536/systemd-logind   
unix  3      [ ]         STREAM     CONNECTED     16108    908/winbindd         
unix  3      [ ]         STREAM     CONNECTED     16846    791/cupsd            
unix  3      [ ]         STREAM     CONNECTED     18715    1198/crond           
unix  2      [ ]         DGRAM                    933      545/avahi-daemon: r  
unix  3      [ ]         STREAM     CONNECTED     57347    908/winbindd         
unix  3      [ ]         STREAM     CONNECTED     13938    791/cupsd            
unix  3      [ ]         STREAM     CONNECTED     12900    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  2      [ ]         STREAM     CONNECTED     60790    5643/smbd            
unix  3      [ ]         STREAM     CONNECTED     48652    898/samba            /usr/local/samba/var/run/ncalrpc/np/samr
unix  2      [ ]         STREAM     CONNECTING    0        -                    /usr/local/samba/var/run/winbindd/pipe
unix  3      [ ]         STREAM     CONNECTED     14436    387/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     14479    537/NetworkManager   
unix  2      [ ]         DGRAM                    949      537/NetworkManager   
unix  3      [ ]         STREAM     CONNECTED     14939    906/smbd             
unix  3      [ ]         STREAM     CONNECTED     14551    387/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     17519    387/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         STREAM     CONNECTED     60835    908/winbindd         /usr/local/samba/var/locks/winbindd_privileged/pipe
unix  2      [ ]         STREAM     CONNECTED     60784    5641/smbd            
unix  2      [ ]         DGRAM                    12760    538/dbus-daemon      
unix  3      [ ]         STREAM     CONNECTED     15685    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     15940    908/winbindd         
unix  3      [ ]         STREAM     CONNECTED     937      549/avahi-daemon: c  
unix  2      [ ]         STREAM     CONNECTED     60736    5637/smbd            
unix  2      [ ]         DGRAM                    27437    1882/smbd            
unix  3      [ ]         STREAM     CONNECTED     12745    1/systemd            
unix  3      [ ]         STREAM     CONNECTED     13724    537/NetworkManager   
unix  2      [ ]         STREAM     CONNECTING    0        -                    /usr/local/samba/var/run/winbindd/pipe
unix  2      [ ]         DGRAM                    58792    5641/smbd            
unix  2      [ ]         STREAM     CONNECTED     15252    1058/clamd           
unix  3      [ ]         STREAM     CONNECTED     16148    387/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     936      545/avahi-daemon: r  
unix  2      [ ]         DGRAM                    59923    5643/smbd            
unix  3      [ ]         STREAM     CONNECTED     13263    387/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         STREAM     CONNECTING    0        -                    /usr/local/samba/var/run/winbindd/pipe
unix  3      [ ]         STREAM     CONNECTED     16107    1051/winbindd        
unix  3      [ ]         STREAM     CONNECTED     13723    537/NetworkManager   
unix  2      [ ]         DGRAM                    14099    830/iscsid           
unix  2      [ ]         STREAM     CONNECTED     54844    4971/CROND           
unix  3      [ ]         STREAM     CONNECTED     27951    898/samba            /usr/local/samba/var/run/ncalrpc/np/netlogon
unix  3      [ ]         STREAM     CONNECTED     14502    387/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     20499    387/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         STREAM     CONNECTED     15268    1068/httpd           
unix  3      [ ]         STREAM     CONNECTED     14628    387/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     12758    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     14481    387/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    825      510/auditd           
unix  3      [ ]         STREAM     CONNECTED     16202    1058/clamd           
unix  3      [ ]         STREAM     CONNECTED     12756    538/dbus-daemon      
unix  2      [ ]         STREAM     CONNECTED     58754    5640/smbd            
unix  3      [ ]         STREAM     CONNECTED     13622    536/systemd-logind   
unix  2      [ ]         DGRAM                    19705    1198/crond           
unix  2      [ ]         DGRAM                    13391    411/systemd-udevd    
unix  2      [ ]         STREAM     CONNECTING    0        -                    /usr/local/samba/var/run/winbindd/pipe
unix  2      [ ]         DGRAM                    59881    5639/smbd            
unix  2      [ ]         STREAM     CONNECTING    0        -                    /usr/local/samba/var/run/winbindd/pipe
unix  3      [ ]         STREAM     CONNECTED     871      532/sedispatch       
unix  3      [ ]         STREAM     CONNECTED     13669    546/irqbalance       
unix  3      [ ]         STREAM     CONNECTED     13803    557/polkitd          
unix  3      [ ]         STREAM     CONNECTED     12755    538/dbus-daemon      
unix  2      [ ]         STREAM     CONNECTED     15276    1071/httpd           
unix  3      [ ]         STREAM     CONNECTED     823      527/audispd          
unix  3      [ ]         STREAM     CONNECTED     9940     387/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     16830    387/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         STREAM     CONNECTED     55177    5166/ntpd            
unix  3      [ ]         STREAM     CONNECTED     13984    796/sshd             
unix  2      [ ]         DGRAM                    46636    3604/sshd: root@pts  
unix  3      [ ]         STREAM     CONNECTED     12807    387/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     12757    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    16862    829/iscsid           
unix  2      [ ]         DGRAM                    866      527/audispd          
unix  3      [ ]         STREAM     CONNECTED     15939    919/winbindd         
unix  2      [ ]         DGRAM                    9881     1/systemd            
unix  3      [ ]         STREAM     CONNECTED     13216    387/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     15705    791/cupsd            
unix  2      [ ]         DGRAM                    13616    536/systemd-logind   
unix  3      [ ]         STREAM     CONNECTED     16833    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  2      [ ]         STREAM     CONNECTED     59880    5639/smbd            
unix  2      [ ]         STREAM     CONNECTED     27426    1882/smbd            
unix  3      [ ]         STREAM     CONNECTED     14480    387/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         STREAM     CONNECTING    0        -                    /usr/local/samba/var/run/winbindd/pipe
unix  2      [ ]         DGRAM                    58777    5640/smbd            
unix  2      [ ]         DGRAM                    14086    823/xinetd           
unix  2      [ ]         STREAM     CONNECTED     17523    1069/httpd           
unix  3      [ ]         STREAM     CONNECTED     14215    798/python           
unix  3      [ ]         STREAM     CONNECTED     824      510/auditd           
unix  2      [ ]         STREAM     CONNECTED     43074    3249/httpd           
unix  2      [ ]         STREAM     CONNECTED     42298    3231/httpd           
unix  3      [ ]         STREAM     CONNECTED     13261    798/python           
unix  2      [ ]         STREAM     CONNECTED     15278    1067/httpd           
unix  3      [ ]         DGRAM                    13401    411/systemd-udevd    
unix  3      [ ]         STREAM     CONNECTED     13804    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     57346    5115/winbindd        
unix  3      [ ]         STREAM     CONNECTED     15754    888/colord           
unix  3      [ ]         STREAM     CONNECTED     941      545/avahi-daemon: r  
unix  3      [ ]         STREAM     CONNECTED     16847    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    57466    5166/ntpd            
unix  2      [ ]         STREAM     CONNECTED     40952    3248/httpd           
unix  3      [ ]         STREAM     CONNECTED     14688    387/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         STREAM     CONNECTED     46626    3604/sshd: root@pts  
unix  2      [ ]         STREAM     CONNECTED     19923    1223/named           
unix  3      [ ]         STREAM     CONNECTED     13791    556/wpa_supplicant   
unix  2      [ ]         DGRAM                    1009     557/polkitd          
unix  2      [ ]         STREAM     CONNECTED     43071    3247/httpd           
unix  3      [ ]         STREAM     CONNECTED     14544    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    16229    1057/httpd           
unix  3      [ ]         STREAM     CONNECTED     12806    545/avahi-daemon: r  
unix  3      [ ]         STREAM     CONNECTED     14278    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     14030    807/sh               
unix  3      [ ]         STREAM     CONNECTED     13610    538/dbus-daemon      
unix  3      [ ]         STREAM     CONNECTED     9939     411/systemd-udevd    
unix  2      [ ]         DGRAM                    17636    1223/named           
unix  3      [ ]         STREAM     CONNECTED     15706    538/dbus-daemon      /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     25471    1882/smbd            
unix  3      [ ]         STREAM     CONNECTED     16832    888/colord           
unix  2      [ ]         DGRAM                    43035    1066/logger          
netstat: no support for `AF IPX' on this system.
netstat: no support for `AF AX25' on this system.
netstat: no support for `AF X25' on this system.
netstat: no support for `AF NETROM' on this system.





smbcontrol pool-usage 898
Can't find pid for destination 'pool-usage'
[root@~]# 

[global]
    ldap server require strong auth = no
    allow dcerpc auth level connect = yes
    client ipc signing = default
    client ipc max protocol = default
    client ipc min protocol = default
    ntlm auth = No
    lanman auth = No
    raw NTLMv2 auth = No
    client NTLMv2 auth = Yes
    client lanman auth = Yes
    server max protocol = SMB3_11
    server min protocol = LANMAN1
    client max protocol = SMB3_11
    client min protocol = CORE
    nsupdate command =  /usr/bin/nsupdate -g
#    netbios aliases = XXXXX
    log level = 3 passdb:3 auth:10 winbind:2
    log file = /var/log/samba/log.%m
    max log size = 0
    logging = file
    allow dns updates = nonsecure and secure
    dns proxy = No
    dns forwarder =
    restrict anonymous = 0
    usershare allow guests = No
    security = USER
    allow trusted domains = Yes
    bind interfaces only = Yes
    obey pam restrictions = Yes
    interfaces = lo ens192
    auth methods =
    server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl, winbindd, ntp_signd, kcc, dnsupdate
    dcerpc endpoint servers = epmapper, wkssvc, rpcecho, samr, netlogon, lsarpc, drsuapi, dssetup, unixinfo, browser, eventlog6, backupkey, remote, dnsserver
    dos charset = CP850
    unix charset = UTF-8
    workgroup = TEST
    realm = TEST.LOCAL
    netbios name = XXXXX
    server string = XXXXX Samba Server
    hosts allow = ALL 127.0.0.1
    guest ok = No
    guest account = nobody
    map to guest = Bad User
    guest only = No
    config backend = file
    server role = active directory domain controller
    server role check:inhibit = yes
    encrypt passwords = Yes
    old password allowed period = 120
    password server = XXXXX.TEST.local
    smb passwd file = /usr/local/samba/private/smbpasswd
    private dir = /usr/local/samba/private
    passdb backend = tdbsam
    algorithmic rid base = 1000
    pam password change = Yes
    passwd chat = *New*password* %n\n *ReType*new*password* %n\n*passwd:*all*authentication*tokens*updated*successfully*
    passwd chat debug = No
    passwd chat timeout = 2
    username level = 0
    unix password sync = Yes
    client plaintext auth = Yes
    dedicated keytab file = /etc/krb5.keytab
    kerberos method = system keytab
    map untrusted to domain = Yes
    debug timestamp = Yes
    debug prefix timestamp = No
    debug hires timestamp = Yes
    debug pid = No
    debug uid = No
    debug class = No
    enable core files = Yes
    smb ports = 445 137 138 139
    large readwrite = Yes
    client schannel = Auto
    server schannel = Auto
    unicode = Yes
    min receivefile size = 16384
    read raw = Yes
    write raw = Yes
    disable netbios = No
    reset on zero vc = No
    log writeable files on exit = Yes
    defer sharing violations = Yes
    nt pipe support = Yes
    nt status support = Yes
    max mux = 50
    max xmit = 32768
    name resolve order = wins bcast hosts
    max ttl = 259200
    max wins ttl = 518400
    min wins ttl = 21600
    time server = Yes
    unix extensions = No
#    client signing = required
#    server signing = required
    client signing = mandatory
    server signing = mandatory
    client use spnego = Yes
    client ldap sasl wrapping = sign
    enable asu support = No
    cldap port = 389
    krb5 port = 88
    kpasswd port = 464
    web port = 901
    rpc big endian = No
    deadtime = 0
    getwd cache = Yes
    keepalive = 300
    lpq cache time = 30
    smbd profiling level = off
    spotlight = no
    tls priority = NORMAL:-VERS-SSL3.0
    max smbd processes = 0
    max disk size = 0
    max open files = 65535
    socket options = IPTOS_LOWDELAY TCP_NODELAY SO_KEEPALIVE
    use mmap = Yes
    hostname lookups = No
    name cache timeout = 3600
    clustering = No
    ctdb timeout = 0
    ctdb locktime warn threshold = 0
    smb2 max read = 8388608
    smb2 max write = 8388608
    smb2 max trans = 8388608
    smb2 max credits = 8192
    mangling method = hash
    mangle prefix = 1
    max stat cache size = 256
    stat cache = No
    machine password timeout = 0
    username map cache time = 0
    logon script =
    logon path =
    logon drive =
    logon home =
    domain logons = Yes
    init logon delay = 100
    os level = 255
    lm announce = Auto
    lm interval = 60
    preferred master = Yes
    local master = Yes
    domain master = Yes
    browse list = Yes
    enhanced browsing = No
    wins proxy = Yes
    wins support = Yes
    smb2 leases = Yes
    ldap admin dn = 
    ldap delete dn = No
    ldap group suffix = 
    ldap idmap suffix = 
    ldap machine suffix = 
    ldap passwd sync = no
    ldap replication sleep = 1000
    ldap suffix = 
    ldap ssl = start tls
    ldap ssl ads = No
    ldap deref = auto
    ldap follow referral = Auto
    ldap timeout = 15
    ldap connection timeout = 2
    ldap page size = 1024
    ldap user suffix = 
    ldap debug level = 0
    ldap debug threshold = 10
    lock directory = /usr/local/samba/var/lock
    state directory = /usr/local/samba/var/locks
    cache directory = /usr/local/samba/var/cache
    pid directory = /usr/local/samba/var/run
    ntp signd socket directory = /usr/local/samba/var/lib/ntp_signd
    utmp = No
    nmbd bind explicit broadcast = Yes
    homedir map = auto.home
    afs token lifetime = 604800
    afs share = No
    afs username map =
    NIS homedir = Yes
    registry shares = Yes
    usershare max shares = 0
    usershare owner only = No
    usershare path = /usr/local/samba/var/locks/usershares
    allow insecure wide links = No
    async smb echo handler = No
    host msdfs = No
    msdfs shuffle referrals = No
    passdb expand explicit = No
    idmap cache time = 604800
    idmap negative cache time = 120
    template homedir = /home/%D/%U
    template shell = /bin/bash
    winbind cache time = 3600
    winbind reconnect delay = 30
    winbind request timeout = 60
    winbind max clients = 2000
    winbind enum users = Yes
    winbind enum groups = Yes
    winbind use default domain = Yes
    winbind trusted domains only = No
    winbind nested groups = Yes
    winbind expand groups = 10
    winbind nss info = rfc2307
    winbind refresh tickets = Yes
    winbind offline logon = Yes
    winbind normalize names = Yes
    winbind rpc only = Yes
    create krb5 conf = Yes
    ncalrpc dir = /usr/local/samba/var/run/ncalrpc
    winbind max domain connections = 1
    winbindd socket directory = /usr/local/samba/var/run/winbindd
    winbindd privileged socket directory = /usr/local/samba/var/lib/winbindd_privileged
    winbind sealed pipes = No
    neutralize nt4 emulation = Yes
    winbind sealed pipes:TEST = Yes
    require strong key:TEST = Yes
    reject md5 servers = No
    require strong key = No
#    require strong key = Yes
    multicast dns register = Yes
    samba kcc command = /usr/local/samba/sbin/samba_kcc
    spn update command = /usr/local/samba/sbin/samba_spnupdate
    share backend = classic
    allow nt4 crypto = Yes
    reject md5 clients = No
    tls enabled = Yes
    tls keyfile = tls/key.pem
    tls certfile = tls/cert.pem
    tls cafile = tls/ca.pem
    tls crlfile = 
    tls dh params file = 
    spoolssd:prefork_child_min_life = 60
    spoolssd:prefork_max_allowed_clients = 200
    spoolssd:prefork_spawn_rate = 5
    spoolssd:prefork_max_children = 75
    spoolssd:prefork_min_children = 5
    rpc_server:tcpip = no
    rpc_daemon:spoolssd = fork
    rpc_server:default = external
    rpc_server:spoolss = external
    rpc_server:svcctl = embedded
    rpc_server:srvsvc = embedded
    rpc_server:eventlog = embedded
    rpc_server:ntsvcs = embedded
    rpc_server:winreg = embedded
    dsdb:schema update allowed = yes
    sdb:schema update allowed = yes
    idmap config TEST : range = 2000000-9999999
    idmap config TEST : default = yes
    idmap config TEST : backend = ad
    idmap config TEST : readonly = no
    idmap config TEST : schema_mode = rfc2307
    idmap config TEST : cache time = 3600
    idmap config * : default = yes
    idmap config * : readonly = no
    idmap config * : schema_mode = rfc2307
    idmap config * : backend = tdb
    idmap config * : range = 2000000-9999999
    idmap_ldb:use rfc2307 = yes
    idmap config all : readonly = yes
    idmap config all : default = yes
    idmap config all : backend = tdb
    dbwrap_tdb_mutexes:* = yes
    prefork children:smb = 4
    registry:hkey_users = hku.ldb
    registry:hkey_local_machine = hklm.ldb
    kccsrv:samba_kcc = true
    read only = Yes
##acl  ayarlari###
    acl group control = Yes
    acl map full control = Yes
    acl allow execute always = Yes
    force unknown acl user = Yes
    inherit permissions = Yes
    inherit acls = Yes
    inherit owner = No
##inherit owner No olmalidir. hangi kullanici ile dosya kaydettigini belirtir
    map acl inherit = Yes
    nt acl support = Yes
    acl:search = no
    acl_xattr:ignore system acl = yes
    profile acls = Yes
##acl ayarlari sonu
    administrative share = No
    allocation roundup size = 1048576
    aio read size = 16384
    aio write size = 16384
    aio max threads = 100
    ea support = Yes
    smb encrypt = default
    durable handles = Yes
    block size = 1024
    change notify = Yes
    directory name cache size = 100
    kernel change notify = No
    max connections = 65535
    strict allocate = Yes
    strict rename = No
    strict sync = No
    sync always = No
    use sendfile = Yes
    write cache size = 0
    default case = lower
    case sensitive = No
    preserve case = Yes
    short preserve case = Yes
    mangling char = ~
    hide dot files = Yes
    hide special files = No
    hide unreadable = No
    hide unwriteable files = No
    delete veto files = No
    map archive = Yes
    map hidden = Yes
    map system = Yes
    map readonly = yes
    mangled names = Yes
    store dos attributes = Yes
    dmapi support = No
    browseable = Yes
    access based share enum = No
    blocking locks = Yes
    csc policy = disable
    lock spin time = 200
    oplock break wait time = 0
    fake oplocks = No
    kernel oplocks = Yes
    kernel share modes = Yes
    locking = Yes
    oplocks = Yes
    level2 oplocks = Yes
    oplock contention limit = 2
    posix locking = Yes
    strict locking = Auto
    dfree cache time = 0
    preexec close = No
    root preexec close = No
    available = Yes
    fstype = NTFS
    wide links = Yes
    follow symlinks = Yes
    delete readonly = No
    dos filemode = Yes
    dos filetimes = Yes
    dos filetime resolution = Yes
    fake directory create times = No
    msdfs root = No
    ntvfs handler = unixuid, default
    vfs objects = acl_xattr full_audit
    full_audit:prefix = IP=%I|USER=%u|MACHINE=%m|VOLUME=%S
    full_audit:failure = connect disconnect
    full_audit:success = connect disconnect opendir mkdir rmdir closedir open close read pread write pwrite sendfile rename unlink chmod fchmod chown fchown chdir ftruncate lock symlink readlink link mknod
    full_audit:TEST = local5
    full_audit:priority = notice
#printer satirlari#
    load printers = No
    use client driver = No
    show add printer wizard = Yes
    printcap cache time = 0
    printcap name = cups
    cups encrypt = No
    cups connection timeout = 60
    disable spoolss = No
    min print space = 0
    max reported print jobs = 0
    max print jobs = 1000
    print notify backchannel = No
    printing = cups
    cups options = raw
    default devmode = Yes
    force printername = Yes
    printjob username = %U
    spoolss: architecture = Windows x64
[homes]
    comment = Home Directories
    path = /strg/homes/%U
    valid users = @"Domain Users"
    admin users = @"Domain Admins"
    read only = No
    browseable = No
    writable = Yes
    create mask = 0644            
    force create mode = 0660      
    force directory mode = 0770  
    directory mask = 0755
    veto files = /*.encrypted/*.ecc/*.ccc/
    hide files = /Recycle Bin/
    vfs objects = acl_xattr full_audit recycle
    recycle:mode = KEEP_DIRECTORIES|VERSION|TOUCH
    recycle:noversions = *.tmp|*.temp|*.dat|*.ini
    recycle:excludedir = /Recycle Bin
    recycle:exclude = *.tmp|*.temp|*.o|*.obj|~$*|*.??|*.log|*.trace|*.TMP|*.ASV|*.$$$|*.asv
    recycle:versions = yes
    recycle:touch_mtime = yes
    recycle:touch = yes
    recycle:maxsize = 0             
    recycle:minsize = 0
    recycle:keeptree = yes
    recycle:repository = Recycle Bin
[profiles]
    comment = Network Profiles Share
    path = /strg/profiles
    directory mask = 0755
    create mask = 0644            
    force create mode = 0660      
    force directory mode = 0770  
    profile acls = Yes
    guest ok = No
    read only = No
#browseable izni bir defalik kurulum esnasinda permision ayarlari esnasinda acilir.sonra kapatilir. 
    browseable = No
    write ok = Yes
[netlogon]
    comment = Network Netlogon Share
    path = /usr/local/samba/var/locks/sysvol/TEST.local/scripts
    read only = No
    guest ok = No
    write ok = Yes
    browseable = No
[sysvol]
    path = /usr/local/samba/var/locks/sysvol
    read only = No
    browseable = No
    write ok = Yes
[printers]
    comment = All Printers
    path = /var/spool/samba
    write list = root administrator
    writeable = No
    printable = Yes
    print ok = Yes
    guest ok = Yes
    public = Yes
    browseable = Yes
    acl_xattr:ignore system acl = yes
[print$]         
    comment = Printer Drivers
    path = /strg/printer_drivers
    invalid users = qwerty
    valid users = @"Domain Users"
    admin users = @"Domain Admins"                  
    write list = root administrator
    writeable = Yes
    read only = Yes        
    browseable = Yes                 
    guest ok = Yes
    create mask = 0660
    create mask = 0644
    force create mode = 0660
    force directory mode = 0770
    directory mask = 0755    
    acl_xattr:ignore system acl = yes
Comment 3 Andrew Bartlett 2016-08-01 10:26:21 UTC
Sorry, please use against all the stuck processes:

smbcontrol <pid> pool-usage

also attach with gdb -p <pid> and get a 'bt full'.

Take care that both may expose sensitive system details, but as this is an rc release I'm assuming this is a test system.

Thanks!
Comment 4 Andrew Bartlett 2016-08-01 19:56:43 UTC
BTW, can you confirm the last version that this worked on?

I'm keen to know if this is a regression from Samba 4.4 (it seems clear it is), so we can mark it as such.

Thanks!
Comment 5 baris tombul 2016-08-02 06:15:10 UTC
(In reply to Andrew Bartlett from comment #4)

thanks for your cooperation. I updated on live system,a problem occured during updating and users were used the computers. Therefore, take this update back to 4.4.5 version. I have no problem about this 4.4.5 version now.

Kind regards.

Barış
Comment 6 Andrew Bartlett 2016-08-03 09:27:25 UTC
If you are able to safely reproduce this somehow, and can get me that 'bt full' from a stuck smbd process, that would help narrow this down a lot.

Thanks,
Comment 7 baris tombul 2016-08-10 16:31:19 UTC
Created attachment 12356 [details]
netstat
Comment 8 baris tombul 2016-08-10 16:33:26 UTC
Created attachment 12357 [details]
smbcontrol
Comment 9 baris tombul 2016-08-10 16:35:05 UTC
Version 4.5.0rc2

critical problem. server very slow. ssh vs.. command dont respond.
Comment 10 baris tombul 2016-08-10 16:36:59 UTC
â samba.service - Samba Daemon
   Loaded: loaded (/usr/lib/systemd/system/samba.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-08-10 19:13:49 EEST; 22min ago
  Process: 879 ExecStart=/usr/local/samba/sbin/samba --daemon $SAMBAOPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 883 (samba)
   CGroup: /system.slice/samba.service
           ââ 883 /usr/local/samba/sbin/samba --daemon
           ââ 927 /usr/local/samba/sbin/samba --daemon
           ââ 928 /usr/local/samba/sbin/samba --daemon
           ââ 929 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ 930 /usr/local/samba/sbin/samba --daemon
           ââ 931 /usr/local/samba/sbin/samba --daemon
           ââ 932 /usr/local/samba/sbin/samba --daemon
           ââ 933 /usr/local/samba/sbin/samba --daemon
           ââ 934 /usr/local/samba/sbin/samba --daemon
           ââ 935 /usr/local/samba/sbin/samba --daemon
           ââ 936 /usr/local/samba/sbin/samba --daemon
           ââ 937 /usr/local/samba/sbin/samba --daemon
           ââ 938 /usr/local/samba/sbin/samba --daemon
           ââ 939 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
           ââ 940 /usr/local/samba/sbin/samba --daemon
           ââ 946 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
           ââ 947 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ 948 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ 949 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1015 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
           ââ1374 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1377 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1382 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1383 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1384 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1385 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1387 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1388 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1389 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1391 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1393 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1523 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1525 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1526 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1527 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1529 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1530 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1535 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1542 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1543 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1544 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1545 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1550 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1551 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1553 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1554 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1555 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1556 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1557 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1558 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1559 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1561 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1692 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1693 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1695 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1697 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1698 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1700 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1701 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1703 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1708 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1710 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1712 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1713 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1714 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1716 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1718 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1720 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1721 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1722 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1724 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1862 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1868 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1869 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1870 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1873 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1874 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1876 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1877 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1878 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1879 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1884 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1886 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1887 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1888 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1901 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1902 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1904 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1906 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1909 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1913 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1915 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1917 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1919 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1922 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2057 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2061 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2062 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2063 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2065 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2082 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2086 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2088 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2090 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2093 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2095 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2096 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2097 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2110 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2117 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2118 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2119 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2125 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2128 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2145 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2164 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2277 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2278 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2281 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2282 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2283 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2302 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2305 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2306 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2307 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2313 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2314 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2316 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2318 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2319 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2327 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2330 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2331 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2333 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2334 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2336 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2338 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2339 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2340 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2344 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2474 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2476 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2477 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2479 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2480 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2483 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2487 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2493 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2494 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2495 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2500 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2502 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2504 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2505 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2507 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2509 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
Comment 11 baris tombul 2016-08-11 08:19:22 UTC
my problem is solved.
problem this line >> obey pam restrictions = Yes

samba 4.4.x (all version) obey pam restrictions = Yes  >>> running >> no problem

samba 4.5.X  obey pam restrictions = No >> running >> no problem

??

best regards
Comment 12 baris tombul 2016-08-12 06:19:35 UTC
Created attachment 12364 [details]
smbcontrol_last_error

:(( problem resuming. doesnt solved.
Comment 13 Andrew Bartlett 2016-08-26 20:25:41 UTC
So, to summarise:

This isn't a case that the process is blocked somewhere deep in the stack, because smbcontrol talloc reports (which don't really show anything interesting as far as I can tell) will only return when the process is in the main tevent loop.

To me this means that the process is not detecting the end of file from the dropped socket, or that something else is keeping the process alive because it isn't for a specific client.

Does the problem go away if you stop using printing, and set 'disable spoolss = yes'?

Team:  Do we know what this process is?  Could it be a helper process of some kind?  How could we tell?

Thanks,
Comment 14 Jeremy Allison 2016-08-26 20:35:59 UTC
(In reply to Andrew Bartlett from comment #13)

If this is on Linux, what does /proc/<pid> say about it ? Specifically what open file descriptors does it have ?
Comment 15 Stefan Metzmacher 2016-09-06 12:52:24 UTC
(In reply to baris tombul from comment #0)

[2016/07/30 07:00:47.601811,  0] ../source3/smbd/server.c:1025(smbd_accept_connection)
  smbd_accept_connection: fork() failed: Cannot allocate memory

Indicated that it's the parent smbd

Can you paste the content of /usr/lib/systemd/system/samba.service
and the output of 'smbstatus'

There's not much we can do if fork(2) fails with ENOMEM.

The number of processes and/or memory could be limited by a
ulimit or cgroups.

As this a not a generic problem I'll remove the regression flag.
This should not block the 4.5.0 release.
Comment 16 baris tombul 2016-09-06 13:43:39 UTC
 cat /usr/lib/systemd/system/samba.service
[Unit]
Description=Samba Daemon
Wants=nss-lookup.target
Wants=named-setup-rndc.service
Before=nss-lookup.target
After=syslog.target network.target nmb.service
[Service]
PermissionsStartOnly=true
Type=forking
Environment=KRB5CCNAME=/usr/local/samba/var/run/krb5cc_samba
EnvironmentFile=-/etc/sysconfig/samba
PIDFile=/usr/local/samba/var/run/samba.pid
LimitNOFILE=16384
ExecStart=/usr/local/samba/sbin/samba  --daemon $SAMBAOPTIONS
ExecReload=/bin/sh -c '/usr/local/samba/sbin/samba reload > /dev/null 2>&1  || /bin/kill -HUP $MAINPID || /bin/kill -HUP -a smbd ||  /bin/kill -HUP -a samba ||  /bin/kill -HUP -a  winbindd'
ExecStop=/bin/sh -c '/usr/local/samba/sbin/samba stop > /dev/null 2>&1  || /bin/kill -TERM $MAINPID || /bin/kill -TERM -a smbd ||  /bin/kill -TERM -a samba ||  /bin/kill -TERM -a  winbindd'
ExecStop=/bin/sh -c '/usr/bin/rm -f /usr/local/samba/var/run/smbd.pid'
ExecStop=/bin/sh -c '/usr/bin/rm -f /usr/local/samba/var/run/samba.pid'
ExecStop=/bin/sh -c '/usr/bin/rm -f /usr/local/samba/var/run/winbindd.pid'
LimitCORE=infinity
PrivateTmp=true
Restart=on-failure
[Install]
WantedBy=multi-user.target


------------

 service samba status
Redirecting to /bin/systemctl status  samba.service
â samba.service - Samba Daemon
   Loaded: loaded (/usr/lib/systemd/system/samba.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-09-06 14:54:30 EEST; 1h 48min ago
  Process: 913 ExecStart=/usr/local/samba/sbin/samba --daemon $SAMBAOPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 915 (samba)
   CGroup: /system.slice/samba.service
           ââ 915 /usr/local/samba/sbin/samba --daemon
           ââ 960 /usr/local/samba/sbin/samba --daemon
           ââ 961 /usr/local/samba/sbin/samba --daemon
           ââ 962 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ 963 /usr/local/samba/sbin/samba --daemon
           ââ 964 /usr/local/samba/sbin/samba --daemon
           ââ 965 /usr/local/samba/sbin/samba --daemon
           ââ 966 /usr/local/samba/sbin/samba --daemon
           ââ 967 /usr/local/samba/sbin/samba --daemon
           ââ 968 /usr/local/samba/sbin/samba --daemon
           ââ 969 /usr/local/samba/sbin/samba --daemon
           ââ 970 /usr/local/samba/sbin/samba --daemon
           ââ 971 /usr/local/samba/sbin/samba --daemon
           ââ 972 /usr/local/samba/sbin/samba --daemon
           ââ 973 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
           ââ 978 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ 979 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ 980 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
           ââ 982 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ1247 /usr/local/samba/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
           ââ1266 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ2569 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ4955 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ8067 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
           ââ8068 /usr/local/samba/sbin/smbd -D --option=server role check:inhibit=yes --foreground
Comment 17 baris tombul 2016-09-06 13:46:07 UTC
smbstatus

Samba version 4.5.0rc4-GIT-51a6036
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing              
----------------------------------------------------------------------------------------------------------------------------------------
8321    nobody       3000004      10.0.23.13 (ipv4:10.0.23.13:61117)        SMB2_10           -                    -                    
8323    FACILITY\tdb04$ FACILITY\domain_computers 10.0.23.13 (ipv4:10.0.23.13:61119)        SMB2_10           -                    HMAC-SHA256          
8322    nobody       3000004      tdb04 (ipv4:10.0.23.13:61118)             NT1               -                    -                    

Service      pid     Machine       Connected at                     Encryption   Signing     
---------------------------------------------------------------------------------------------
IPC$         8322    tdb04         Tue Sep  6 04:45:29 PM 2016 EEST -            -           
IPC$         8323    10.0.23.13    Tue Sep  6 04:45:29 PM 2016 EEST -            HMAC-SHA256 
IPC$         8321    10.0.23.13    Tue Sep  6 04:45:29 PM 2016 EEST -            -           

No locked files

[root@mems ~]# smbstatus

Samba version 4.5.0rc4-GIT-51a6036
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing              
----------------------------------------------------------------------------------------------------------------------------------------
8333    FACILITY\biomems07$ FACILITY\domain_computers 10.0.10.55 (ipv4:10.0.10.55:58201)        SMB2_10           -                    HMAC-SHA256          

Service      pid     Machine       Connected at                     Encryption   Signing     
---------------------------------------------------------------------------------------------
IPC$         8333    10.0.10.55    Tue Sep  6 04:45:42 PM 2016 EEST -            HMAC-SHA256
Comment 18 Dmitry Vlasov 2016-09-14 07:47:36 UTC
Created attachment 12459 [details]
/var/log/messages

I have same issue with latest samba 4.5.0 on CentOS 7.2.1511. 
Kernel: Linux smb.dc.serviceand.ru 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Comment 19 baris tombul 2016-09-15 16:42:26 UTC
I tried for days. I've found  the problem. ""dns_lookup_kdc ""
nano /etc/krb5.conf
edit:
dns_lookup_kdc = false

my new krb5.conf:

[logging]
   default = FILE:/var/log/krb5libs.log
   kdc = FILE:/var/log/krb5kdc.log
   admin_server = FILE:/var/log/kadmind.log
[kdc]
   check-ticket-addresses = false
[libdefaults]
   default_realm = TEST.LOCAL
   dns_lookup_realm = false
   dns_lookup_kdc = false
   rdns = false
   proxiable = true
   forwardable = true
   allow_weak_crypto = true
   noaddresses = true
   ticket_lifetime = 24h
   renew_lifetime = 7d
[appdefaults]
    pam = {
    debug = false
    ticket_lifetime = 36000
    renew_lifefime = 36000
    forwardable = true
    krb4_convert = false
}
[domain_realm]
   facility.local = TEST.LOCAL
   .facility.local = TEST.LOCAL
[realms]
  FACILITY.LOCAL = {
  kdc = xxx.test.local
  admin_server = XXX.TEST.LOCAL
  default_domain = TEST.LOCAL
}
Comment 20 Jeremy Allison 2016-09-15 18:48:41 UTC
So this looks then like a memory leak inside the krb5 libraries, as dns_lookup_kdc = false is only relevent in krb5.conf.

What version of kerberos are you using ?
Comment 21 baris tombul 2016-09-15 19:03:19 UTC
(In reply to Jeremy Allison from comment #20)

rpm -qa | grep krb
krb5-libs-1.13.2-12.el7_2.x86_64
krb5-devel-1.13.2-12.el7_2.x86_64
krb5-workstation-1.13.2-12.el7_2.x86_64
krb5-pkinit-1.13.2-12.el7_2.x86_64
pam_krb5-2.4.8-4.el7.x86_64
Comment 22 Sander Plas 2016-09-16 10:29:27 UTC
*** Bug 12267 has been marked as a duplicate of this bug. ***
Comment 23 Sander Plas 2016-09-16 10:34:32 UTC
I'm having the same issue on 10 servers in 1 domain. It started after upgrading from 4.4.5 to 4.5.0. About 1 or 2 crashing servers each day. 

I have now changed the dns_lookup_kdc setting on all 10 servers. Will let you know if this solves the problem.
Comment 24 Andrew Bartlett 2016-09-16 14:21:59 UTC
Some thoughts:

If you don't have a KDC listed in your krb5.conf for TEST.LOCAL, then setting 'dns_lookup_kdc=false' is most likely to just break replication.

Of course, that might be the clue.

However we do need to work out what kind of process the apparently un-attached children are?

Specifically, we need the open file descriptors that Jeremy asked for.  

Also, to isolate the issue, we need to eliminate printing: Does 'disable spoolss = yes' help?
Comment 25 Andrew Bartlett 2016-09-16 14:31:18 UTC
Furthermore:

Are the 'stuck' processes consuming CPU?

If so, can you please get me a flamegraph of that per:
http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html

Are they consuming ram?
if so, can you confirm the talloc tree provided earlier is from a stuck smbd process?

What else can we be told about them?  We know they are not network-connected as they don't show up in netstat. 

Can we please get a level 10 log? (mark as private if need be)
Comment 26 Sander Plas 2016-09-16 15:06:12 UTC
Looks like you're right - "samba-tool drs showrepl" was showing thousands of failures on each server. 

After re-enabling dns_lookup_kdc and restarting samba, it's back to "0 consecutive failure(s)".


This is a live environment with 10 servers and there's only 1 or 2 servers crashing each day. As you can imagine, when one does crash, there's quite some pressure from users to restart the thing a.s.a.p. 

So, to summarize, i should set log level to 10 and then when a crash occurs, before restarting samba, collect this info: 

- open file descriptors
- flamegraph
- talloc tree from a stuck smbd process
- log files
- smbcontrol <pid> pool-usage
- gdb -p <pid> -> 'bt full'

...correct? 

Anything else? 

Should i recompile with debugging symbols?
Comment 27 Andrew Bartlett 2016-09-16 15:18:28 UTC
Presumably this problem builds over time, then the server overloads?

It should be enough to get this kind of info from a server pre-failure, when it is starting to show the extra tasks without matching client connections. 

I'm most interested in what these extra smbd processes are, if you see them.  smbcontrol can turn up the log level per-process if level 10 is too much to hold across the whole system, but at this point I need as many clues as possible.  Getting log files per-pid may be helpful with %p in the log file option.

Also run 'samba-tool processes' so we at least get the names for some of the other tasks, as context. 

While a patch for master has just been accepted to squash this message, I'm still deeply suspicious that it is something like a cleanup failing, and so the task smbd not exiting:

Sep 13 15:58:31 smb smbd[3160]: [2016/09/13 15:58:31.530724,  0] ../source3/smbd/smbd_cleanupd.c:172(smbd_cleanupd_process_exited)
Sep 13 15:58:31 smb smbd[3160]:  smbd_cleanupd_process_exited: got 0 cleanup events, expected at least 1

Finally, yes this will all be much better with debug symbols.
Comment 28 Andrew Bartlett 2016-09-16 15:34:23 UTC
Also 

/proc/[pid]/comm

for each stuck (smbd) process
Comment 29 baris tombul 2016-09-16 17:33:28 UTC
my full stable smb.conf:

[global]
    enumports command = /usr/local/bin/show-ports.sh
    ntlm auth = Yes
    lanman auth = No
    raw NTLMv2 auth = No
    client NTLMv2 auth = No
    client lanman auth = No
    idmap_ldb:use rfc2307 = Yes
    algorithmic rid base = 1000
    kerberos method = secrets and keytab
    dedicated keytab file = /etc/krb5.keytab
    winbind max clients = 2000
    winbindd:use external pipes = true
    winbind cache time = 300
    winbind reconnect delay = 30
    winbind request timeout = 60
    winbind max domain connections = 1
    winbindd socket directory = /usr/local/samba/var/run/winbindd
    winbindd privileged socket directory = /usr/local/samba/var/lib/winbindd_privileged
    winbind enum users = Yes
    winbind enum groups = Yes
    winbind use default domain = Yes
    winbind trusted domains only = No
    winbind nested groups = Yes
    winbind expand groups = 10
    winbind nss info = rfc2307
    winbind refresh tickets = Yes
    winbind offline logon = Yes
    winbind normalize names = Yes
    winbind sealed pipes = Yes
    winbind rpc only = Yes
    wins proxy = Yes
    wins support = Yes
    obey pam restrictions = No
    ldap server require strong auth = no
    server max protocol = SMB3
    server min protocol = LANMAN1
    server multi channel support = No
    client max protocol = default
    client min protocol = CORE
    restrict anonymous = 0
    security = USER
    bind interfaces only = Yes
    interfaces = lo ens192
    auth methods =
    server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl, winbindd, ntp_signd, kcc, dnsupdate
    dcerpc endpoint servers = epmapper, wkssvc, rpcecho, samr, netlogon, lsarpc, drsuapi, dssetup, unixinfo, browser, eventlog6, backupkey, remote, dnsserver
    dos charset = CP850
    unix charset = UTF-8
    workgroup = TEST
    realm = TEST.LOCAL
    netbios name = XXX
    netbios scope =
    server string = Test Samba Server
    hosts allow = ALL 127.0.0.1
    guest ok = No
    server role = active directory domain controller
    server role check:inhibit = yes
    log level = 3 passdb:3 auth:10 winbind:2
    log file = /var/log/samba/log.%m
    rndc command = /usr/sbin/rndc
    max log size = 0
    set primary group script =
    logging = file
    allow dns updates = nonsecure and secure
    dns update command = /usr/local/samba/sbin/samba_dnsupdate
    pam password change = Yes
    smb ports = 445 139
    nbt port = 137
    kpasswd port = 464 
    krb5 port = 88
    web port = 901
    nbt port = 137
    dgram port = 138
    cldap port = 389
    socket options = IPTOS_LOWDELAY TCP_NODELAY SO_KEEPALIVE
    domain logons = Yes
    os level = 255
    preferred master = Yes
    local master = Yes
    domain master = Yes
    load printers = No
    use client driver = No
    show add printer wizard = Yes
    printcap cache time = 0
    printcap name = cups
    cups encrypt = No
    cups connection timeout = 60
    disable spoolss = No
    min print space = 0
    max reported print jobs = 0
    max print jobs = 1000
    print notify backchannel = No
    printing = cups
    cups options = raw
    default devmode = Yes
    force printername = Yes
    printjob username = %U
    lpq cache time = 30
    spoolss: architecture = Windows x64
    debug timestamp = Yes
    debug prefix timestamp = No
    debug hires timestamp = Yes
    debug pid = No
    debug uid = No
    debug class = No
    timestamp logs = Yes
    require strong key = Yes
    allow dcerpc auth level connect = No
    client ipc signing = default
    client ipc max protocol = default
    client ipc min protocol = default
    nsupdate command =  /usr/bin/nsupdate -g
    dns proxy = No
    allow trusted domains = Yes
    guest account = nobody
    map to guest = Bad User
    guest only = No
    config backend = file
    encrypt passwords = Yes
    smb passwd file = /usr/local/samba/private/smbpasswd
    private dir = /usr/local/samba/private
    passdb expand explicit = No
    passdb backend = tdbsam
    passwd chat debug = No
    passwd chat timeout = 2
    passwd program = /usr/local/samba/bin/smbpasswd %u
    passwd chat = *New*password* %n\n *ReType*new*password* %n\n*passwd:*all*authentication*tokens*updated*successfully*
    password server = xxx.test.local
    old password allowed period = 120
    unix password sync = Yes
    client plaintext auth = No
    map untrusted to domain = Yes
    enable core files = Yes
    large readwrite = Yes
    unicode = Yes
    read raw = Yes
    write raw = Yes
    disable netbios = No
    reset on zero vc = No
    log writeable files on exit = No
    defer sharing violations = Yes
    nt pipe support = Yes
    nt status support = Yes
    max mux = 50
    max xmit = 32768
    name resolve order = lmhosts wins host bcast
    max ttl = 259200
    max wins ttl = 518400
    min wins ttl = 21600
    min receivefile size = 16384
    time server = Yes
    time server = No
    unix extensions = Yes
    server signing = mandatory
    client signing = mandatory
    client schannel = Auto
    server schannel = Auto
    client use spnego = Yes
    client ldap sasl wrapping = sign
    enable asu support = No
    rpc big endian = No
    deadtime = 0
    getwd cache = Yes
    keepalive = 300
    smbd profiling level = off
    spotlight = No
    max smbd processes = 0
    max disk size = 0
    max open files = 65535
    use mmap = Yes
    hostname lookups = No
    name cache timeout = 3600
    clustering = No
    ctdb timeout = 0
    ctdb locktime warn threshold = 0
    smb2 max read = 8388608
    smb2 max write = 8388608
    smb2 max trans = 8388608
    smb2 max credits = 8192
    mangling method = hash2
    mangle prefix = 1
    max stat cache size = 256
    stat cache = Yes
    machine password timeout = 604800
    username map cache time = 0
    username level = 0
    init logon delay = 100
    lm announce = Auto
    lm interval = 60
    browse list = Yes
    enhanced browsing = Yes
    smb2 leases = Yes
    lock directory = /usr/local/samba/var/lock
    state directory = /usr/local/samba/var/locks
    cache directory = /usr/local/samba/var/cache
    pid directory = /usr/local/samba/var/run
    ntp signd socket directory = /usr/local/samba/var/lib/ntp_signd
    utmp = No
    nmbd bind explicit broadcast = Yes
    homedir map = auto.home
    afs token lifetime = 604800
    afs share = No
    NIS homedir = No
    registry shares = No
    usershare allow guests = No
    usershare max shares = 0
    usershare owner only = Yes
    usershare path = /usr/local/samba/var/locks/usershares
    async smb echo handler = No
    template homedir = /home/%D/%U
    template shell = /bin/bash
    create krb5 conf = Yes
    ncalrpc dir = /usr/local/samba/var/run/ncalrpc
    neutralize nt4 emulation = No
    reject md5 servers = No
    reject md5 clients = No
    set quota command =
    multicast dns register = Yes
    samba kcc command = /usr/local/samba/sbin/samba_kcc
    spn update command = /usr/local/samba/sbin/samba_spnupdate
    share backend = classic
    allow nt4 crypto = No
    tls enabled = Yes
    tls keyfile = tls/key.pem
    tls certfile = tls/cert.pem
    tls cafile = tls/ca.pem
    tls crlfile =
    tls dh params file =
    tls verify peer = as_strict_as_possible
    tls priority = NORMAL:-VERS-SSL3.0
    rpc_server:spoolss = external
    rpc_daemon:spoolssd = fork
    spoolssd:prefork_child_min_life = 60
    spoolssd:prefork_max_allowed_clients = 200            
    spoolssd:prefork_spawn_rate = 5     
    spoolssd:prefork_max_children = 75      
    spoolssd:prefork_min_children = 5
    acl group control = No
    acl map full control = Yes
    acl allow execute always = No
    force unknown acl user = No
    inherit permissions = Yes
    inherit acls = Yes
    inherit owner = No
    map acl inherit = Yes
    nt acl support = Yes
    profile acls = No
    administrative share = No
    allocation roundup size = 1048576
    aio read size = 16384
    aio write size = 16384
    aio max threads = 100
    ea support = No
    smb encrypt = default
    durable handles = Yes
    block size = 1024
    change notify = Yes
    directory name cache size = 100
    kernel change notify = Yes
    max connections = 0
    strict allocate = No
    strict rename = No
    strict sync = No
    sync always = No
    use sendfile = No
    write cache size = 0
    default case = lower
    case sensitive = No
    preserve case = Yes
    short preserve case = Yes
    mangling char = ~
    hide dot files = Yes
    hide special files = No
    hide unreadable = No
    hide unwriteable files = No
    delete veto files = No
    map archive = No
    map hidden = No
    map system = No
    map readonly = No
    mangled names = Yes
    mangling char = ~
    store dos attributes = Yes
    dmapi support = No
    browseable = Yes
    access based share enum = No
    blocking locks = Yes
    csc policy = manual
    lock spin time = 200
    oplock break wait time = 0
    fake oplocks = No
    kernel oplocks = No
    kernel share modes = Yes
    locking = Yes
    oplocks = Yes
    level2 oplocks = Yes
    oplock contention limit = 2
    posix locking = Yes
    strict locking = Auto
    dfree cache time = 0
    preexec close = No
    root preexec close = No
    available = Yes
    fstype = NTFS
    wide links = No
    allow insecure wide links = No
    follow symlinks = Yes
    delete readonly = No
    dos filemode = No
    dos filetimes = Yes
    dos filetime resolution = No
    fake directory create times = No
    host msdfs = Yes
    msdfs root = No
    msdfs shuffle referrals = No
    ntvfs handler = unixuid, default
    vfs objects = dfs_samba4 acl_xattr
    full_audit:prefix = IP=%I|USER=%u|MACHINE=%m|VOLUME=%S
    full_audit:failure = connect disconnect
    full_audit:success = connect disconnect opendir mkdir rmdir closedir open close read pread write pwrite sendfile rename unlink chmod fchmod chown fchown chdir ftruncate lock symlink readlink link mknod
    full_audit:facility = local5
    full_audit:priority = notice
[profiles]
    comment = Network Profiles Share
    path = /mnt/storage/profiles
    profile acls = Yes
    browseable = No
    create mask = 0644
    force create mode = 0660
    force directory mode = 0770
    read only = No
[netlogon]
    comment = Network Netlogon Share
    path = /usr/local/samba/var/locks/sysvol/facility.local/scripts
    read only = No
    guest ok = No
    write ok = Yes
    browseable = No
[sysvol]
    path = /usr/local/samba/var/locks/sysvol
    read only = No
    browseable = No
    write ok = Yes
[printers]
    comment = All Printers
    path = /var/spool/samba
    create mask = 0700
    browseable = yes
    guest ok = no
    printable = yes
    create mode=0700
    write list = administrator "@Domain Admins"
[print$]
    comment = Printer Drivers
    path = /mnt/storage/printer_drivers
    invalid users = qwerty
    valid users = @"Domain Users"
    admin users = @"Domain Admins"                  
    write list = root administrator
    writeable = Yes
    read only = Yes        
    browseable = Yes                 
    guest ok = Yes
    create mask = 0660
    create mask = 0644
    force create mode = 0660
    force directory mode = 0770
    directory mask = 0755    
    acl_xattr:ignore system acl = yes
Comment 30 Sander Plas 2016-09-16 18:02:29 UTC
Any idea how i can identify the 'extra' processes without shutting down samba and looking what's left? ;)

One of the servers just had more than 50 running smbd processes, while the other ones were all < 20. 

So i stopped samba using /etc/init.d/sernet-samba-ad (a leftover from we were using the free sernet packages). 

I now have 54 "stuck" smbd processes. 

"samba-tool processes" shows an empty list. 

One of the processes is running as user 3001439; the other ones as root. 

The processes are not consuming CPU: top shows 0.0% and the perf command example from top the flamegraph site says 'The perf.data file has no samples!'. 

/proc/[pid]/comm on all of them contains 'smbd'. 

"smbcontrol <pid> pool-usage" hangs for a while and then reports "No replies received". 

i'll attach:
- "ls -l /proc/[pid]/fd" output per process
- netstat -avp output
- log files (smb.conf currently has no log level specified)
- gdb bt full output (sorry, no debug symbols yet)

I'm going to increase the log levels and compile with debug symbols. 

These are primary schools and they are closed for the weekend now, so i'll leave this specific server in this state for now, in case someone comes up with other idea on how to get more information out of it.
Comment 31 Sander Plas 2016-09-16 18:04:18 UTC
Created attachment 12468 [details]
netstat -avp output on server 'juliana'
Comment 32 Sander Plas 2016-09-16 18:06:01 UTC
Created attachment 12469 [details]
gdb "bt full" output of stuck smbd processes on server 'juliana'
Comment 33 Sander Plas 2016-09-16 18:17:51 UTC
Andrew, you mentioned "mark as private if need be". How do i do this? The log files and open file descriptors contain some private data like user and file names.
Comment 34 Andrew Bartlett 2016-09-16 18:28:04 UTC
Created attachment 12470 [details]
revert gencache of remote arch

Can you try this patch?

I've only checked that it doesn't cause outright failure in a testenv, but I think it might help.

It reverts the addition of a gencache get and set during logon and logoff.  The backtraces clearly show that we are blocked trying to operate on gencache.tdb, mostly adding or removing this cache item.

This won't be the real bug, but it might buy you time.  There is at least one process locked up in gencache_stablise(), but that won't be as likely to swamp the system. 

I'll include next a patch to remove mutexes from gencache.tdb so we can try and find who is holdling the lock on the whole DB.
Comment 35 Andrew Bartlett 2016-09-16 18:31:25 UTC
Created attachment 12471 [details]
remove muxtex suppot from gencache

This larger hammer may allow us to use OS tools to work out who is currently holding the lock
Comment 36 Andrew Bartlett 2016-09-16 18:38:33 UTC
I don't think the logs will show much, as once we get stuck in a blocking log, we don't do anything until we get it, including log :-)

If you don't see a line like this at the bottom of the add an attachment screen:

Privacy: 	Make attachment and comment private (visible only to members of the samba-devel group) 

Then just mail it to me and I'll look at it and upload with that ticked (as I have more privileges)
Comment 37 Andrew Bartlett 2016-09-16 18:42:41 UTC
Re-assigning as file server bug, as I'm pretty sure the issue is not AD DC specific (as most of the AD DC can't even call gencache, and it is smbd processes blocked from accepting connections or terminating).
Comment 38 Andrew Bartlett 2016-09-16 18:56:44 UTC
(In reply to Sander Plas from comment #30)

BTW, I consider a process stuck if it has no corresponding entry in smbstatus, and/or no active connection to a client.  There are helper processes that are like that, but we can quickly discount them, the backtraces you gave clearly show the issue being in the main client-connected process.
Comment 39 Sander Plas 2016-09-16 19:02:07 UTC
I get this with your second patch: 

[1693/3898] Compiling source3/lib/server_contexts.c
../source3/lib/gencache.c: In function ‘gencache_init’:
../source3/lib/gencache.c:126:11: error: too few arguments to function ‘tdb_wrap_open’
           open_flags, 0644);
           ^
In file included from ../source3/lib/gencache.c:28:0:
../lib/tdb_wrap/tdb_wrap.h:39:18: note: declared here
 struct tdb_wrap *tdb_wrap_open(TALLOC_CTX *mem_ctx,
                  ^
Waf: Leaving directory `/root/samba-4.5.0/bin'
Build failed:  -> task failed (err #1): 
	{task: cc gencache.c -> gencache_58.o}
make: *** [all] Error 1
Comment 40 Andrew Bartlett 2016-09-16 19:05:49 UTC
Created attachment 12472 [details]
remove mutex support from gencache

Sorry about the bung patch.  This one is what I built with, but I gave you the wrong file.
Comment 41 Sander Plas 2016-09-16 20:12:08 UTC
Ok, all servers except the one with the 54 stuck smbd processes are now running with debug symbols, your 2 patches and log level 10. 

I don't expect anything to happen soon as there won't be many users until monday. 



I can't seem to get the per-process logging to work though - this:

log file = /var/log/samba/log.%p

just results in a log file called "/var/log/samba/log.%p". Tried %d too, same result. Am i doing something wrong?
Comment 42 Andrew Bartlett 2016-09-16 21:16:01 UTC
(In reply to Sander Plas from comment #41)
Sorry, use %d to get the PID in the log file.

It will still produce a %d for the 'samba' processes, but the smbd tasks will use it, and that is where it matters.

If you are able to help us by running with both patches on some servers, only the gencache patch on others and only the mutex patch on others still, that may be more informative, as I strongly suspect that problem will 'go away' enough with the both that we never really understand it.  

Naturally you also have a production env to run, so make your judgements accordingly.  

I'll upload your logs as private in a moment.
Comment 43 Jeremy Allison 2016-09-16 21:31:18 UTC
As a quick look this looks like a deadlock problem.

12831.bt has:

#2  0x00007f86b2984b42 in __pthread_mutex_lock_full (mutex=0x7f86b2fc80a8)
    at ../nptl/pthread_mutex_lock.c:258
        result = <optimized out>
        oldval = <optimized out>
        id = 12831
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock_full"
#3  0x00007f86ac810f04 in allrecord_mutex_lock ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#4  0x00007f86ac8110fb in tdb_mutex_lock ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#5  0x00007f86ac805e23 in fcntl_lock ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#6  0x00007f86ac805fb4 in tdb_brlock ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#7  0x00007f86ac8064ef in tdb_nest_lock ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#8  0x00007f86ac80673b in tdb_lock_list ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#9  0x00007f86ac8067d8 in tdb_lock ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#10 0x00007f86ac80353e in tdb_find_lock_hash ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#11 0x00007f86ac8038c6 in tdb_parse_record ()
   from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#12 0x00007f86afc8c62d in gencache_set_data_blob ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#13 0x00007f86afc8d687 in gencache_set ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#14 0x00007f86afc8cb5d in gencache_parse ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#15 0x00007f86afc8c7bc in gencache_del ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#16 0x00007f86afc71175 in remote_arch_cache_delete ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#17 0x00007f86b22c1ff4 in smbd_smb2_logoff_shutdown_done ()
   from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#18 0x00007f86b1703377 in _tevent_req_notify_callback ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#19 0x00007f86b170344a in tevent_req_finish ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#20 0x00007f86b170356f in tevent_req_trigger ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#21 0x00007f86b170278c in tevent_common_loop_immediate ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#22 0x00007f86afc8e10f in run_events_poll ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#23 0x00007f86afc8e783 in s3_event_loop_once ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#24 0x00007f86b17018d3 in _tevent_loop_once ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#25 0x00007f86b1701b1d in tevent_common_loop_wait ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#26 0x00007f86b1701be8 in _tevent_loop_wait ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#27 0x00007f86b229eaa3 in smbd_process ()
   from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#28 0x00007f86b2dcf0be in smbd_accept_connection ()
No symbol table info available.
#29 0x00007f86afc8e608 in run_events_poll ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#30 0x00007f86afc8e894 in s3_event_loop_once ()
   from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#31 0x00007f86b17018d3 in _tevent_loop_once ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#32 0x00007f86b1701b1d in tevent_common_loop_wait ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#33 0x00007f86b1701be8 in _tevent_loop_wait ()
   from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#34 0x00007f86b2dcfe35 in smbd_parent_loop ()

all the others seem to be (note the difference in stack trace, the below is doing a chainlock, the above is not:

#2  0x00007f86b2984b42 in __pthread_mutex_lock_full (mutex=0x7f86b2fc80a8) at ../nptl/pthread_mutex_lock.c:258
        result = <optimized out>
        oldval = <optimized out>
        id = 26202
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock_full"
#3  0x00007f86ac810f04 in allrecord_mutex_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#4  0x00007f86ac8110fb in tdb_mutex_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#5  0x00007f86ac805e23 in fcntl_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#6  0x00007f86ac805fb4 in tdb_brlock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#7  0x00007f86ac8064ef in tdb_nest_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#8  0x00007f86ac80673b in tdb_lock_list () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#9  0x00007f86ac8067d8 in tdb_lock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#10 0x00007f86ac8071d3 in tdb_chainlock () from /usr/lib/x86_64-linux-gnu/private/libtdb.so.1
No symbol table info available.
#11 0x00007f86afc8ca8e in gencache_parse () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#12 0x00007f86afc70f5e in remote_arch_cache_get () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#13 0x00007f86afc710df in remote_arch_cache_update () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#14 0x00007f86b22bcf26 in smbd_smb2_request_process_negprot () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#15 0x00007f86b22b7e79 in smbd_smb2_request_dispatch () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#16 0x00007f86b22bb118 in smbd_smb2_process_negprot () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#17 0x00007f86b2298b9a in process_smb () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#18 0x00007f86b229a22f in smbd_server_connection_read_handler () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#19 0x00007f86b229a310 in smbd_server_connection_handler () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#20 0x00007f86afc8e608 in run_events_poll () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#21 0x00007f86afc8e894 in s3_event_loop_once () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#22 0x00007f86b17018d3 in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#23 0x00007f86b1701b1d in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#24 0x00007f86b1701be8 in _tevent_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#25 0x00007f86b229eaa3 in smbd_process () from /usr/lib/x86_64-linux-gnu/private/libsmbd-base-samba4.so
No symbol table info available.
#26 0x00007f86b2dcf0be in smbd_accept_connection ()
No symbol table info available.
#27 0x00007f86afc8e608 in run_events_poll () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#28 0x00007f86afc8e894 in s3_event_loop_once () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
No symbol table info available.
#29 0x00007f86b17018d3 in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#30 0x00007f86b1701b1d in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#31 0x00007f86b1701be8 in _tevent_loop_wait () from /usr/lib/x86_64-linux-gnu/private/libtevent.so.0
No symbol table info available.
#32 0x00007f86b2dcfe35 in smbd_parent_loop ()
No symbol table info available.
#33 0x00007f86b2dd1800 in main ()
Comment 44 Andrew Bartlett 2016-09-16 21:38:25 UTC
Question: Are you running the new gpgme code for password sync?

I see a core file in there.  Potentially if mutexes are not as robust as we expect, a crash happens while holding the mutex.

Core was generated by `/usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fa9303e4c37 in ?? ()

Can you:
 - set max log size = 0 (so we don't rotate the log)
 - get me the gdb backtrace from the core file

Finally, a tip:  when sending large logs, 'tar --xz -cf logs.tar.xz logs/' makes really, really tiny files because it finds all the duplicates much better than zip.

I'll upload the private logs in a moment.
Comment 46 Sander Plas 2016-09-17 10:15:18 UTC
The new gpgme code needs to be enabled explicitly, right? In that case, no. 

This is what i use; all the explicit paths are there to make it compatible with the old sernet packages. 

./configure --sbindir=/usr/sbin --bindir=/usr/bin --with-configdir=/etc/samba --with-logfilebase=/var/log/samba --libdir=/usr/lib/x86_64-linux-gnu --with-modulesdir=/usr/lib/x86_64-linux-gnu/samba --with-lockdir=/var/cache/samba --with-statedir=/var/lib/samba --with-cachedir=/var/cache/samba --with-piddir=/var/run/samba --with-privatedir=/var/lib/samba/private --enable-debug

I'm now running 3 servers with just the first patch, 3 with just the other and 4 with both. 

Can i delete logs for pid's that are no longer running? Some of the servers are low on disk space.
Comment 47 Sander Plas 2016-09-17 10:29:43 UTC
(In reply to Andrew Bartlett from comment #44)
The core is from 2016-07-03, so it must be from a previous version of samba. The oldest smbd binary i can find in our backups of this server is from 2016-07-08. 

"bt full" (same output with the smbd binary from the backup and the current one) : 

(gdb) bt full
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7ffef3733548: 
#0  0x00007fa9303e4c37 in ?? ()
No symbol table info available.
Cannot access memory at address 0x7ffef3733548
Comment 48 Andrew Bartlett 2016-09-17 18:14:41 UTC
(In reply to Sander Plas from comment #46)
Yes, I think it is reasonable to delete logs of PIDs that exit successfully.

I also think Jeremy is onto something regarding a deadlock - I've not tried to get my head into tdb locking enough to understand why we can't make progress from here, but I know this:

Many Samba Team members will be in a lab at SNIA SDC next Monday US time, and you now have the attention of those who can sit together and fix this.  

I really appreciate your assistance with getting us the debug info we need, we simply were not able to make progress without it.
Comment 49 Andrew Bartlett 2016-09-17 18:29:28 UTC
(In reply to Andrew Bartlett from comment #44)
The SIGABRT in the core file could be from: https://bugzilla.redhat.com/show_bug.cgi?id=1375973
Comment 50 Sander Plas 2016-09-19 07:16:35 UTC
This morning at 8 (Netherlands time) we were called by 4 of the schools who said that they were having the problem. 

Strangely, none of them had the typical "stuck smbd processes" symptom. 

One of the servers, running both patches, was just completely unresponsive. They are going to powercycle it. 

Another one, only running "patch 1" aka the gencache patch, did not have many smbd processes running and all smbd and samba processes exited without any problems using the init.d script. 

The third server (vondel), only had a stuck samba (not smbd) process. Apparently this one was already stuck for days since saturday when i recompiled & restarted samba on this server. I only checked for stuck smbd processes then. GDB complains about the deleted binary. This is another one of those "Python Exception <class 'gdb.MemoryError'> Cannot access memory at address" cases. 

The last server (cray-dep), running both patches, neither had any stuck smbd processes, but it did have one stuck samba process. 

I'll attach logs for vondel and cray-dep.
Comment 51 Sander Plas 2016-09-19 07:23:28 UTC
Created attachment 12478 [details]
cray-dep backtrace
Comment 52 Sander Plas 2016-09-19 07:23:42 UTC
Created attachment 12479 [details]
cray-dep netstat -avp
Comment 53 Sander Plas 2016-09-19 07:24:03 UTC
Created attachment 12480 [details]
cray-dep file descriptors
Comment 54 Sander Plas 2016-09-19 07:24:18 UTC
Created attachment 12481 [details]
cray-dep samba-tool processes
Comment 55 Sander Plas 2016-09-19 07:25:31 UTC
Created attachment 12482 [details]
vondel backtrace
Comment 56 Sander Plas 2016-09-19 07:25:43 UTC
Created attachment 12483 [details]
vondel netstat -avp
Comment 57 Sander Plas 2016-09-19 07:25:56 UTC
Created attachment 12484 [details]
vondel file descriptors
Comment 58 Sander Plas 2016-09-19 07:26:10 UTC
Created attachment 12485 [details]
vondel samba-tool processes
Comment 59 Sander Plas 2016-09-19 09:33:18 UTC
Another crash on server 'juliana'. looks like a 'classic' one with 32 stuck smbd processes. This one is still running without any patches, but with log level 10 and logging per pid.
Comment 60 Sander Plas 2016-09-19 09:33:49 UTC
Created attachment 12486 [details]
netstat -avp output on server 'juliana' ; 2nd crash
Comment 61 Sander Plas 2016-09-19 09:34:42 UTC
Created attachment 12487 [details]
gdb "bt full" output of stuck smbd processes on server 'juliana' ; 2nd crash
Comment 62 Andrew Bartlett 2016-09-19 11:14:49 UTC
(In reply to Sander Plas from comment #50)

> The last server (cray-dep), running both patches, neither had any stuck smbd 
> processes, but it did have one stuck samba process.

This server has a segfault in a samba process.  

Can you set:  	

panic action = path/to/gdb_backtrace %d

This is in selftest/gdb_backtrace in the source tree.

As for some reason your build doesn't include the internal backtrace handler (and gdb is better anyway).

Other than that, if you can get me level 10 logs on juliana?

The backtrace on cray-dep is truly weird: A failure getting a lock to allocate memory?  Is there any chance of a full backtrace, as this one is stuck at only the first page.

I'm also quite concerned about the impact of all this on production, and I'm wondering if the 'stuck samba process' thing is a different bug entirely.  Is that process busy, or has it consumed a lot of memory?
Comment 63 Sander Plas 2016-09-19 11:44:53 UTC
Ok, added "panic action" setting to all servers. 

I'm sorry for the incomplete backtrace on cray-dep. Unfortunately i already killed the process, so no chance to get a full backtrace. The process was not consuming a lot of cpu or memory. 

The juliana logs are huge - still 14 GB after removing all the non needed stuff. The tar.xz job is running, but it will probably take a while.
Comment 64 baris tombul 2016-09-19 13:09:23 UTC
Two days I've been using the "Version 4.5.1-GIT-dbbf8dc" version. I encounter problems.
Comment 65 Sander Plas 2016-09-19 13:59:42 UTC
Another stuck 'samba' process. this one is on the server 'stopoz-pdc', which is a central server in a data center. It has a remote desktop server as a client. 

samba-tool processes = empty
/proc/31194/comm = samba
smbcontrol 31194 pool-usage = No replies received

I'll attach more information. 

I've sent the private juliana logs to Andrew and will send the stopoz-pdc ones too. 

I don't know how soon he will be able to upload them though, so if another team member wants to have them, just let me know and i'll email them.
Comment 66 Sander Plas 2016-09-19 14:00:35 UTC
Created attachment 12490 [details]
stopoz-pdc samba process backtrace
Comment 67 Sander Plas 2016-09-19 14:01:24 UTC
Created attachment 12491 [details]
stopoz-pdc samba process file descriptors
Comment 68 Andrew Bartlett 2016-09-19 15:20:19 UTC
(In reply to Sander Plas from comment #66)
This looks just like a busy process that is trying to print a lot of replication traffic out into the logs.  Unless it is really stuck in exactly that lock for a long time, I'm just going to assume that it is the load of level 10 and some not-so-efficient debugging code.

It is probably time to turn the log level back down, I think we have what we need.  The folks at SNIA SDC will be getting into the lab soon, so I hope they can make some sense of things.

The backtraces are the most important part.  Thanks!
Comment 69 Volker Lendecke 2016-09-19 21:33:09 UTC
Created attachment 12494 [details]
Patch
Comment 70 Jeremy Allison 2016-09-19 21:47:26 UTC
Comment on attachment 12494 [details]
Patch

5 minutes looking at it in the plugfest, Volker found the lock order violation :-).

Pushing to master now - will create backports for the bug.

Longer answer - inside gencache_set() we take a chainlock on the record, but inside gencache_set_data_blob() called from gencache_set() on every 100 writes we call gencache_stabilize().

Inside gencache_stabilize() the old code called : tdb_lockall() - needs to be tdb_lockall_nonblock() to prevent lock ordering violations (we alredy hold the chainlock).

Checking into the back traces from the entire set of processes Volker found one process inside gencache_stabilize() - which gave it away !
Comment 71 Sander Plas 2016-09-20 07:13:37 UTC
Wow, that's great! I'm going to try the patch and let you know the results. 

THANK YOU Volker, Jeremy and Andrew!
Comment 72 Jeremy Allison 2016-09-20 15:55:18 UTC
Created attachment 12499 [details]
git-am fix for 4.5.next, 4.4.next, 4.3.next.

OK, here's the cherry-pick from what went into master. Applies cleanly to all supported branches !
Comment 73 Volker Lendecke 2016-09-20 16:17:51 UTC
Comment on attachment 12499 [details]
git-am fix for 4.5.next, 4.4.next, 4.3.next.

This patch is good, but before we mark the bug fixed, I'd prefer the bug reporter to test.
Comment 74 baris tombul 2016-09-20 19:28:47 UTC
thanks for your attention.
I'm trying all the samba-test(git pull) suite regularly.
now i 'm testing the patch.
If there are problems , I will inform you..
# samba -V
Version 4.5.1-GIT-dbbf8dc
Kind Regards...
Comment 75 Sander Plas 2016-09-21 07:51:55 UTC
24 hours without a single crash now. 

This is looking really good, although i must admit that we did have one day without any crashes last week too. 

So, if it stays up for another full day i'd be pretty confident that the bug really is fixed.
Comment 76 baris tombul 2016-09-21 12:20:43 UTC

samba -V 

Version 4.5.1-GIT-dbbf8dc


This patch ' after the body was


samba_dnsupdate --verbose --all-names

dns_tkey_gssnegotiate: TKEY is unacceptable 
dns_tkey_gssnegotiate: TKEY is unacceptable 
dns_tkey_gssnegotiate: TKEY is unacceptable
Comment 77 baris tombul 2016-09-21 16:45:48 UTC
(In reply to baris tombul from comment #76)


You do not have my previous message . sorry. false flag.
Comment 78 Sander Plas 2016-09-21 18:14:32 UTC
Another day without any crashes. I think the issue has been solved!
Comment 79 Jeremy Allison 2016-09-21 18:35:15 UTC
Confirmation from bug reporter. Karolin please push to 4.5.next, 4.4.next, 4.3.next.

Thanks !

Jeremy.
Comment 80 Karolin Seeger 2016-09-22 08:30:44 UTC
(In reply to Jeremy Allison from comment #79)
Pushed to autobuild-v4-{5,4,3}-test.
Comment 81 Karolin Seeger 2016-10-18 07:47:48 UTC
(In reply to Karolin Seeger from comment #80)
Pushed to all branches.
Closing out bug report.

Thanks!
Comment 82 Andrew Bartlett 2017-06-01 20:15:50 UTC
Giving this a title was can find again.

The issue is traced to a deadlock in gencache_stabilize()