For some odd reason on my FreeBSD system the number of open files (I assume these are memory addresses because they aren't open filehandles, but appear to be memory addresses) on my FreeBSD 6.1 system with open with smbd grow out of control, and I am forced to either restart the daemon or the entire machine as the number of open 'filehandles' reaches the system maximum (~8000 if memory serves me correctly). Prior to restarting the machine I checked to see how many open filehandles there were and lsof | grep smbd | wc reported 4731 different lines with smbd on them, so some resources aren't being allocated and freed properly. Will try and track down the bug; this is definitely an issue as it makes my system inoperable after a period of time. --More information-- smb.conf: [root@hoover /usr/ports/distfiles]# cat /usr/local/etc/smb.conf [global] workgroup = WORKGROUP encrypt passwords = yes log file = /var/log/samba/log.%m log level = 2 passdb:5 auth:5 # log level = 5 socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192 local master = yes preferred master = yes dns proxy = no guest ok = yes [shared] path = /shared writeable = yes public = yes hosts deny = shiina pinocchio # hosts allow = sexbox guest ok = yes create mask = 0775 [www] path = /usr/local/www/apache22/data writeable = yes public = yes create mask = 0755 guest ok = no [clean] path = /clean writeable = yes public = yes guest ok = yes create mask = 0744 #[dvd] # path = /cdrom # writeable = no # public = yes # create mask = 0000 FreeBSD OS version: [root@hoover /usr/ports/distfiles]# uname -a FreeBSD hoover.localdomain 6.1-RELEASE-p10 FreeBSD 6.1-RELEASE-p10 #9: Mon Oct 16 02:14:29 PDT 2006 gcooper@hoover.localdomain:/usr/obj/usr/src/sys/HOOVER i386 CFLAGS/arch definition: [root@hoover /usr/ports/distfiles]# cat /etc/make.conf #snip CFLAGS= -O2 -pipe CPUTYPE?=pentium3 #/snip make config output: lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x Options for samba 3.0.23c_2,1 x x lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk x x x [ ] LDAP With LDAP support x x x x [ ] ADS With Active Directory support x x x x [ ] CUPS With CUPS printing support x x x x [ ] WINBIND With WinBIND support x x x x [ ] ACL_SUPPORT With ACL support x x x x [ ] AIO_SUPPORT With experimental AIO support x x x x [ ] FAM_SUPPORT With File Alteration Monitor x x x x [X] SYSLOG With Syslog support x x x x [ ] QUOTAS With Disk quota support x x x x [X] UTMP With UTMP accounting support x x x x [ ] MSDFS With MSDFS support x x x x [ ] SMBSH With SMBSH wrapper for UNIX commands x x x x [X] PAM_SMBPASS With PAM authentication vs passdb backends x x x x [ ] EXP_MODULES With experimental modules x x x x [ ] POPT With system-wide POPT library x x tqmqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqjqu x [ OK ] Cancel x mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
Issue appears to not be directly related to the number of open sockets because there were a number of sockets left open on my machine, as of late. The only machines connecting to my FreeBSD box with Samba are 3 XP-SP2 clients with (I believe) the latest updated patches. Will post additional information later (attachments from lsof and netstat).
Created attachment 2205 [details] Netstat output of all open sockets prior to smbd/nmbd 'restart'
Created attachment 2206 [details] Output containing the running processes for smbd under review
Created attachment 2207 [details] partial lsof output, grepping smbd, with wc output listed at bottom
Created attachment 2208 [details] lsof output after smbd/nmbd restart
Ok, now to help interpret what I've provided, logs wise. The netstat output lists a snippet of all of the open sockets, and a word count of all of the open Unix sockets down at the bottom. The ps aux output is given to help provide a basis for what processes were running at the time that were related to smbd. The partial lsof output is the important one. If you note, there are a number of virtual files open, which almost directly relates to the number of sockets; the discrepancy seen in the number of sockets listed in netstat differs most likely because a few other daemons had open sockets as well. And, the most interesting output is when I restarted smbd/nmbd ("lsof output after.."). The point is that the number of files as well as the number of open sockets decreased to near zero, following the daemon restarts. So, it appears that the issue is being created by something not terminating sockets (XP SP2 clients?), so I will look into the code a bit and see if there is an apparent issue with socket creation / deletion and timeouts..
Created attachment 2209 [details] lsof/netstat output after smbd/nmbd restart
Created attachment 2210 [details] source/smbd/server.c patch This patch just makes some things more readible, and I moved some initializations outside of for loops and commented out some if -> continue statements, to--I believe--help eliminate some repetitive initializations and better use CPU cycles. Also--and this is the 'major' code change--I took some of the code inside open_sockets_smbd(..) and moved it into parse_ports_and_listen(..) [my function], in an effort to make things a lot more readible and reduce redundant code through an additional function call.
Created attachment 2211 [details] Fixes SAFE_FREE misplacement I accidentally misplaced SAFE_FREE(ports), thinking that it was better to do things that way in the function.. After thing over things, I left it where it was supposed to be in the first place--after the if-else statement.
Crazy question.. why in the heck isn't num_sockets decremented or the value changed after freeing the file descriptors? Should be done that way, unless one simply wants to keep the file descriptors open after the sockets are closed, which is an unneeded waste of resources.. Line: if (allowable_number_of_smbd_processes() && smbd_server_fd() != -1 && sys_fork()==0) { /* Child code ... */ /* close the listening socket(s) */ for(i = 0; i < num_sockets; i++) close(fd_listenset[i]); /* close our standard file descriptors */ close_low_fds(False); am_parent = 0; According to my grep output, the only references to num_sockets are here: gcooper@sprsd-xp /cygdrive/d/samba-3.0.23c/source $ grep -r num_sockets */* smbd/server.c:static BOOL parse_ports_and_listen(const int sock_addr, char* ports, fdset *listenset, int *fd_listenset, int *num_sockets) { smbd/server.c: // "fd_listenset[num_sockets] = s" equivalent below smbd/server.c: *(fd_listenset+(*num_sockets)) = s; smbd/server.c: (*num_sockets)++; smbd/server.c: if (*num_sockets >= FD_SETSIZE) { smbd/server.c: int num_sockets = 0; smbd/server.c: &num_sockets) smbd/server.c: &num_sockets) smbd/server.c: for(i = 0; i < num_sockets; i++) { smbd/server.c: for(i = 0; i < num_sockets; i++) smbd/server.orig: int num_sockets = 0; smbd/server.orig: s = fd_listenset[num_sockets] = open_socket_in(SOCK_STREAM, port, 0, ifip->s_addr, True); smbd/server.orig: num_sockets++; smbd/server.orig: if (num_sockets >= FD_SETSIZE) { smbd/server.orig: fd_listenset[num_sockets] = s; smbd/server.orig: num_sockets++; smbd/server.orig: if (num_sockets >= FD_SETSIZE) { smbd/server.orig: for(i = 0; i < num_sockets; i++) { smbd/server.orig: for(i = 0; i < num_sockets; i++) Furthermore, the only location where fd_listenset and listenset are affected are in server.c (server.orig is the stable version of server.c). grep output listed below: gcooper@sprsd-xp /cygdrive/d/samba-3.0.23c/source $ grep -r fd_listenset */* smbd/server.c:static BOOL parse_ports_and_listen(const int sock_addr, char* ports, fdset *listenset, int *fd_listenset, int *num_sockets) { smbd/server.c: // "fd_listenset[num_sockets] = s" equivalent below smbd/server.c: *(fd_listenset+(*num_sockets)) = s; smbd/server.c: int fd_listenset[FD_SETSIZE]; smbd/server.c: &fd_listenset[0], smbd/server.c: &fd_listenset[0], smbd/server.c: if(FD_ISSET(fd_listenset[i], &lfds)) { smbd/server.c: s = fd_listenset[i]; smbd/server.c: FD_CLR(fd_listenset[i],&lfds); smbd/server.c: close(fd_listenset[i]); smbd/server.orig: int fd_listenset[FD_SETSIZE]; smbd/server.orig: s = fd_listenset[num_sockets] = open_socket_in(SOCK_STREAM, port, 0, ifip->s_addr, True); smbd/server.orig: fd_listenset[num_sockets] = s; smbd/server.orig: if(FD_ISSET(fd_listenset[i],&lfds)) { smbd/server.orig: s = fd_listenset[i]; smbd/server.orig: FD_CLR(fd_listenset[i],&lfds); smbd/server.orig: close(fd_listenset[i]); gcooper@sprsd-xp /cygdrive/d/samba-3.0.23c/source $ grep -r listenset */* smbd/server.c:static BOOL parse_ports_and_listen(const int sock_addr, char* ports, fdset *listenset, int *fd_listenset, int *num_sockets) { smbd/server.c: // "fd_listenset[num_sockets] = s" equivalent below smbd/server.c: *(fd_listenset+(*num_sockets)) = s; smbd/server.c: int fd_listenset[FD_SETSIZE]; smbd/server.c: &fd_listenset[0], smbd/server.c: &fd_listenset[0], smbd/server.c: if(FD_ISSET(fd_listenset[i], &lfds)) { smbd/server.c: s = fd_listenset[i]; smbd/server.c: FD_CLR(fd_listenset[i],&lfds); smbd/server.c: close(fd_listenset[i]); smbd/server.orig: int fd_listenset[FD_SETSIZE]; smbd/server.orig: s = fd_listenset[num_sockets] = open_socket_in(SOCK_STREAM, port, 0, ifip->s_addr, True); smbd/server.orig: fd_listenset[num_sockets] = s; smbd/server.orig: if(FD_ISSET(fd_listenset[i],&lfds)) { smbd/server.orig: s = fd_listenset[i]; smbd/server.orig: FD_CLR(fd_listenset[i],&lfds); smbd/server.orig: close(fd_listenset[i]); gcooper@sprsd-xp /cygdrive/d/samba-3.0.23c/source $
Should be done -> Shouldn't be done
Created attachment 2212 [details] grep output for fd_listenset
Created attachment 2213 [details] grep output for listenset These two files are readible textfiles of my grep output.
Created attachment 2214 [details] New smb.conf -- seems like it causes less resource issues with filehandles Changed a few options in smb.conf and it seems to be holding smbd's resource usage down much better now. I've attached my current smb.conf just to provide a reference for a properly working smbd--not one that is out of control with filehandles.
Created attachment 2215 [details] Final version of the patch--created for readability, clarity, and modularity Nevermind about the closing sockets comment. It appears that the issue stemmed partially from a low maxfiles setting for the kernel and a bad login.conf file (mistook filehandles for maximum process number). Problem has been fixed and there isn't a problem with samba that I can tell.. This patch is a final version, guaranteed to compile and link, where I've made the diff from the base of the source tree so injecting it straight into the source will be more simplified than the patch I provided earlier. You can either choose to add it or refuse it.
Marking invalid since the issue with lack of resources appears to have been caused by invalid maximums set in /etc/login.conf. Also, after reading the smbd/server.c file it appears that smbd is supposed to 'dynamically' consume sockets, up to the maximum number of file descriptors openable by the daemon user (root?). The only thing I can suggest is that maybe this value should be set to the real user, via root's limits via setuid?
*real user's limits, not root's limits via setuid.