The reason for the crash is not known. The crash is not reproduceable, but happens quite often. There is no damage in user data, users do not even notice the crash. Excerpt from logging, with log level 3: [2007/03/07 13:48:10, 3] rpc_server/srv_pipe.c:api_rpcTNP(2265) api_rpcTNP: rpc command: LSA_LOOKUPSIDS2 [2007/03/07 13:48:10, 0] lib/fault.c:fault_report(41) =============================================================== [2007/03/07 13:48:10, 0] lib/fault.c:fault_report(42) INTERNAL ERROR: Signal 6 in pid 31097 (3.0.24-0.1.49-1172-SUSE-SL10.2) Please read the Trouble-Shooting section of the Samba3-HOWTO [2007/03/07 13:48:10, 0] lib/fault.c:fault_report(44) From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf [2007/03/07 13:48:10, 0] lib/fault.c:fault_report(45) =============================================================== [2007/03/07 13:48:10, 0] lib/util.c:smb_panic(1599) PANIC (pid 31097): internal error [2007/03/07 13:48:10, 0] lib/util.c:log_stack_trace(1706) BACKTRACE: 24 stack frames: #0 /usr/sbin/smbd(log_stack_trace+0x2d) [0x802165fd] #1 /usr/sbin/smbd(smb_panic+0x5d) [0x8021672d] #2 /usr/sbin/smbd [0x802024ea] #3 [0xb7fd7420] #4 /lib/libc.so.6(abort+0x101) [0xb7b2e801] #5 /usr/sbin/smbd [0x8021c9d6] #6 /usr/sbin/smbd(talloc_steal+0x3a) [0x8021cbaa] #7 /usr/sbin/smbd(lookup_sids+0x3aa) [0x801d3baa] #8 /usr/sbin/smbd [0x80106da1] #9 /usr/sbin/smbd(_lsa_lookup_sids2+0x12d) [0x801073fd] #10 /usr/sbin/smbd [0x80103601] #11 /usr/sbin/smbd(api_rpcTNP+0x198) [0x80160ac8] #12 /usr/sbin/smbd(api_pipe_request+0x19e) [0x8016111e] #13 /usr/sbin/smbd [0x8015cbaf] #14 /usr/sbin/smbd [0x8015d0ec] #15 /usr/sbin/smbd(write_to_pipe+0x6e) [0x8015b91e] #16 /usr/sbin/smbd [0x8005ee81] #17 /usr/sbin/smbd [0x8005f3b6] #18 /usr/sbin/smbd(reply_trans+0x5e0) [0x8005fcc0] #19 /usr/sbin/smbd [0x800b3d20] #20 /usr/sbin/smbd(smbd_process+0x78b) [0x800b4e1b] #21 /usr/sbin/smbd(main+0xbd0) [0x802bebe0] #22 /lib/libc.so.6(__libc_start_main+0xdc) [0xb7b19f9c] #23 /usr/sbin/smbd [0x80042d91] [2007/03/07 13:48:10, 0] lib/fault.c:dump_core(168) unable to change to /var/log/samba/cores/smbdrefusing to dump core Configuration: [global] unix charset = ISO-8859-1 workgroup = HELLER realm = HELLER.BIZ security = ADS client schannel = No obey pam restrictions = Yes password server = hntads01, hntads02, * log level = 1 log file = /var/log/samba/%m max log size = 1000 disable netbios = Yes deadtime = 15 socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192 load printers = No printcap name = /dev/null lm announce = No preferred master = No local master = No domain master = No wins server = 10.1.3.83, 10.1.3.84 oplock break wait time = 100 idmap uid = 10000-20000 idmap gid = 10000-20000 template homedir = /l1/user/%U template shell = /bin/bash winbind use default domain = Yes winbind refresh tickets = Yes winbind offline logon = Yes invalid users = root create mask = 0666 directory mask = 0777 printing = bsd print command = lpr -r -P'%p' %s lpq command = lpq -P'%p' lprm command = lprm -P'%p' %j include = /etc/samba/.conf wide links = No dos filetime resolution = Yes fake directory create times = Yes [homes] comment = Home Directory of %u valid users = heller\%S force group = www read only = No create mask = 0660 directory mask = 0770 browseable = No
Created attachment 2316 [details] smbd log file
Could you create per-client logfiles with log file = /var/log/samba/log.%m and for a short test period max log size = 0 please? The logfile seems incomplete. Thanks, Volker
I have configured per-client log files. There are about 120 clients, and only 4 clients are producing smbd crashes. For these 4 clients I set the log level to 3. The 4 clients produce about 300 crashes per week. The smbd log file that I uploaded as attachment is an excerpt of a large log file. But this excerpt includes all lines for one smbd process from start to termination. So this logging is complete.
Ah, ok. debug level 3 is not enough I'm afraid. We need debug level 10. Volker
Created attachment 2317 [details] smbd log with level 10
Ok, better. Thanks. I'll look at it later today. Volker
I'm very sorry, but I don't see any code path in the plain 3.0.24 code that could lead to this kind of crash. Can you recompile your Samba with debugging symbols and possibly run it under valgrind? And, some lines in your log file indicate that you might not run the plain samba.org code. Can you retry with a self-compiled Samba? Thanks, Volker
I did not compile my Samba. I installed the package samba-3.0.24-0.1.49.i586.rpm that I got from http://ftp.suse.com/pub/projects/samba/3.0/10.0/i386/ Whether this is "the plain samba.org code", I do not know. When I find the time next week I can install plain samba and see what happens.
You can install the samba-debuginfo package from the same ftp server, that way you would have debug symbols. Further debugging/backtrace info can be found at: http://en.opensuse.org/Samba#Backtraces
Created attachment 2323 [details] smbd log New versions installed: samba-winbind-3.0.24-5.1.52 samba-3.0.24-5.1.52 samba-debuginfo-3.0.24-5.1.52 samba-doc-3.0.24-6.52 samba-client-3.0.24-5.1.52 smb.conf: panic action defined panic action = /usr/share/doc/packages/samba/examples/scripts/debugging/linux/backtrace %d New smbd log file produced. But gdb did not work as it should.
Created attachment 2324 [details] new backtrace script The script /usr/share/doc/packages/samba/examples/scripts/debugging/linux/backtrace does not work. So I made this simple script, which uses a setuid root copy of gdb, which is named sugdb.
Created attachment 2325 [details] smbd log (excerpt) Debugging smbd is still not possible. ptrace now works. But now: "No symbol table info available."
Created attachment 2326 [details] smbd log (excerpt) backtrace is useless. So I tried to get a core file. I put "ulimit -Sc 20000" into /etc/init.d/smb. chmod 755 /var/log/samba; chmod 777 /var/log/samba/cores; chmod 777 /var/log/samba/cores/smbd; Now smbd logs "dumping core in /var/log/samba/cores/smbd" But there is no core file!
Lars, can you take a look at the backtrace stuff?
I made a new backtrace script: #! /bin/sh exec 1>&2 cd /var/log/samba ulimit -Sc 50000 PID=$1 cat << EOF > /tmp/backtrace"${PID}" generate-core-file quit EOF /usr/bin/sugdb -x /tmp/backtrace"${PID}" /usr/sbin/smbd "${PID}" /bin/rm -f /tmp/backtrace"${PID}" This script was able to produce a core file. I used gdb to look into it. 41# gdb /usr/sbin/smbd core.746 GNU gdb 6.5 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i586-suse-linux"...Using host libthread_db library "/lib/libthread_db.so.1". warning: Can't read pathname for load map: Input/output error. Loaded symbols for /usr/sbin/smbd Reading symbols from /usr/lib/libldap-2.2.so.7...done. Loaded symbols for /usr/lib/libldap-2.2.so.7 ...... Reading symbols from <many libraries> ...... Reading symbols from /lib/libnss_winbind.so.2...Reading symbols from /usr/lib/debug/lib/libnss_winbind.so.2.debug...done. done. Loaded symbols for /lib/libnss_winbind.so.2 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Failed to read a valid object file image from memory. Core was generated by `/usr/sbin/smbd'. #0 0xb7f53410 in ?? () (gdb) set height 0 (gdb) up 8 #2 0x00000000 in ?? () (gdb) bt full #0 0xb7f53410 in ?? () No symbol table info available. #1 0xbfab8318 in ?? () No symbol table info available. #2 0x00000000 in ?? () No symbol table info available. (gdb) where #0 0xb7f53410 in ?? () #1 0xbfab8318 in ?? () #2 0x00000000 in ?? () (gdb) quit 42# So there are problems with symbol tables. And no useful backtrace data. Shall I upload the core file (13MB)?
My valgrind arguments: valgrind --tool=memcheck --leak-check=yes -v --num-callers=20 --trace-children=yes Volker
I try to get a smbd with included debug information. I did not find a samba-3.0.24-5.1.52.src.rpm. But I found a samba-3.0.24-8.src.rpm at http://download.opensuse.org/distribution/SL-OSS-factory/inst-source/suse/src/. I installed this. "rpmbuild -bp SPECS/samba.spec" was okay. "rpmbuild -bc SPECS/samba.spec" produced errors: + /usr/bin/make -C tdb tdbtest tdbtorture make: Entering directory `/usr/src/packages/BUILD/samba-3.0.24/source/tdb' gcc -DSTANDALONE -DTDB_DEBUG -g -DHAVE_MMAP=1 -c -o tdbtest.o tdbtest.c tdbtest.c:13:18: error: gdbm.h: No such file or directory tdbtest.c:23: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'gdbm' tdbtest.c: In function 'compare_db': tdbtest.c:58: error: 'datum' undeclared (first use in this function) tdbtest.c:58: error: (Each undeclared identifier is reported only once tdbtest.c:58: error: for each function it appears in.) tdbtest.c:58: error: expected ';' before 'gd' tdbtest.c:63: error: 'gkey' undeclared (first use in this function) tdbtest.c:66: error: 'gd' undeclared (first use in this function) tdbtest.c:66: error: 'gdbm' undeclared (first use in this function) tdbtest.c:95: error: 'gnextkey' undeclared (first use in this function) tdbtest.c: In function 'addrec_gdbm': tdbtest.c:153: error: 'datum' undeclared (first use in this function) tdbtest.c:153: error: expected ';' before 'key' tdbtest.c:161: error: 'key' undeclared (first use in this function) tdbtest.c:164: error: 'data' undeclared (first use in this function) tdbtest.c:168: error: 'gdbm' undeclared (first use in this function) tdbtest.c:170: error: 'GDBM_REPLACE' undeclared (first use in this function) tdbtest.c: In function 'main': tdbtest.c:231: error: 'gdbm' undeclared (first use in this function) tdbtest.c:231: error: 'GDBM_WRITER' undeclared (first use in this function) tdbtest.c:231: error: 'GDBM_NEWDB' undeclared (first use in this function) tdbtest.c:231: error: 'GDBM_FAST' undeclared (first use in this function) make: *** [tdbtest.o] Error 1 make: Leaving directory `/usr/src/packages/BUILD/samba-3.0.24/source/tdb' error: Bad exit status from /var/tmp/rpm-tmp.82314 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.82314 (%build) 283.553u 497.707s 15:10.05 85.8% 0+0k 0+0io 102pf+0w What to do now? 1) Use the unstripped smbd in /usr/src/packages/BUILD/samba-3.0.24/source/bin with valgrind ? 2) Try "rpmbuild -bi SPECS/samba.spec" ? 3) Use another version that compiles? Which one?
Created attachment 2328 [details] smbd log (excerpt) Now I am using /usr/src/packages/BUILD/samba-3.0.24/source/bin/smbd (not stripped) from samba-3.0.24-8.src.rpm. Again there is no useful backtrace information.
Created attachment 2329 [details] smbd start script Now I use valgrind.
Created attachment 2330 [details] smbd log file The logging for smbd (pid 3720), which did a segfault.
Created attachment 2331 [details] valgrind output Process 2742 is the first valgrind process. Process 3720 is the smbd that crashed. valgrind put the output for all processes into the same log file, don't know why. A valgrind bug? We see many errors in the output. I hope this helps.
Okay, that's a *LOT* better. Now I need to find the original sources of your package, in plain 3.0.24 in line lookup_sids.c line 809 there is no call to talloc_steal, but this shall not be your problem :-) Thanks! Volker
Volker identified an invalid hunk in patches/suse/winbind-fixes.diff. As soon as he's back home he'll update bugzilla.
Created attachment 2338 [details] Patch This is a patch over the SuSE src rpm. Lars will create a new package out of this. Thanks for the log! Volker
Closing as fixed. Lars, can you add a comment here when new packages arrive? Thanks, Volker
The the packages with 54 at the end of the RPM release have the fix. Rolf: Please mark this bug as verified if you no longer run into this problem.
No more crashes with samba-3.0.24-8.1.54. Dankeschön!