Bug 4434 - INTERNAL ERROR in smbd (signal 6)
INTERNAL ERROR in smbd (signal 6)
Product: Samba 3.0
Classification: Unclassified
Component: Packaging
x86 Linux
: P3 normal
: none
Assigned To: Gerald (Jerry) Carter
Samba QA Contact
Depends on:
  Show dependency treegraph
Reported: 2007-03-07 07:37 UTC by Rolf Barinka
Modified: 2007-10-03 01:41 UTC (History)
2 users (show)

See Also:

smbd log file (230.50 KB, text/plain)
2007-03-07 07:40 UTC, Rolf Barinka
no flags Details
smbd log with level 10 (872.05 KB, text/plain)
2007-03-08 01:39 UTC, Rolf Barinka
no flags Details
smbd log (197.38 KB, text/plain)
2007-03-12 03:51 UTC, Rolf Barinka
no flags Details
new backtrace script (322 bytes, text/plain)
2007-03-12 09:12 UTC, Rolf Barinka
no flags Details
smbd log (excerpt) (6.03 KB, text/plain)
2007-03-12 09:16 UTC, Rolf Barinka
no flags Details
smbd log (excerpt) (5.83 KB, text/plain)
2007-03-13 04:48 UTC, Rolf Barinka
no flags Details
smbd log (excerpt) (6.79 KB, text/plain)
2007-03-15 02:53 UTC, Rolf Barinka
no flags Details
smbd start script (304 bytes, text/plain)
2007-03-15 04:17 UTC, Rolf Barinka
no flags Details
smbd log file (412.78 KB, text/plain)
2007-03-15 04:19 UTC, Rolf Barinka
no flags Details
valgrind output (293.86 KB, text/plain)
2007-03-15 04:28 UTC, Rolf Barinka
no flags Details
Patch (675 bytes, patch)
2007-03-16 08:12 UTC, Volker Lendecke
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Rolf Barinka 2007-03-07 07:37:31 UTC
The reason for the crash is not known.
The crash is not reproduceable, but happens quite often.
There is no damage in user data, users do not even notice the crash.

Excerpt from logging, with log level 3:

[2007/03/07 13:48:10, 3] rpc_server/srv_pipe.c:api_rpcTNP(2265)
  api_rpcTNP: rpc command: LSA_LOOKUPSIDS2
[2007/03/07 13:48:10, 0] lib/fault.c:fault_report(41)
[2007/03/07 13:48:10, 0] lib/fault.c:fault_report(42)
  INTERNAL ERROR: Signal 6 in pid 31097 (3.0.24-0.1.49-1172-SUSE-SL10.2)
  Please read the Trouble-Shooting section of the Samba3-HOWTO
[2007/03/07 13:48:10, 0] lib/fault.c:fault_report(44)
  From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
[2007/03/07 13:48:10, 0] lib/fault.c:fault_report(45)
[2007/03/07 13:48:10, 0] lib/util.c:smb_panic(1599)
  PANIC (pid 31097): internal error
[2007/03/07 13:48:10, 0] lib/util.c:log_stack_trace(1706)
  BACKTRACE: 24 stack frames:
   #0 /usr/sbin/smbd(log_stack_trace+0x2d) [0x802165fd]
   #1 /usr/sbin/smbd(smb_panic+0x5d) [0x8021672d]
   #2 /usr/sbin/smbd [0x802024ea]
   #3 [0xb7fd7420]
   #4 /lib/libc.so.6(abort+0x101) [0xb7b2e801]
   #5 /usr/sbin/smbd [0x8021c9d6]
   #6 /usr/sbin/smbd(talloc_steal+0x3a) [0x8021cbaa]
   #7 /usr/sbin/smbd(lookup_sids+0x3aa) [0x801d3baa]
   #8 /usr/sbin/smbd [0x80106da1]
   #9 /usr/sbin/smbd(_lsa_lookup_sids2+0x12d) [0x801073fd]
   #10 /usr/sbin/smbd [0x80103601]
   #11 /usr/sbin/smbd(api_rpcTNP+0x198) [0x80160ac8]
   #12 /usr/sbin/smbd(api_pipe_request+0x19e) [0x8016111e]
   #13 /usr/sbin/smbd [0x8015cbaf]
   #14 /usr/sbin/smbd [0x8015d0ec]
   #15 /usr/sbin/smbd(write_to_pipe+0x6e) [0x8015b91e]
   #16 /usr/sbin/smbd [0x8005ee81]
   #17 /usr/sbin/smbd [0x8005f3b6]
   #18 /usr/sbin/smbd(reply_trans+0x5e0) [0x8005fcc0]
   #19 /usr/sbin/smbd [0x800b3d20]
   #20 /usr/sbin/smbd(smbd_process+0x78b) [0x800b4e1b]
   #21 /usr/sbin/smbd(main+0xbd0) [0x802bebe0]
   #22 /lib/libc.so.6(__libc_start_main+0xdc) [0xb7b19f9c]
   #23 /usr/sbin/smbd [0x80042d91]
[2007/03/07 13:48:10, 0] lib/fault.c:dump_core(168)
  unable to change to /var/log/samba/cores/smbdrefusing to dump core


        unix charset = ISO-8859-1
        workgroup = HELLER
        realm = HELLER.BIZ
        security = ADS
        client schannel = No
        obey pam restrictions = Yes
        password server = hntads01, hntads02, *
        log level = 1
        log file = /var/log/samba/%m
        max log size = 1000
        disable netbios = Yes
        deadtime = 15
        socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
        load printers = No
        printcap name = /dev/null
        lm announce = No
        preferred master = No
        local master = No
        domain master = No
        wins server =,
        oplock break wait time = 100
        idmap uid = 10000-20000
        idmap gid = 10000-20000
        template homedir = /l1/user/%U
        template shell = /bin/bash
        winbind use default domain = Yes
        winbind refresh tickets = Yes
        winbind offline logon = Yes
        invalid users = root
        create mask = 0666
        directory mask = 0777
        printing = bsd
        print command = lpr -r -P'%p' %s
        lpq command = lpq -P'%p'
        lprm command = lprm -P'%p' %j
        include = /etc/samba/.conf
        wide links = No
        dos filetime resolution = Yes
        fake directory create times = Yes
        comment = Home Directory of %u
        valid users = heller\%S
        force group = www
        read only = No
        create mask = 0660
        directory mask = 0770
        browseable = No
Comment 1 Rolf Barinka 2007-03-07 07:40:32 UTC
Created attachment 2316 [details]
smbd log file
Comment 2 Volker Lendecke 2007-03-07 13:38:56 UTC
Could you create per-client logfiles with

log file = /var/log/samba/log.%m

and for a short test period

max log size = 0

please? The logfile seems incomplete.


Comment 3 Rolf Barinka 2007-03-08 00:36:10 UTC
I have configured per-client log files.
There are about 120 clients, and only 4 clients are producing smbd crashes.
For these 4 clients I set the log level to 3.
The 4 clients produce about 300 crashes per week.

The smbd log file that I uploaded as attachment is an excerpt of a large log file.
But this excerpt includes all lines for one smbd process from start to termination. So this logging is complete.
Comment 4 Volker Lendecke 2007-03-08 00:51:24 UTC
Ah, ok. debug level 3 is not enough I'm afraid. We need debug level 10.

Comment 5 Rolf Barinka 2007-03-08 01:39:27 UTC
Created attachment 2317 [details]
smbd log with level 10
Comment 6 Volker Lendecke 2007-03-08 01:57:18 UTC
Ok, better. Thanks. I'll look at it later today.

Comment 7 Volker Lendecke 2007-03-08 12:20:24 UTC
I'm very sorry, but I don't see any code path in the plain 3.0.24 code that could lead to this kind of crash. Can you recompile your Samba with debugging symbols and possibly run it under valgrind? And, some lines in your log file indicate that you might not run the plain samba.org code. Can you retry with a self-compiled Samba?


Comment 8 Rolf Barinka 2007-03-09 07:30:02 UTC
I did not compile my Samba.
I installed the package samba-3.0.24-0.1.49.i586.rpm
that I got from http://ftp.suse.com/pub/projects/samba/3.0/10.0/i386/
Whether this is "the plain samba.org code", I do not know.

When I find the time next week I can install plain samba and see what happens.
Comment 9 Guenther Deschner 2007-03-09 08:01:02 UTC
You can install the samba-debuginfo package from the same ftp server, that way you would have debug symbols. Further debugging/backtrace info can be found at: http://en.opensuse.org/Samba#Backtraces
Comment 10 Rolf Barinka 2007-03-12 03:51:59 UTC
Created attachment 2323 [details]
smbd log

New versions installed:

smb.conf: panic action defined
 panic action = /usr/share/doc/packages/samba/examples/scripts/debugging/linux/backtrace %d

New smbd log file produced.
But gdb did not work as it should.
Comment 11 Rolf Barinka 2007-03-12 09:12:14 UTC
Created attachment 2324 [details]
new backtrace script

The script /usr/share/doc/packages/samba/examples/scripts/debugging/linux/backtrace does not work. So I made this simple script, which uses a setuid root copy of gdb, which is named sugdb.
Comment 12 Rolf Barinka 2007-03-12 09:16:44 UTC
Created attachment 2325 [details]
smbd log (excerpt)

Debugging smbd is still not possible.
ptrace now works.
But now: "No symbol table info available."
Comment 13 Rolf Barinka 2007-03-13 04:48:21 UTC
Created attachment 2326 [details]
smbd log (excerpt)

backtrace is useless.
So I tried to get a core file.
I put "ulimit -Sc 20000" into /etc/init.d/smb.
chmod 755 /var/log/samba;
chmod 777 /var/log/samba/cores;
chmod 777 /var/log/samba/cores/smbd;

Now smbd logs "dumping core in /var/log/samba/cores/smbd"
But there is no core file!
Comment 14 Guenther Deschner 2007-03-13 04:53:17 UTC
Lars, can you take a look at the backtrace stuff?
Comment 15 Rolf Barinka 2007-03-14 05:20:44 UTC
I made a new backtrace script:

#! /bin/sh
exec 1>&2
cd /var/log/samba
ulimit -Sc 50000
cat << EOF  > /tmp/backtrace"${PID}"
/usr/bin/sugdb -x /tmp/backtrace"${PID}" /usr/sbin/smbd "${PID}"
/bin/rm -f /tmp/backtrace"${PID}"

This script was able to produce a core file.
I used gdb to look into it.

41# gdb /usr/sbin/smbd core.746
GNU gdb 6.5
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i586-suse-linux"...Using host libthread_db library "/lib/libthread_db.so.1".

warning: Can't read pathname for load map: Input/output error.
Loaded symbols for /usr/sbin/smbd
Reading symbols from /usr/lib/libldap-2.2.so.7...done.
Loaded symbols for /usr/lib/libldap-2.2.so.7
...... Reading symbols from <many libraries> ......
Reading symbols from /lib/libnss_winbind.so.2...Reading symbols from /usr/lib/debug/lib/libnss_winbind.so.2.debug...done.
Loaded symbols for /lib/libnss_winbind.so.2
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Failed to read a valid object file image from memory.
Core was generated by `/usr/sbin/smbd'.
#0  0xb7f53410 in ?? ()
(gdb) set height 0
(gdb) up 8
#2  0x00000000 in ?? ()
(gdb) bt full
#0  0xb7f53410 in ?? ()
No symbol table info available.
#1  0xbfab8318 in ?? ()
No symbol table info available.
#2  0x00000000 in ?? ()
No symbol table info available.
(gdb) where
#0  0xb7f53410 in ?? ()
#1  0xbfab8318 in ?? ()
#2  0x00000000 in ?? ()
(gdb) quit

So there are problems with symbol tables.
And no useful backtrace data.

Shall I upload the core file (13MB)?
Comment 16 Volker Lendecke 2007-03-14 05:53:46 UTC
My valgrind arguments:

valgrind --tool=memcheck --leak-check=yes -v --num-callers=20 --trace-children=yes

Comment 17 Rolf Barinka 2007-03-14 10:15:54 UTC
I try to get a smbd with included debug information.
I did not find a samba-3.0.24-5.1.52.src.rpm.

But I found a samba-3.0.24-8.src.rpm at
I installed this.

"rpmbuild -bp SPECS/samba.spec" was okay.

"rpmbuild -bc SPECS/samba.spec" produced errors:

+ /usr/bin/make -C tdb tdbtest tdbtorture
make: Entering directory `/usr/src/packages/BUILD/samba-3.0.24/source/tdb'
gcc -DSTANDALONE -DTDB_DEBUG -g -DHAVE_MMAP=1   -c -o tdbtest.o tdbtest.c
tdbtest.c:13:18: error: gdbm.h: No such file or directory
tdbtest.c:23: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'gdbm'
tdbtest.c: In function 'compare_db':
tdbtest.c:58: error: 'datum' undeclared (first use in this function)
tdbtest.c:58: error: (Each undeclared identifier is reported only once
tdbtest.c:58: error: for each function it appears in.)
tdbtest.c:58: error: expected ';' before 'gd'
tdbtest.c:63: error: 'gkey' undeclared (first use in this function)
tdbtest.c:66: error: 'gd' undeclared (first use in this function)
tdbtest.c:66: error: 'gdbm' undeclared (first use in this function)
tdbtest.c:95: error: 'gnextkey' undeclared (first use in this function)
tdbtest.c: In function 'addrec_gdbm':
tdbtest.c:153: error: 'datum' undeclared (first use in this function)
tdbtest.c:153: error: expected ';' before 'key'
tdbtest.c:161: error: 'key' undeclared (first use in this function)
tdbtest.c:164: error: 'data' undeclared (first use in this function)
tdbtest.c:168: error: 'gdbm' undeclared (first use in this function)
tdbtest.c:170: error: 'GDBM_REPLACE' undeclared (first use in this function)
tdbtest.c: In function 'main':
tdbtest.c:231: error: 'gdbm' undeclared (first use in this function)
tdbtest.c:231: error: 'GDBM_WRITER' undeclared (first use in this function)
tdbtest.c:231: error: 'GDBM_NEWDB' undeclared (first use in this function)
tdbtest.c:231: error: 'GDBM_FAST' undeclared (first use in this function)
make: *** [tdbtest.o] Error 1
make: Leaving directory `/usr/src/packages/BUILD/samba-3.0.24/source/tdb'
error: Bad exit status from /var/tmp/rpm-tmp.82314 (%build)
RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.82314 (%build)
283.553u 497.707s 15:10.05 85.8%        0+0k 0+0io 102pf+0w

What to do now?
1) Use the unstripped smbd in /usr/src/packages/BUILD/samba-3.0.24/source/bin
   with valgrind ?
2) Try "rpmbuild -bi SPECS/samba.spec" ?
3) Use another version that compiles? Which one?

Comment 18 Rolf Barinka 2007-03-15 02:53:37 UTC
Created attachment 2328 [details]
smbd log (excerpt)

Now I am using /usr/src/packages/BUILD/samba-3.0.24/source/bin/smbd
(not stripped) from samba-3.0.24-8.src.rpm.

Again there is no useful backtrace information.
Comment 19 Rolf Barinka 2007-03-15 04:17:48 UTC
Created attachment 2329 [details]
smbd start script

Now I use valgrind.
Comment 20 Rolf Barinka 2007-03-15 04:19:57 UTC
Created attachment 2330 [details]
smbd log file

The logging for smbd (pid 3720), which did a segfault.
Comment 21 Rolf Barinka 2007-03-15 04:28:23 UTC
Created attachment 2331 [details]
valgrind output

Process 2742 is the first valgrind process.
Process 3720 is the smbd that crashed.

valgrind put the output for all processes into the same log file,
don't know why. A valgrind bug?

We see many errors in the output.

I hope this helps.
Comment 22 Volker Lendecke 2007-03-15 12:03:46 UTC
Okay, that's a *LOT* better. Now I need to find the original sources of your package, in plain 3.0.24 in line lookup_sids.c line 809 there is no call to talloc_steal, but this shall not be your problem :-)


Comment 23 Lars Müller 2007-03-15 13:59:05 UTC
Volker identified an invalid hunk in patches/suse/winbind-fixes.diff.  As soon as he's back home he'll update bugzilla.
Comment 24 Volker Lendecke 2007-03-16 08:12:30 UTC
Created attachment 2338 [details]

This is a patch over the SuSE src rpm. Lars will create a new package out of this.  

Thanks for the log!

Comment 25 Volker Lendecke 2007-03-16 08:13:25 UTC
Closing as fixed. Lars, can you add a comment here when new packages arrive?


Comment 26 Lars Müller 2007-03-18 14:27:17 UTC
The the packages with 54 at the end of the RPM release have the fix.

Rolf: Please mark this bug as verified if you no longer run into this problem.
Comment 27 Rolf Barinka 2007-03-19 07:42:22 UTC
No more crashes with samba-3.0.24-8.1.54.