Bug 10052 - Internal Error: Signal 11 PANIC on one of two DCs
Internal Error: Signal 11 PANIC on one of two DCs
Status: RESOLVED FIXED
Product: Samba 4.0
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB
4.0.6
x86 Linux
: P5 major
: ---
Assigned To: Karolin Seeger
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-28 04:58 UTC by Andrew Martin
Modified: 2013-10-21 08:37 UTC (History)
3 users (show)

See Also:


Attachments
full backtrace from this PANIC (13.47 KB, text/plain)
2013-07-28 04:58 UTC, Andrew Martin
no flags Details
compile options used when building samba (1.54 KB, text/plain)
2013-07-28 04:59 UTC, Andrew Martin
no flags Details
smb.conf (620 bytes, text/plain)
2013-07-28 05:00 UTC, Andrew Martin
no flags Details
patch to use dsdb_search_one to avoid this crash (2.39 KB, patch)
2013-07-29 22:51 UTC, Andrew Bartlett
abartlet: review? (amartin)
metze: review+
Details
Patch for v4-1-test (2.74 KB, patch)
2013-10-15 11:16 UTC, Stefan Metzmacher
abartlet: review+
Details
Patch for v4-0-test (2.74 KB, patch)
2013-10-15 11:17 UTC, Stefan Metzmacher
abartlet: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Martin 2013-07-28 04:58:53 UTC
Created attachment 9083 [details]
full backtrace from this PANIC

I am running Samba 4.0.6 as an AD DC on two Ubuntu 12.04 servers, dc0 (which currently holds the FSMO roles) and dc1. Replication occurs between the two DCs. Recently, samba on dc0 has started PANICing fairly frequently with this error in the syslog:
Jul 27 23:25:23 dc0 smbd[27824]: [2013/07/27 23:25:23.081517,  0] ../lib/util/fault.c:72(fault_report)
Jul 27 23:25:23 dc0 smbd[27824]:   ===============================================================
Jul 27 23:25:23 dc0 smbd[27824]: [2013/07/27 23:25:23.082650,  0] ../lib/util/fault.c:73(fault_report)
Jul 27 23:25:23 dc0 smbd[27824]:   INTERNAL ERROR: Signal 11 in pid 27824 (4.0.6-Debian)
Jul 27 23:25:23 dc0 smbd[27824]:   Please read the Trouble-Shooting section of the Samba HOWTO
Jul 27 23:25:23 dc0 smbd[27824]: [2013/07/27 23:25:23.082767,  0] ../lib/util/fault.c:75(fault_report)
Jul 27 23:25:23 dc0 smbd[27824]:   ===============================================================
Jul 27 23:25:23 dc0 smbd[27824]: [2013/07/27 23:25:23.082828,  0] ../source3/lib/util.c:810(smb_panic_s3)
Jul 27 23:25:23 dc0 smbd[27824]:   PANIC (pid 27824): internal error
Jul 27 23:25:23 dc0 smbd[27824]: [2013/07/27 23:25:23.083881,  0] ../source3/lib/util.c:921(log_stack_trace)
Jul 27 23:25:23 dc0 smbd[27824]:   BACKTRACE: 30 stack frames:
Jul 27 23:25:23 dc0 smbd[27824]:    #0 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f8b63ea307a]
Jul 27 23:25:23 dc0 smbd[27824]:    #1 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(smb_panic_s3+0x22) [0x7f8b63ea3152]
Jul 27 23:25:23 dc0 smbd[27824]:    #2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x2f) [0x7f8b653a94bf]
Jul 27 23:25:23 dc0 smbd[27824]:    #3 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(+0x1a6d8) [0x7f8b653a96d8]
Jul 27 23:25:23 dc0 smbd[27824]:    #4 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f8b655cecb0]
Jul 27 23:25:23 dc0 smbd[27824]:    #5 /usr/lib/x86_64-linux-gnu/samba/libdfs_server_ad.so(+0x1412) [0x7f8b61888412]
Jul 27 23:25:23 dc0 smbd[27824]:    #6 /usr/lib/x86_64-linux-gnu/samba/libdfs_server_ad.so(+0x1985) [0x7f8b61888985]
Jul 27 23:25:23 dc0 smbd[27824]:    #7 /usr/lib/x86_64-linux-gnu/samba/libdfs_server_ad.so(dfs_server_ad_get_referrals+0xa82) [0x7f8b61889aa2]
Jul 27 23:25:23 dc0 smbd[27824]:    #8 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so(+0x16b98c) [0x7f8b6501198c]
Jul 27 23:25:23 dc0 smbd[27824]:    #9 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so(smb_vfs_call_get_dfs_referrals+0x2a) [0x7f8b64fa17ea]
Jul 27 23:25:23 dc0 smbd[27824]:    #10 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so(setup_dfs_referral+0xa6) [0x7f8b64fc2926]
Jul 27 23:25:23 dc0 smbd[27824]:    #11 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so(smbd_smb2_request_process_ioctl+0xab8) [0x7f8b64fd9018]
Jul 27 23:25:23 dc0 smbd[27824]:    #12 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so(smbd_smb2_request_dispatch+0xc69) [0x7f8b64fcb439]
Jul 27 23:25:23 dc0 smbd[27824]:    #13 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so(+0x1259b1) [0x7f8b64fcb9b1]
Jul 27 23:25:23 dc0 smbd[27824]:    #14 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so(+0x1224d6) [0x7f8b64fc84d6]
Jul 27 23:25:23 dc0 smbd[27824]:    #15 /usr/lib/x86_64-linux-gnu/samba/libsamba-sockets.so(+0xa1e2) [0x7f8b63c751e2]
Jul 27 23:25:23 dc0 smbd[27824]:    #16 /usr/lib/x86_64-linux-gnu/samba/libsamba-sockets.so(+0x9cb4) [0x7f8b63c74cb4]
Jul 27 23:25:23 dc0 smbd[27824]:    #17 /usr/lib/x86_64-linux-gnu/samba/libsamba-sockets.so(+0x8bb4) [0x7f8b63c73bb4]
Jul 27 23:25:23 dc0 smbd[27824]:    #18 /usr/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_immediate+0xe2) [0x7f8b62899dc2]
Jul 27 23:25:23 dc0 smbd[27824]:    #19 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(run_events_poll+0x48) [0x7f8b63eba148]
Jul 27 23:25:23 dc0 smbd[27824]:    #20 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(+0x3843b) [0x7f8b63eba43b]
Jul 27 23:25:23 dc0 smbd[27824]:    #21 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f8b6289950d]
Jul 27 23:25:23 dc0 smbd[27824]:    #22 /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so(smbd_process+0xa7e) [0x7f8b64fb657e]
Jul 27 23:25:23 dc0 smbd[27824]:    #23 /usr/sbin/smbd() [0x4096a4]
Jul 27 23:25:23 dc0 smbd[27824]:    #24 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(run_events_poll+0x294) [0x7f8b63eba394]
Jul 27 23:25:23 dc0 smbd[27824]:    #25 /usr/lib/x86_64-linux-gnu/libsmbconf.so.0(+0x384e8) [0x7f8b63eba4e8]
Jul 27 23:25:23 dc0 smbd[27824]:    #26 /usr/lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x9d) [0x7f8b6289950d]
Jul 27 23:25:23 dc0 smbd[27824]:    #27 /usr/sbin/smbd(main+0x1200) [0x406670]
Jul 27 23:25:23 dc0 smbd[27824]:    #28 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f8b624f876d]
Jul 27 23:25:23 dc0 smbd[27824]:    #29 /usr/sbin/smbd() [0x406b69]
Jul 27 23:25:23 dc0 smbd[27824]: [2013/07/27 23:25:23.085784,  0] ../source3/lib/dumpcore.c:317(dump_core)
Jul 27 23:25:23 dc0 smbd[27824]:   dumping core in /var/log/samba/cores/smbd

I have attached a full backtrace (with debug symbols) from the core dump. I think that Samba is actually restarting whatever is failing when this happens, as it is still running after this error occurs, and then this error appears again in the syslog within a few minutes or up to a couple hours.

At first glance, this appears to be very similar to #10004, however I believe it to be a different problem since the backtrace differs widely.

I have also attached the compile options I used, as well as my smb.conf file. Any ideas on what is causing this, or how to resolve/debug it?

Thanks!
Comment 1 Andrew Martin 2013-07-28 04:59:51 UTC
Created attachment 9084 [details]
compile options used when building samba
Comment 2 Andrew Martin 2013-07-28 05:00:07 UTC
Created attachment 9085 [details]
smb.conf
Comment 3 Andrew Bartlett 2013-07-29 22:51:02 UTC
Created attachment 9086 [details]
patch to use dsdb_search_one to avoid this crash

Can you try this patch?
Comment 4 Andrew Martin 2013-07-31 16:51:23 UTC
A  bit more debug: it appears that after this PANIC happens, group policy is not applied from this DC until samba is manually restarted. 

(In reply to comment #3)
> Created attachment 9086 [details]
> patch to use dsdb_search_one to avoid this crash
> 
> Can you try this patch?

The patch you provided appears to resolve this bug! :)

Thanks a lot for the help!
Comment 5 waynea 2013-10-14 11:59:01 UTC
Comment on attachment 9086 [details]
patch to use dsdb_search_one to avoid this crash

I was having the same problem and this patch has fixed it for me. I was epxeriencing this crash starting with 4.0.9, also in 4.0.10 and 4.1.0.
Comment 6 Andrew Bartlett 2013-10-14 19:51:43 UTC
Comment on attachment 9086 [details]
patch to use dsdb_search_one to avoid this crash

Metze,

Can you add my sign-off and review this into master?
Comment 7 Stefan Metzmacher 2013-10-15 06:05:54 UTC
Comment on attachment 9086 [details]
patch to use dsdb_search_one to avoid this crash

This looks good. Andrew can you push it to master with my review?
Comment 8 Stefan Metzmacher 2013-10-15 11:16:29 UTC
Comment on attachment 9086 [details]
patch to use dsdb_search_one to avoid this crash

This is already in master as commit 859182da6d06be0e9d37d7ed3448efc3dc78bdb2
Comment 9 Stefan Metzmacher 2013-10-15 11:16:57 UTC
Created attachment 9290 [details]
Patch for v4-1-test
Comment 10 Stefan Metzmacher 2013-10-15 11:17:25 UTC
Created attachment 9291 [details]
Patch for v4-0-test
Comment 11 Karolin Seeger 2013-10-17 08:30:54 UTC
Pushed to autobuild-v4-1-test and autobuild-v4-0-test.
Comment 12 Karolin Seeger 2013-10-21 08:37:48 UTC
Pushed to v4-1-test and v4-0-test.
Closing out bug report.

Thanks!