Bug 11182 - Panic triggered by smbd_smb2_request_notify_done() -> smbXsrv_session_find_channel() in smbd
Panic triggered by smbd_smb2_request_notify_done() -> smbXsrv_session_find_ch...
Status: RESOLVED FIXED
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services
4.2.1
All All
: P5 normal
: ---
Assigned To: Karolin Seeger
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-03-27 17:01 UTC by Nick Semenkovich
Modified: 2015-05-21 07:43 UTC (History)
3 users (show)

See Also:


Attachments
packets right around crash (5.63 KB, application/octet-stream)
2015-04-28 19:53 UTC, Nick Semenkovich
no flags Details
Work in progress patches (on v4-2-test) (51.79 KB, patch)
2015-05-03 08:10 UTC, Stefan Metzmacher
no flags Details
Patches for v4-2-test (56.19 KB, patch)
2015-05-07 09:05 UTC, Stefan Metzmacher
obnox: review+
metze: review? (jra)
Details
Patches for v4-1-test (51.40 KB, patch)
2015-05-08 17:25 UTC, Stefan Metzmacher
obnox: review+
jra: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Semenkovich 2015-03-27 17:01:10 UTC
Running from -O3 from the 4.2.0 git tag.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f5c85591a1c in __libc_waitpid (pid=31757, stat_loc=stat_loc@entry=0x7fffafe1ff90, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
#0  0x00007f5c85591a1c in __libc_waitpid (pid=31757, stat_loc=stat_loc@entry=0x7fffafe1ff90, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
        resultvar = 18446744073709551104
        oldtype = -2054617344
#1  0x00007f5c855158e2 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
        __result = <optimized out>
        _buffer = {__routine = 0x7f5c85515b80 <cancel_handler>, __arg = 0x7fffafe1ff60, __canceltype = 0, __prev = 0x0}
        _avail = 1
        status = -1344143424
        save = <optimized out>
        pid = 31757
        sa = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {65536, 0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x7f5c8b26d3c0}
        omask = {__val = {7296, 140035428010736, 140035350168384, 140035418767360, 140736144217072, 0, 0, 140035403880138, 999428753, 140035374757896, 4294967295, 296385334000, 0, 206158430224, 140736144212208, 140035416670336}}
#2  0x00007f5c86c08662 in smb_panic_s3 () from /usr/local/samba/lib/libsmbconf.so.0
No symbol table info available.
#3  0x00007f5c8901bf15 in smb_panic () from /usr/local/samba/lib/libsamba-util.so.0
No symbol table info available.
#4  0x00007f5c8901bbed in fault_report () from /usr/local/samba/lib/libsamba-util.so.0
No symbol table info available.
#5  0x00007f5c8901bc02 in sig_fault () from /usr/local/samba/lib/libsamba-util.so.0
No symbol table info available.
#6  <signal handler called>
No locals.
#7  0x00007f5c88bd3817 in smbXsrv_session_find_channel () from /usr/local/samba/lib/private/libsmbd-base-samba4.so
No symbol table info available.
#8  0x00007f5c88ba8774 in smbd_smb2_signing_key () from /usr/local/samba/lib/private/libsmbd-base-samba4.so
No symbol table info available.
#9  0x00007f5c88bac40f in smbd_smb2_request_reply () from /usr/local/samba/lib/private/libsmbd-base-samba4.so
No symbol table info available.
#10 0x00007f5c88bad01e in smbd_smb2_request_done_ex () from /usr/local/samba/lib/private/libsmbd-base-samba4.so
No symbol table info available.
#11 0x00007f5c88bad2c8 in smbd_smb2_request_error_ex () from /usr/local/samba/lib/private/libsmbd-base-samba4.so
No symbol table info available.
#12 0x00007f5c88bc942d in smbd_smb2_request_notify_done () from /usr/local/samba/lib/private/libsmbd-base-samba4.so
No symbol table info available.
#13 0x00007f5c881f6016 in _tevent_req_notify_callback () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#14 0x00007f5c881f60e9 in tevent_req_finish () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#15 0x00007f5c881f620e in tevent_req_trigger () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#16 0x00007f5c881f53f2 in tevent_common_loop_immediate () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#17 0x00007f5c86c2881d in run_events_poll () from /usr/local/samba/lib/libsmbconf.so.0
No symbol table info available.
#18 0x00007f5c86c28e91 in s3_event_loop_once () from /usr/local/samba/lib/libsmbconf.so.0
No symbol table info available.
#19 0x00007f5c881f4539 in _tevent_loop_once () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#20 0x00007f5c881f4783 in tevent_common_loop_wait () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#21 0x00007f5c881f484e in _tevent_loop_wait () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#22 0x00007f5c88b90a0e in smbd_process () from /usr/local/samba/lib/private/libsmbd-base-samba4.so
No symbol table info available.
#23 0x00007f5c8967916b in smbd_accept_connection ()
No symbol table info available.
#24 0x00007f5c86c28d16 in run_events_poll () from /usr/local/samba/lib/libsmbconf.so.0
No symbol table info available.
#25 0x00007f5c86c28fa5 in s3_event_loop_once () from /usr/local/samba/lib/libsmbconf.so.0
No symbol table info available.
#26 0x00007f5c881f4539 in _tevent_loop_once () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#27 0x00007f5c881f4783 in tevent_common_loop_wait () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#28 0x00007f5c881f484e in _tevent_loop_wait () from /usr/local/samba/lib/private/libtevent.so.0
No symbol table info available.
#29 0x00007f5c89679f81 in smbd_parent_loop ()
No symbol table info available.
#30 0x00007f5c8967b8df in main ()
No symbol table info available.
A debugging session is active.

        Inferior 1 [process 11221] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]


  ===============================================================
[2015/03/27 11:34:29.288536,  0] ../lib/util/fault.c:79(fault_report)
  INTERNAL ERROR: Signal 11 in pid 11221 (4.2.0)
  Please read the Trouble-Shooting section of the Samba HOWTO
[2015/03/27 11:34:29.288559,  0] ../lib/util/fault.c:81(fault_report)
  ===============================================================
[2015/03/27 11:34:29.288575,  0] ../source3/lib/util.c:788(smb_panic_s3)
  PANIC (pid 11221): internal error
[2015/03/27 11:34:29.289324,  0] ../source3/lib/util.c:899(log_stack_trace)
  BACKTRACE: 32 stack frames:
   #0 /usr/local/samba/lib/libsmbconf.so.0(log_stack_trace+0x1f) [0x7f5c86c0875c]
   #1 /usr/local/samba/lib/libsmbconf.so.0(smb_panic_s3+0x6f) [0x7f5c86c085a7]
   #2 /usr/local/samba/lib/libsamba-util.so.0(smb_panic+0x28) [0x7f5c8901bf15]
   #3 /usr/local/samba/lib/libsamba-util.so.0(+0x2abed) [0x7f5c8901bbed]
   #4 /usr/local/samba/lib/libsamba-util.so.0(+0x2ac02) [0x7f5c8901bc02]
   #5 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc90) [0x7f5c8923ac90]
   #6 /usr/local/samba/lib/private/libsmbd-base-samba4.so(smbXsrv_session_find_channel+0x6c) [0x7f5c88bd3817]
   #7 /usr/local/samba/lib/private/libsmbd-base-samba4.so(+0x194774) [0x7f5c88ba8774]
   #8 /usr/local/samba/lib/private/libsmbd-base-samba4.so(+0x19840f) [0x7f5c88bac40f]
   #9 /usr/local/samba/lib/private/libsmbd-base-samba4.so(smbd_smb2_request_done_ex+0x589) [0x7f5c88bad01e]
   #10 /usr/local/samba/lib/private/libsmbd-base-samba4.so(smbd_smb2_request_error_ex+0x29f) [0x7f5c88bad2c8]
   #11 /usr/local/samba/lib/private/libsmbd-base-samba4.so(+0x1b542d) [0x7f5c88bc942d]
   #12 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a) [0x7f5c881f6016]
   #13 /usr/local/samba/lib/private/libtevent.so.0(+0x60e9) [0x7f5c881f60e9]
   #14 /usr/local/samba/lib/private/libtevent.so.0(+0x620e) [0x7f5c881f620e]
   #15 /usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_immediate+0x1f5) [0x7f5c881f53f2]
   #16 /usr/local/samba/lib/libsmbconf.so.0(run_events_poll+0x56) [0x7f5c86c2881d]
   #17 /usr/local/samba/lib/libsmbconf.so.0(+0x44e91) [0x7f5c86c28e91]
   #18 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_once+0xf4) [0x7f5c881f4539]
   #19 /usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_wait+0x25) [0x7f5c881f4783]
   #20 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_wait+0x2b) [0x7f5c881f484e]
   #21 /usr/local/samba/lib/private/libsmbd-base-samba4.so(smbd_process+0xb28) [0x7f5c88b90a0e]
   #22 /usr/local/samba/sbin/smbd(+0xb16b) [0x7f5c8967916b]
   #23 /usr/local/samba/lib/libsmbconf.so.0(run_events_poll+0x54f) [0x7f5c86c28d16]
   #24 /usr/local/samba/lib/libsmbconf.so.0(+0x44fa5) [0x7f5c86c28fa5]
   #25 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_once+0xf4) [0x7f5c881f4539]
   #26 /usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_wait+0x25) [0x7f5c881f4783]
   #27 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_wait+0x2b) [0x7f5c881f484e]
   #28 /usr/local/samba/sbin/smbd(+0xbf81) [0x7f5c89679f81]
   #29 /usr/local/samba/sbin/smbd(main+0x17a7) [0x7f5c8967b8df]
   #30 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f5c854f2ec5]
   #31 /usr/local/samba/sbin/smbd(+0x5ed9) [0x7f5c89673ed9]
[2015/03/27 11:34:29.289587,  0] ../source3/lib/util.c:800(smb_panic_s3)
  smb_panic(): calling panic action [/home/semenko/panic-action 11221]
[2015/03/27 11:34:29.309582,  2] ../lib/util/modules.c:191(do_smb_load_module)
  Module 'acl_xattr' loaded
Comment 1 Volker Lendecke 2015-03-27 17:34:36 UTC
In case you have a corefile or can reproduce it -- a stacktrace with debug symbols would be extremely helpful
Comment 2 Nick Semenkovich 2015-03-27 18:54:38 UTC
(In reply to Volker Lendecke from comment #1)

Yep -- I've only seen it once -- will recompile w/ debug & repro if/when I see it again.
Comment 3 Nick Semenkovich 2015-04-20 18:12:18 UTC
Ok -- I've now seen this a number of times.

Running: 4.2.1 (latest git tag, compiled with --enable-debug & -Og -g)

This client is a fully-patched Windows 8.1 domain-joined machine.


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f5f8d73ea1c in __libc_waitpid (pid=14271, stat_loc=stat_loc@entry=0x7fffd3f2bb50, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
#0  0x00007f5f8d73ea1c in __libc_waitpid (pid=14271, stat_loc=stat_loc@entry=0x7fffd3f2bb50, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
        resultvar = 18446744073709551104
        oldtype = -1918642432
#1  0x00007f5f8d6c28e2 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
        __result = <optimized out>
        _buffer = {__routine = 0x7f5f8d6c2b80 <cancel_handler>, __arg = 0x7fffd3f2bb20, __canceltype = 0, __prev = 0x0}
        _avail = 1
        status = -739067008
        save = <optimized out>
        pid = 14271
        sa = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {65536, 0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x7f5f93f903a0}
        omask = {__val = {7296, 140048460905200, 140048371045184, 140048439648256, 140736749293520, 0, 0, 140048424756938, 999428753, 140048395634696, 4294967295, 296382331632, 0, 206158430224, 140736749288624, 140048437551232}}
#2  0x00007f5f8edb5662 in smb_panic_s3 (why=0x7f5f911d0aad "internal error") at ../source3/lib/util.c:801
        cmd = 0x7f5f93f903a0 "/home/semenko/panic-action 25283"
        result = 32607
        __FUNCTION__ = "smb_panic_s3"
#3  0x00007f5f911c9f15 in smb_panic (why=0x7f5f911d0aad "internal error") at ../lib/util/fault.c:166
No locals.
#4  0x00007f5f911c9bed in fault_report (sig=11) at ../lib/util/fault.c:83
        counter = 1
        __FUNCTION__ = "fault_report"
#5  0x00007f5f911c9c02 in sig_fault (sig=11) at ../lib/util/fault.c:94
No locals.
#6  <signal handler called>
No locals.
#7  0x00007f5f90d80b0d in smbXsrv_session_find_channel (session=0x7f5f937cbaa0, conn=0x7f5f93ec6270, _c=0x7fffd3f2c328) at ../source3/smbd/smbXsrv_session.c:1289
        i = 0
#8  0x00007f5f90d55a6a in smbd_smb2_signing_key (session=0x7f5f937cbaa0, xconn=0x7f5f93ec6270) at ../source3/smbd/smb2_server.c:1429
        c = 0x0
        status = {v = 5}
        key = {data = 0x0, length = 0}
#9  0x00007f5f90d59705 in smbd_smb2_request_reply (req=0x7f5f92f289d0) at ../source3/smbd/smb2_server.c:2499
        x = 0x7f5f937cbaa0
        signing_key = {data = 0x7f5f93b73690 "\260\006Ē_\177", length = 549}
        xconn = 0x7f5f93ec6270
        first_idx = 1
        firsttf = 0x7f5f92f28b30
        outhdr = 0x7f5f92f28b40
        outdyn = 0x7f5f92f28b60
        status = {v = 2475449088}
#10 0x00007f5f90d5a314 in smbd_smb2_request_done_ex (req=0x7f5f92f289d0, status=..., body=..., dyn=0x7fffd3f2c540, location=0x7f5f90ee0878 "../source3/smbd/smb2_server.c:2737") at ../source3/smbd/smb2_server.c:2673
        outhdr = 0x7f5f92f28b70 "\376SMB@"
        outbody_v = 0x7f5f92f28b50
        outdyn_v = 0x7f5f92f28b60
        next_command_ofs = 0
        __FUNCTION__ = "smbd_smb2_request_done_ex"
#11 0x00007f5f90d5a5be in smbd_smb2_request_error_ex (req=0x7f5f92f289d0, status=..., info=0x7fffd3f2c540, location=0x7f5f90ee9310 "../source3/smbd/smb2_notify.c:123") at ../source3/smbd/smb2_server.c:2737
        xconn = 0x7f5f93ec6270
        body = {data = 0x7f5f92f28bb0 "\t", length = 8}
        _dyn = {data = 0x7f5f92f28bb8 "", length = 1}
        outhdr = 0x7f5f92f28b70 "\376SMB@"
        unread_bytes = 0
        __FUNCTION__ = "smbd_smb2_request_error_ex"
#12 0x00007f5f90d76723 in smbd_smb2_request_notify_done (subreq=0x0) at ../source3/smbd/smb2_notify.c:123
        req = 0x7f5f92f289d0
        outbody = {data = 0x7f5f92f28d60 "tfא_\177", length = 140048463990016}
        outdyn = {data = 0x7fffd3f2c600 "H\227\356\220_\177", length = 140048422621689}
        out_output_buffer_offset = 0
        out_output_buffer = {data = 0x0, length = 0}
        status = {v = 268}
        error = {v = 2431481472}
#13 0x00007f5f903a3016 in _tevent_req_notify_callback (req=0x7f5f92f28d60, location=0x7f5f90ee9748 "../source3/smbd/smb2_notify.c:355") at ../lib/tevent/tevent_req.c:112
No locals.
#14 0x00007f5f903a30e9 in tevent_req_finish (req=0x7f5f92f28d60, state=TEVENT_REQ_USER_ERROR, location=0x7f5f90ee9748 "../source3/smbd/smb2_notify.c:355") at ../lib/tevent/tevent_req.c:149
No locals.
#15 0x00007f5f903a320e in tevent_req_trigger (ev=0x7f5f92c406b0, im=0x7f5f92f28e40, private_data=0x7f5f92f28d60) at ../lib/tevent/tevent_req.c:206
        req = 0x7f5f92f28d60
#16 0x00007f5f903a23f2 in tevent_common_loop_immediate (ev=0x7f5f92c406b0) at ../lib/tevent/tevent_immediate.c:135
        im = 0x7f5f92f28e40
        handler = 0x7f5f903a31bf <tevent_req_trigger>
        private_data = 0x7f5f92f28d60
#17 0x00007f5f8edd581d in run_events_poll (ev=0x7f5f92c406b0, pollrtn=0, pfds=0x0, num_pfds=0) at ../source3/lib/events.c:192
        state = 0x7f5f911b8725 <talloc_pop>
        pollfd_idx = 0x7fffd3f2c730
        fde = 0x7fffd3f2c730
        __FUNCTION__ = "run_events_poll"
#18 0x00007f5f8edd5e91 in s3_event_loop_once (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../source3/lib/events.c:303
        state = 0x7f5f92c416c0
        timeout = 2147483647
        num_pfds = 32767
        ret = -739063936
        poll_errno = 32767
#19 0x00007f5f903a1539 in _tevent_loop_once (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:533
        ret = 0
        nesting_stack_ptr = 0x0
#20 0x00007f5f903a1783 in tevent_common_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:637
        ret = 0
#21 0x00007f5f903a184e in _tevent_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:656
No locals.
#22 0x00007f5f90d3dd04 in smbd_process (ev_ctx=0x7f5f92c406b0, msg_ctx=0x7f5f92c407a0, sock_fd=46, interactive=false) at ../source3/smbd/process.c:3992
        trace_state = {frame = 0x7f5f93f901f0, smbd_idle_profstamp = 0}
        client = 0x7f5f9391de40
        sconn = 0x7f5f93000320
        xconn = 0x7f5f93ec6270
        locaddr = 0x7f5f9314ab20 "\220\267\001\224_\177"
        remaddr = 0x7f5f93afa550 "ipv4:192.168.0.109:49541"
        ret = 32607
        status = {v = 0}
        __FUNCTION__ = "smbd_process"
#23 0x00007f5f9182716b in smbd_accept_connection (ev=0x7f5f92c406b0, fde=0x7f5f9428da20, flags=1, private_data=0x7f5f94658be0) at ../source3/smbd/server.c:627
        status = {v = 0}
        s = 0x0
        msg_ctx = 0x7f5f92c407a0
        addr = {ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 16 times>, "\030\063\027\223_\177\000\000 \312\362\323\377\177\000\000\240\311\362\323\377\177\000\000\205f\034\221_\177\000\000\030\063\027\223_\177\000\000 \312\362\323\377\177\000\000\065\000\000\000\000\000\000\000\021|\r\000\000\000\000\000@\312\362\323\377\177\000\000\243Wݎ_\177\000\000\313\030\065U\000\000\000\000h\312\362\323\377\177\000"}
        in_addrlen = 16
        fd = 46
        pid = 0
        unique_id = 9956686513520984566
        __FUNCTION__ = "smbd_accept_connection"
#24 0x00007f5f8edd5d16 in run_events_poll (ev=0x7f5f92c406b0, pollrtn=1, pfds=0x7f5f940e85a0, num_pfds=8) at ../source3/lib/events.c:257
        pfd = 0x7f5f940e85c8
        flags = 1
        state = 0x7f5f92c416c0
        pollfd_idx = 0x7f5f93e09b00
        fde = 0x7f5f9428da20
        __FUNCTION__ = "run_events_poll"
#25 0x00007f5f8edd5fa5 in s3_event_loop_once (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../source3/lib/events.c:326
        state = 0x7f5f92c416c0
        timeout = 53884
        num_pfds = 8
        ret = 1
        poll_errno = 0
#26 0x00007f5f903a1539 in _tevent_loop_once (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:533
        ret = 0
        nesting_stack_ptr = 0x0
#27 0x00007f5f903a1783 in tevent_common_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:637
        ret = 0
#28 0x00007f5f903a184e in _tevent_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:656
No locals.
#29 0x00007f5f91827f81 in smbd_parent_loop (ev_ctx=0x7f5f92c406b0, parent=0x7f5f92c40920) at ../source3/smbd/server.c:985
        trace_state = {frame = 0x7f5f92c413a0}
        ret = 0
        __FUNCTION__ = "smbd_parent_loop"
#30 0x00007f5f918298df in main (argc=4, argv=0x7fffd3f2cfd8) at ../source3/smbd/server.c:1626
        is_daemon = true
        interactive = false
        Fork = false
        no_process_group = false
        log_stdout = false
        ports = 0x0
        profile_level = 0x0
        opt = -1
        pc = 0x7f5f92c31a30
        print_build_options = false
        long_options = {{longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f5f8dc4d3c0 <poptHelpOptions>, val = 0, descrip = 0x7f5f9182bfc9 "Help options:", argDescrip = 0x0}, {longName = 0x7f5f9182bfd7 "daemon", shortName = 68 'D', argInfo = 0, arg = 0x0, val = 1000, descrip = 0x7f5f9182bfde "Become a daemon (default)", argDescrip = 0x0}, {longName = 0x7f5f9182bff8 "interactive", shortName = 105 'i', argInfo = 0, arg = 0x0, val = 1001, descrip = 0x7f5f9182c008 "Run interactive (not a daemon)", argDescrip = 0x0}, {longName = 0x7f5f9182c027 "foreground", shortName = 70 'F', argInfo = 0, arg = 0x0, val = 1002, descrip = 0x7f5f9182c038 "Run daemon in foreground (for daemontools, etc.)", argDescrip = 0x0}, {longName = 0x7f5f9182c069 "no-process-group", shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 1003, descrip = 0x7f5f9182c080 "Don't create a new process group", argDescrip = 0x0}, {longName = 0x7f5f9182c0a1 "log-stdout", shortName = 83 'S', argInfo = 0, arg = 0x0, val = 1004, descrip = 0x7f5f9182c0ac "Log to stdout", argDescrip = 0x0}, {longName = 0x7f5f9182c0ba "build-options", shortName = 98 'b', argInfo = 0, arg = 0x0, val = 98, descrip = 0x7f5f9182c0c8 "Print build options", argDescrip = 0x0}, {longName = 0x7f5f9182c0dc "port", shortName = 112 'p', argInfo = 1, arg = 0x7fffd3f2cbc0, val = 0, descrip = 0x7f5f9182c0e1 "Listen on the specified ports", argDescrip = 0x0}, {longName = 0x7f5f9182c0ff "profiling-level", shortName = 80 'P', argInfo = 1, arg = 0x7fffd3f2cbc8, val = 0, descrip = 0x7f5f9182c10f "Set profiling level", argDescrip = 0x7f5f9182c123 "PROFILE_LEVEL"}, {longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f5f8f450380 <popt_common_samba>, val = 0, descrip = 0x7f5f9182c131 "Common samba options:", argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 0, descrip = 0x0, argDescrip = 0x0}}
        parent = 0x7f5f92c40920
        frame = 0x7f5f92c2f1a0
        status = {v = 0}
        ev_ctx = 0x7f5f92c406b0
        msg_ctx = 0x7f5f92c407a0
        server_id = {pid = 17147, task_id = 0, vnn = 4294967295, unique_id = 3072772009711624623}
        se = 0x7f5f92c4bcd0
        np_dir = 0x7f5f94183c30 "\320\023\034\223_\177"
        smbd_shim_fns = {cancel_pending_lock_requests_by_fid = 0x7f5f90d19618 <smbd_cancel_pending_lock_requests_by_fid>, send_stat_cache_delete_message = 0x7f5f90d23c49 <smbd_send_stat_cache_delete_message>, change_to_root_user = 0x7f5f90d00d32 <smbd_change_to_root_user>, become_authenticated_pipe_user = 0x7f5f90d00de8 <smbd_become_authenticated_pipe_user>, unbecome_authenticated_pipe_user = 0x7f5f90d00eda <smbd_unbecome_authenticated_pipe_user>, contend_level2_oplocks_begin = 0x7f5f90d95938 <smbd_contend_level2_oplocks_begin>, contend_level2_oplocks_end = 0x7f5f90d959ab <smbd_contend_level2_oplocks_end>, become_root = 0x7f5f90d010fa <smbd_become_root>, unbecome_root = 0x7f5f90d01122 <smbd_unbecome_root>, exit_server = 0x7f5f90d88fab <smbd_exit_server>, exit_server_cleanly = 0x7f5f90d88fc8 <smbd_exit_server_cleanly>}
        __FUNCTION__ = "main"
A debugging session is active.

        Inferior 1 [process 25283] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
Comment 4 Nick Semenkovich 2015-04-20 21:34:12 UTC
And smb.conf for completeness. Nothing interesting on the clients (except that we enforce SMB client signing).

# Global parameters
[global]
	workgroup = CORP
	realm = CORP.EXAMPLE.COM
	netbios name = EXAMPLE-CONTROLLER
	server role = active directory domain controller
	server signing = auto
	dns forwarder = 192.168.0.1
	guest account = nobody

	# Be the master
	domain master = yes
	preferred master = yes

	## Performance tweaks
	# performance boost on EXT4
	strict allocate = yes
	# general performance boost on linux
	use sendfile = true

	# huge write cache performance boost (256k/file cache)
	write cache size = 262144

	# network settings
	socket options = IPTOS_LOWDELAY TCP_NODELAY SO_KEEPALIVE

	# TDB locking performance, per https://download.samba.org/pub/samba/rc/WHATSNEW-4.2.0rc3.txt
	dbwrap_tdb_mutexes:* = yes

	## No printing
	load printers = no
	printing = bsd
	printcap name = /dev/null
	disable spoolss = yes

	panic action = /home/semenko/panic-action %d
	log level = 2
	log file = /ramcache/log.%U


[netlogon]
	path = /usr/local/samba/var/locks/sysvol/corp.example.com/scripts
	read only = No

[sysvol]
	path = /usr/local/samba/var/locks/sysvol
	read only = No

[profiles]
        path = /home/PROFILES
        read only = No

[homes]
        path = /home/CORP
        read only = No
        browseable = No

[dropbox]
	path = /srv/samba/dropbox
	read only = No
	browseable = Yes
	create mask = 664
	directory mask = 775
	guest ok = Yes
Comment 5 Alex Paulo Manzo 2015-04-21 16:40:07 UTC
Have a bug when logon into 2008 R2 an 2012 Terminal services. When I log on this systems the Welcome screen have been stoped (frozen) and return normally when stop/stat samba service.
This bug was observed on 4.2.0 and 4.2.1 version. I return to 4.1.17 version and this systems (2k8 ans 2k12) goes OK.
This problem occurs only with Terminal Services, join domain and other tasks (dns, gpo, acl, file share, squid3 with ntlm auth) it's ok
Comment 6 Nick Semenkovich 2015-04-22 19:50:07 UTC
@alex -- you may have a different bug -- unless you're seeing a similar trace.

Got another one of these (pasted below), with similar parameters. Seems to be happening a couple times a day.

FWIW, in syslog, I see:

Apr 22 14:40:22 runway samba[17158]: [2015/04/22 14:40:22.384650,  0] ../lib/util/util_runcmd.c:324(samba_runcmd_io_handler)
Apr 22 14:40:22 runway samba[17158]:   /usr/local/samba/sbin/samba_spnupdate: ldb: unable to dlopen /usr/local/samba/lib/ldb/dns_notify.so : /usr/local/samba/lib/private/libndr-samba4.so: version `SAMBA_4.3.0PRE1_GIT_84D4270' not found (required by /usr/local/samba/lib/ldb/dns_notify.so)
Apr 22 14:40:22 runway samba[17158]: [2015/04/22 14:40:22.401512,  0] ../lib/util/util_runcmd.c:324(samba_runcmd_io_handler)
Apr 22 14:40:22 runway samba[17158]:   /usr/local/samba/sbin/samba_spnupdate: ldb: unable to dlopen /usr/local/samba/lib/ldb/tombstone_reanimate.so : /usr/local/samba/lib/private/libldbsamba-samba4.so: version `SAMBA_4.3.0PRE1_GIT_84D4270' not found (required by /usr/local/samba/lib/ldb/tombstone_reanimate.so)
Apr 22 14:40:22 runway samba[17158]: [2015/04/22 14:40:22.408256,  0] ../lib/util/util_runcmd.c:324(samba_runcmd_io_handler)
Apr 22 14:40:22 runway samba[17158]:   /usr/local/samba/sbin/samba_dnsupdate: ldb: unable to dlopen /usr/local/samba/lib/ldb/dns_notify.so : /usr/local/samba/lib/private/libndr-samba4.so: version `SAMBA_4.3.0PRE1_GIT_84D4270' not found (required by /usr/local/samba/lib/ldb/dns_notify.so)
Apr 22 14:40:22 runway samba[17158]: [2015/04/22 14:40:22.424108,  0] ../lib/util/util_runcmd.c:324(samba_runcmd_io_handler)
Apr 22 14:40:22 runway samba[17158]:   /usr/local/samba/sbin/samba_dnsupdate: ldb: unable to dlopen /usr/local/samba/lib/ldb/tombstone_reanimate.so : /usr/local/samba/lib/private/libldbsamba-samba4.so: version `SAMBA_4.3.0PRE1_GIT_84D4270' not found (required by /usr/local/samba/lib/ldb/tombstone_reanimate.so)

(I'm somewhat confused by the SAMBA_4.3.0PRE1_GIT_84D4270 tags, since i'm running the 4.2.1 git tag, and build clean / git clean -f -d -x for each new tag.)



[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f5f8d73ea1c in __libc_waitpid (pid=11243, stat_loc=stat_loc@entry=0x7fffd3f2bb50, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
#0  0x00007f5f8d73ea1c in __libc_waitpid (pid=11243, stat_loc=stat_loc@entry=0x7fffd3f2bb50, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
        resultvar = 18446744073709551104
        oldtype = -1918642432
#1  0x00007f5f8d6c28e2 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
        __result = <optimized out>
        _buffer = {__routine = 0x7f5f8d6c2b80 <cancel_handler>, __arg = 0x7fffd3f2bb20, __canceltype = 0, __prev = 0x0}
        _avail = 1
        status = -739067008
        save = <optimized out>
        pid = 11243
        sa = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {65536, 0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x7f5f94542a70}
        omask = {__val = {7296, 140048460905200, 140048371045184, 140048439648256, 140736749293520, 0, 0, 140048424756938, 999428753, 140048395634696, 4294967295, 292087364336, 0, 206158430224, 140736749288624, 140048437551232}}
#2  0x00007f5f8edb5662 in smb_panic_s3 (why=0x7f5f911d0aad "internal error") at ../source3/lib/util.c:801
        cmd = 0x7f5f94542a70 "/home/semenko/panic-action 9820"
        result = 32607
        __FUNCTION__ = "smb_panic_s3"
#3  0x00007f5f911c9f15 in smb_panic (why=0x7f5f911d0aad "internal error") at ../lib/util/fault.c:166
No locals.
#4  0x00007f5f911c9bed in fault_report (sig=11) at ../lib/util/fault.c:83
        counter = 1
        __FUNCTION__ = "fault_report"
#5  0x00007f5f911c9c02 in sig_fault (sig=11) at ../lib/util/fault.c:94
No locals.
#6  <signal handler called>
No locals.
#7  0x00007f5f90d80b0d in smbXsrv_session_find_channel (session=0x7f5f94370eb0, conn=0x7f5f93ec6270, _c=0x7fffd3f2c328) at ../source3/smbd/smbXsrv_session.c:1289
        i = 0
#8  0x00007f5f90d55a6a in smbd_smb2_signing_key (session=0x7f5f94370eb0, xconn=0x7f5f93ec6270) at ../source3/smbd/smb2_server.c:1429
        c = 0x0
        status = {v = 5}
        key = {data = 0x0, length = 0}
#9  0x00007f5f90d59705 in smbd_smb2_request_reply (req=0x7f5f9367dae0) at ../source3/smbd/smb2_server.c:2499
        x = 0x7f5f94370eb0
        signing_key = {data = 0x7f5f938f6e90 "\260\006Ē_\177", length = 549}
        xconn = 0x7f5f93ec6270
        first_idx = 1
        firsttf = 0x7f5f9367dc40
        outhdr = 0x7f5f9367dc50
        outdyn = 0x7f5f9367dc70
        status = {v = 2466299888}
#10 0x00007f5f90d5a314 in smbd_smb2_request_done_ex (req=0x7f5f9367dae0, status=..., body=..., dyn=0x7fffd3f2c540, location=0x7f5f90ee0878 "../source3/smbd/smb2_server.c:2737") at ../source3/smbd/smb2_server.c:2673
        outhdr = 0x7f5f9367dc80 "\376SMB@"
        outbody_v = 0x7f5f9367dc60
        outdyn_v = 0x7f5f9367dc70
        next_command_ofs = 0
        __FUNCTION__ = "smbd_smb2_request_done_ex"
#11 0x00007f5f90d5a5be in smbd_smb2_request_error_ex (req=0x7f5f9367dae0, status=..., info=0x7fffd3f2c540, location=0x7f5f90ee9310 "../source3/smbd/smb2_notify.c:123") at ../source3/smbd/smb2_server.c:2737
        xconn = 0x7f5f93ec6270
        body = {data = 0x7f5f9367dcc0 "\t", length = 8}
        _dyn = {data = 0x7f5f9367dcc8 "", length = 1}
        outhdr = 0x7f5f9367dc80 "\376SMB@"
        unread_bytes = 0
        __FUNCTION__ = "smbd_smb2_request_error_ex"
#12 0x00007f5f90d76723 in smbd_smb2_request_notify_done (subreq=0x0) at ../source3/smbd/smb2_notify.c:123
        req = 0x7f5f9367dae0
        outbody = {data = 0x7f5f9367de70 "tfא_\177", length = 140048471678480}
        outdyn = {data = 0x7fffd3f2c600 "H\227\356\220_\177", length = 140048422621689}
        out_output_buffer_offset = 0
        out_output_buffer = {data = 0x0, length = 0}
        status = {v = 268}
        error = {v = 2431481472}
#13 0x00007f5f903a3016 in _tevent_req_notify_callback (req=0x7f5f9367de70, location=0x7f5f90ee9748 "../source3/smbd/smb2_notify.c:355") at ../lib/tevent/tevent_req.c:112
No locals.
#14 0x00007f5f903a30e9 in tevent_req_finish (req=0x7f5f9367de70, state=TEVENT_REQ_USER_ERROR, location=0x7f5f90ee9748 "../source3/smbd/smb2_notify.c:355") at ../lib/tevent/tevent_req.c:149
No locals.
#15 0x00007f5f903a320e in tevent_req_trigger (ev=0x7f5f92c406b0, im=0x7f5f9367df50, private_data=0x7f5f9367de70) at ../lib/tevent/tevent_req.c:206
        req = 0x7f5f9367de70
#16 0x00007f5f903a23f2 in tevent_common_loop_immediate (ev=0x7f5f92c406b0) at ../lib/tevent/tevent_immediate.c:135
        im = 0x7f5f9367df50
        handler = 0x7f5f903a31bf <tevent_req_trigger>
        private_data = 0x7f5f9367de70
#17 0x00007f5f8edd581d in run_events_poll (ev=0x7f5f92c406b0, pollrtn=0, pfds=0x0, num_pfds=0) at ../source3/lib/events.c:192
        state = 0x7f5f911b8725 <talloc_pop>
        pollfd_idx = 0x7fffd3f2c730
        fde = 0x7fffd3f2c730
        __FUNCTION__ = "run_events_poll"
#18 0x00007f5f8edd5e91 in s3_event_loop_once (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../source3/lib/events.c:303
        state = 0x7f5f92c416c0
        timeout = 2147483647
        num_pfds = 32767
        ret = -739063936
        poll_errno = 32767
#19 0x00007f5f903a1539 in _tevent_loop_once (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:533
        ret = 0
        nesting_stack_ptr = 0x0
#20 0x00007f5f903a1783 in tevent_common_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:637
        ret = 0
#21 0x00007f5f903a184e in _tevent_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:656
No locals.
#22 0x00007f5f90d3dd04 in smbd_process (ev_ctx=0x7f5f92c406b0, msg_ctx=0x7f5f92c407a0, sock_fd=46, interactive=false) at ../source3/smbd/process.c:3992
        trace_state = {frame = 0x7f5f94542a10, smbd_idle_profstamp = 0}
        client = 0x7f5f9391de40
        sconn = 0x7f5f93000320
        xconn = 0x7f5f93ec6270
        locaddr = 0x7f5f9314ab20 "\001db"
        remaddr = 0x7f5f93afa550 "p\256\343\223_\177"
        ret = 32607
        status = {v = 0}
        __FUNCTION__ = "smbd_process"
#23 0x00007f5f9182716b in smbd_accept_connection (ev=0x7f5f92c406b0, fde=0x7f5f9428da20, flags=1, private_data=0x7f5f94658be0) at ../source3/smbd/server.c:627
        status = {v = 0}
        s = 0x0
        msg_ctx = 0x7f5f92c407a0
        addr = {ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 16 times>, "\030FT\223_\177\000\000 \312\362\323\377\177\000\000\240\311\362\323\377\177\000\000\205f\034\221_\177\000\000\030FT\223_\177\000\000 \312\362\323\377\177\000\000;\000\000\000\000\000\000\000f<\017\000\000\000\000\000@\312\362\323\377\177\000\000\243Wݎ_\177\000\000\222\311\067U\000\000\000\000h\312\362\323\377\177\000"}
        in_addrlen = 16
        fd = 46
        pid = 0
        unique_id = 2121136114886913927
        __FUNCTION__ = "smbd_accept_connection"
#24 0x00007f5f8edd5d16 in run_events_poll (ev=0x7f5f92c406b0, pollrtn=1, pfds=0x7f5f940e85a0, num_pfds=8) at ../source3/lib/events.c:257
        pfd = 0x7f5f940e85d0
        flags = 1
        state = 0x7f5f92c416c0
        pollfd_idx = 0x7f5f93e09b00
        fde = 0x7f5f9428da20
        __FUNCTION__ = "run_events_poll"
#25 0x00007f5f8edd5fa5 in s3_event_loop_once (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../source3/lib/events.c:326
        state = 0x7f5f92c416c0
        timeout = 59999
        num_pfds = 8
        ret = 1
        poll_errno = 0
#26 0x00007f5f903a1539 in _tevent_loop_once (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:533
        ret = 0
        nesting_stack_ptr = 0x0
#27 0x00007f5f903a1783 in tevent_common_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:637
        ret = 0
#28 0x00007f5f903a184e in _tevent_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:656
No locals.
#29 0x00007f5f91827f81 in smbd_parent_loop (ev_ctx=0x7f5f92c406b0, parent=0x7f5f92c40920) at ../source3/smbd/server.c:985
        trace_state = {frame = 0x7f5f92c413a0}
        ret = 0
        __FUNCTION__ = "smbd_parent_loop"
#30 0x00007f5f918298df in main (argc=4, argv=0x7fffd3f2cfd8) at ../source3/smbd/server.c:1626
        is_daemon = true
        interactive = false
        Fork = false
        no_process_group = false
        log_stdout = false
        ports = 0x0
        profile_level = 0x0
        opt = -1
        pc = 0x7f5f92c31a30
        print_build_options = false
        long_options = {{longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f5f8dc4d3c0 <poptHelpOptions>, val = 0, descrip = 0x7f5f9182bfc9 "Help options:", argDescrip = 0x0}, {longName = 0x7f5f9182bfd7 "daemon", shortName = 68 'D', argInfo = 0, arg = 0x0, val = 1000, descrip = 0x7f5f9182bfde "Become a daemon (default)", argDescrip = 0x0}, {longName = 0x7f5f9182bff8 "interactive", shortName = 105 'i', argInfo = 0, arg = 0x0, val = 1001, descrip = 0x7f5f9182c008 "Run interactive (not a daemon)", argDescrip = 0x0}, {longName = 0x7f5f9182c027 "foreground", shortName = 70 'F', argInfo = 0, arg = 0x0, val = 1002, descrip = 0x7f5f9182c038 "Run daemon in foreground (for daemontools, etc.)", argDescrip = 0x0}, {longName = 0x7f5f9182c069 "no-process-group", shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 1003, descrip = 0x7f5f9182c080 "Don't create a new process group", argDescrip = 0x0}, {longName = 0x7f5f9182c0a1 "log-stdout", shortName = 83 'S', argInfo = 0, arg = 0x0, val = 1004, descrip = 0x7f5f9182c0ac "Log to stdout", argDescrip = 0x0}, {longName = 0x7f5f9182c0ba "build-options", shortName = 98 'b', argInfo = 0, arg = 0x0, val = 98, descrip = 0x7f5f9182c0c8 "Print build options", argDescrip = 0x0}, {longName = 0x7f5f9182c0dc "port", shortName = 112 'p', argInfo = 1, arg = 0x7fffd3f2cbc0, val = 0, descrip = 0x7f5f9182c0e1 "Listen on the specified ports", argDescrip = 0x0}, {longName = 0x7f5f9182c0ff "profiling-level", shortName = 80 'P', argInfo = 1, arg = 0x7fffd3f2cbc8, val = 0, descrip = 0x7f5f9182c10f "Set profiling level", argDescrip = 0x7f5f9182c123 "PROFILE_LEVEL"}, {longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f5f8f450380 <popt_common_samba>, val = 0, descrip = 0x7f5f9182c131 "Common samba options:", argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 0, descrip = 0x0, argDescrip = 0x0}}
        parent = 0x7f5f92c40920
        frame = 0x7f5f92c2f1a0
        status = {v = 0}
        ev_ctx = 0x7f5f92c406b0
        msg_ctx = 0x7f5f92c407a0
        server_id = {pid = 17147, task_id = 0, vnn = 4294967295, unique_id = 3072772009711624623}
        se = 0x7f5f92c4bcd0
        np_dir = 0x7f5f94183c30 "@INDEXLIST"
        smbd_shim_fns = {cancel_pending_lock_requests_by_fid = 0x7f5f90d19618 <smbd_cancel_pending_lock_requests_by_fid>, send_stat_cache_delete_message = 0x7f5f90d23c49 <smbd_send_stat_cache_delete_message>, change_to_root_user = 0x7f5f90d00d32 <smbd_change_to_root_user>, become_authenticated_pipe_user = 0x7f5f90d00de8 <smbd_become_authenticated_pipe_user>, unbecome_authenticated_pipe_user = 0x7f5f90d00eda <smbd_unbecome_authenticated_pipe_user>, contend_level2_oplocks_begin = 0x7f5f90d95938 <smbd_contend_level2_oplocks_begin>, contend_level2_oplocks_end = 0x7f5f90d959ab <smbd_contend_level2_oplocks_end>, become_root = 0x7f5f90d010fa <smbd_become_root>, unbecome_root = 0x7f5f90d01122 <smbd_unbecome_root>, exit_server = 0x7f5f90d88fab <smbd_exit_server>, exit_server_cleanly = 0x7f5f90d88fc8 <smbd_exit_server_cleanly>}
        __FUNCTION__ = "main"
A debugging session is active.
Comment 7 Volker Lendecke 2015-04-22 20:08:32 UTC
Just to give you an ack. This is high in my list, I'm just busy with customers right now, sorry
Comment 8 Jeremy Allison 2015-04-22 23:15:22 UTC
Can you also upload a wireshark trace from the client connecting to the crashing smbd here ?

We should see a TCP RST being sent back when smbd crashes and I'd dearly love
to see what's in the packets before that :-).
Comment 9 Nick Semenkovich 2015-04-27 17:19:39 UTC
(In reply to Jeremy Allison from comment #8)

Cool -- I don't have a clean repro. Is there an easy tcpdump command I can run on the server itself that isn't too promiscuous?
Comment 10 Jeremy Allison 2015-04-27 17:30:05 UTC
If you can predict what client IP addr it will affect then you can collect on port 445 between the server and that client only.
Comment 11 Nick Semenkovich 2015-04-28 19:51:53 UTC
Ok, here's another crash, from the same client.

I'm guessing the machine goes to sleep around 13:19, then wakes up at 14:25 and immediately prompts a crash.

Captured with
$ tcpdump -p -s 0 -w crashlog.pcap "host 192.168.0.109 and port 445"


The user log:


[2015/04/28 13:19:58.236611,  2] ../source3/smbd/close.c:780(close_normal_file)
  CORP\xxx closed file xxx/AppData/Roaming/Microsoft/Templates/NormalEmail.dotm (numopen=14) NT_STATUS_OK
[2015/04/28 14:25:53.827603,  2] ../source3/smbd/close.c:780(close_normal_file)
  CORP\xxx closed file xxx/Documents/OUTLOOK/Trace-2015-04-28-10-47-21-p3848.log (numopen=12) NT_STATUS_OK
[2015/04/28 14:25:53.828019,  2] ../source3/smbd/close.c:780(close_normal_file)
  CORP\xxx closed file xxxx/AppData/Roaming/Microsoft/Outlook/xxx@xxxx com - Google Apps 2.srs (numopen=10) NT_STATUS_OK
[2015/04/28 14:25:53.828857,  2] ../source3/smbd/service.c:1134(close_cnum)
  192.168.0.109 (ipv4:192.168.0.109:62449) closed connection to service dropbox
[2015/04/28 14:25:53.829161,  2] ../source3/smbd/service.c:1134(close_cnum)
  192.168.0.109 (ipv4:192.168.0.109:62449) closed connection to service xxx
[2015/04/28 14:25:53.830054,  0] ../lib/util/fault.c:78(fault_report)
  ===============================================================
[2015/04/28 14:25:53.830176,  0] ../lib/util/fault.c:79(fault_report)
  INTERNAL ERROR: Signal 11 in pid 29529 (4.2.1)
  Please read the Trouble-Shooting section of the Samba HOWTO
[2015/04/28 14:25:53.830202,  0] ../lib/util/fault.c:81(fault_report)
  ===============================================================
[2015/04/28 14:25:53.830221,  0] ../source3/lib/util.c:788(smb_panic_s3)
  PANIC (pid 29529): internal error
[2015/04/28 14:25:53.831196,  0] ../source3/lib/util.c:899(log_stack_trace)
  BACKTRACE: 32 stack frames:
   #0 /usr/local/samba/lib/libsmbconf.so.0(log_stack_trace+0x1f) [0x7f5f8edb575c]
   #1 /usr/local/samba/lib/libsmbconf.so.0(smb_panic_s3+0x6f) [0x7f5f8edb55a7]
   #2 /usr/local/samba/lib/libsamba-util.so.0(smb_panic+0x28) [0x7f5f911c9f15]
   #3 /usr/local/samba/lib/libsamba-util.so.0(+0x2abed) [0x7f5f911c9bed]
   #4 /usr/local/samba/lib/libsamba-util.so.0(+0x2ac02) [0x7f5f911c9c02]
   #5 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc90) [0x7f5f913e8c90]
   #6 /usr/local/samba/lib/private/libsmbd-base-samba4.so(smbXsrv_session_find_channel+0x6c) [0x7f5f90d80b0d]
   #7 /usr/local/samba/lib/private/libsmbd-base-samba4.so(+0x194a6a) [0x7f5f90d55a6a]
   #8 /usr/local/samba/lib/private/libsmbd-base-samba4.so(+0x198705) [0x7f5f90d59705]
   #9 /usr/local/samba/lib/private/libsmbd-base-samba4.so(smbd_smb2_request_done_ex+0x589) [0x7f5f90d5a314]
   #10 /usr/local/samba/lib/private/libsmbd-base-samba4.so(smbd_smb2_request_error_ex+0x29f) [0x7f5f90d5a5be]
   #11 /usr/local/samba/lib/private/libsmbd-base-samba4.so(+0x1b5723) [0x7f5f90d76723]
   #12 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a) [0x7f5f903a3016]
   #13 /usr/local/samba/lib/private/libtevent.so.0(+0x60e9) [0x7f5f903a30e9]
   #14 /usr/local/samba/lib/private/libtevent.so.0(+0x620e) [0x7f5f903a320e]
   #15 /usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_immediate+0x1f5) [0x7f5f903a23f2]
   #16 /usr/local/samba/lib/libsmbconf.so.0(run_events_poll+0x56) [0x7f5f8edd581d]
   #17 /usr/local/samba/lib/libsmbconf.so.0(+0x44e91) [0x7f5f8edd5e91]
   #18 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_once+0xf4) [0x7f5f903a1539]
   #19 /usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_wait+0x25) [0x7f5f903a1783]
   #20 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_wait+0x2b) [0x7f5f903a184e]
   #21 /usr/local/samba/lib/private/libsmbd-base-samba4.so(smbd_process+0xb28) [0x7f5f90d3dd04]
   #22 /usr/local/samba/sbin/smbd(+0xb16b) [0x7f5f9182716b]
   #23 /usr/local/samba/lib/libsmbconf.so.0(run_events_poll+0x54f) [0x7f5f8edd5d16]
   #24 /usr/local/samba/lib/libsmbconf.so.0(+0x44fa5) [0x7f5f8edd5fa5]
   #25 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_once+0xf4) [0x7f5f903a1539]
   #26 /usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_wait+0x25) [0x7f5f903a1783]
   #27 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_wait+0x2b) [0x7f5f903a184e]
   #28 /usr/local/samba/sbin/smbd(+0xbf81) [0x7f5f91827f81]
   #29 /usr/local/samba/sbin/smbd(main+0x17a7) [0x7f5f918298df]
   #30 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f5f8d69fec5]
   #31 /usr/local/samba/sbin/smbd(+0x5ed9) [0x7f5f91821ed9]
[2015/04/28 14:25:53.831476,  0] ../source3/lib/util.c:800(smb_panic_s3)
  smb_panic(): calling panic action [/home/semenko/panic-action 29529]
[2015/04/28 14:25:53.854946,  2] ../lib/util/modules.c:191(do_smb_load_module)
  Module 'acl_xattr' loaded
[2015/04/28 14:25:53.855005,  2] ../source3/modules/vfs_acl_xattr.c:193(connect_acl_xattr)
  connect_acl_xattr: setting 'inherit acls = true' 'dos filemode = true' and 'force unknown acl user = true' for service xxx
[2015/04/28 14:25:53.860694,  2] ../source3/smbd/service.c:858(make_connection_snum)
  192.168.0.109 (ipv4:192.168.0.109:54112) connect to service xxx initially as user CORP\xxx (uid=3000054, gid=100) (pid 30530)
31	../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
[2015/04/28 14:25:55.144438,  2] ../source3/smbd/open.c:1005(open_file)
  CORP\xxx opened file xxx/Start Menu/Programs/Accessories/Desktop.ini read=Yes write=No (numopen=4)
[2015/04/28 14:25:55.152181,  2] ../source3/modules/vfs_acl_xattr.c:193(connect_acl_xattr)
  connect_acl_xattr: setting 'inherit acls = true' 'dos filemode = true' and 'force unknown acl user = true' for service IPC$



The trace:


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f5f8d73ea1c in __libc_waitpid (pid=30531, stat_loc=stat_loc@entry=0x7fffd3f2bb50, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
#0  0x00007f5f8d73ea1c in __libc_waitpid (pid=30531, stat_loc=stat_loc@entry=0x7fffd3f2bb50, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
        resultvar = 18446744073709551104
        oldtype = -1918642432
#1  0x00007f5f8d6c28e2 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
        __result = <optimized out>
        _buffer = {__routine = 0x7f5f8d6c2b80 <cancel_handler>, __arg = 0x7fffd3f2bb20, __canceltype = 0, __prev = 0x0}
        _avail = 1
        status = -739067008
        save = <optimized out>
        pid = 30531
        sa = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {65536, 0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x7f5f93f903a0}
        omask = {__val = {7296, 140048460905200, 140048371045184, 140048439648256, 140736749293520, 0, 0, 140048424756938, 999428753, 140048395634696, 4294967295, 296382331632, 0, 206158430224, 140736749288624, 140048437551232}}
#2  0x00007f5f8edb5662 in smb_panic_s3 (why=0x7f5f911d0aad "internal error") at ../source3/lib/util.c:801
        cmd = 0x7f5f93f903a0 "/home/semenko/panic-action 29529"
        result = 32607
        __FUNCTION__ = "smb_panic_s3"
#3  0x00007f5f911c9f15 in smb_panic (why=0x7f5f911d0aad "internal error") at ../lib/util/fault.c:166
No locals.
#4  0x00007f5f911c9bed in fault_report (sig=11) at ../lib/util/fault.c:83
        counter = 1
        __FUNCTION__ = "fault_report"
#5  0x00007f5f911c9c02 in sig_fault (sig=11) at ../lib/util/fault.c:94
No locals.
#6  <signal handler called>
No locals.
#7  0x00007f5f90d80b0d in smbXsrv_session_find_channel (session=0x7f5f937cbaa0, conn=0x7f5f93ec6270, _c=0x7fffd3f2c328) at ../source3/smbd/smbXsrv_session.c:1289
        i = 0
#8  0x00007f5f90d55a6a in smbd_smb2_signing_key (session=0x7f5f937cbaa0, xconn=0x7f5f93ec6270) at ../source3/smbd/smb2_server.c:1429
        c = 0x0
        status = {v = 5}
        key = {data = 0x0, length = 0}
#9  0x00007f5f90d59705 in smbd_smb2_request_reply (req=0x7f5f932e75d0) at ../source3/smbd/smb2_server.c:2499
        x = 0x7f5f937cbaa0
        signing_key = {data = 0x7f5f93ce2c80 "\260\006Ē_\177", length = 549}
        xconn = 0x7f5f93ec6270
        first_idx = 1
        firsttf = 0x7f5f932e7730
        outhdr = 0x7f5f932e7740
        outdyn = 0x7f5f932e7760
        status = {v = 2484895296}
#10 0x00007f5f90d5a314 in smbd_smb2_request_done_ex (req=0x7f5f932e75d0, status=..., body=..., dyn=0x7fffd3f2c540, location=0x7f5f90ee0878 "../source3/smbd/smb2_server.c:2737") at ../source3/smbd/smb2_server.c:2673
        outhdr = 0x7f5f932e7770 "\376SMB@"
        outbody_v = 0x7f5f932e7750
        outdyn_v = 0x7f5f932e7760
        next_command_ofs = 0
        __FUNCTION__ = "smbd_smb2_request_done_ex"
#11 0x00007f5f90d5a5be in smbd_smb2_request_error_ex (req=0x7f5f932e75d0, status=..., info=0x7fffd3f2c540, location=0x7f5f90ee9310 "../source3/smbd/smb2_notify.c:123") at ../source3/smbd/smb2_server.c:2737
        xconn = 0x7f5f93ec6270
        body = {data = 0x7f5f932e77b0 "\t", length = 8}
        _dyn = {data = 0x7f5f932e77b8 "", length = 1}
        outhdr = 0x7f5f932e7770 "\376SMB@"
        unread_bytes = 0
        __FUNCTION__ = "smbd_smb2_request_error_ex"
#12 0x00007f5f90d76723 in smbd_smb2_request_notify_done (subreq=0x0) at ../source3/smbd/smb2_notify.c:123
        req = 0x7f5f932e75d0
        outbody = {data = 0x7f5f932e7960 "tfא_\177", length = 140048467917056}
        outdyn = {data = 0x7fffd3f2c600 "H\227\356\220_\177", length = 140048422621689}
        out_output_buffer_offset = 0
        out_output_buffer = {data = 0x0, length = 0}
        status = {v = 268}
        error = {v = 2431481472}
#13 0x00007f5f903a3016 in _tevent_req_notify_callback (req=0x7f5f932e7960, location=0x7f5f90ee9748 "../source3/smbd/smb2_notify.c:355") at ../lib/tevent/tevent_req.c:112
No locals.
#14 0x00007f5f903a30e9 in tevent_req_finish (req=0x7f5f932e7960, state=TEVENT_REQ_USER_ERROR, location=0x7f5f90ee9748 "../source3/smbd/smb2_notify.c:355") at ../lib/tevent/tevent_req.c:149
No locals.
#15 0x00007f5f903a320e in tevent_req_trigger (ev=0x7f5f92c406b0, im=0x7f5f932e7a40, private_data=0x7f5f932e7960) at ../lib/tevent/tevent_req.c:206
        req = 0x7f5f932e7960
#16 0x00007f5f903a23f2 in tevent_common_loop_immediate (ev=0x7f5f92c406b0) at ../lib/tevent/tevent_immediate.c:135
        im = 0x7f5f932e7a40
        handler = 0x7f5f903a31bf <tevent_req_trigger>
        private_data = 0x7f5f932e7960
#17 0x00007f5f8edd581d in run_events_poll (ev=0x7f5f92c406b0, pollrtn=0, pfds=0x0, num_pfds=0) at ../source3/lib/events.c:192
        state = 0x7f5f911b8725 <talloc_pop>
        pollfd_idx = 0x7fffd3f2c730
        fde = 0x7fffd3f2c730
        __FUNCTION__ = "run_events_poll"
#18 0x00007f5f8edd5e91 in s3_event_loop_once (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../source3/lib/events.c:303
        state = 0x7f5f92c416c0
        timeout = 2147483647
        num_pfds = 32767
        ret = -739063936
        poll_errno = 32767
#19 0x00007f5f903a1539 in _tevent_loop_once (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:533
        ret = 0
        nesting_stack_ptr = 0x0
#20 0x00007f5f903a1783 in tevent_common_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:637
        ret = 0
#21 0x00007f5f903a184e in _tevent_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f90ed8170 "../source3/smbd/process.c:3992") at ../lib/tevent/tevent.c:656
No locals.
#22 0x00007f5f90d3dd04 in smbd_process (ev_ctx=0x7f5f92c406b0, msg_ctx=0x7f5f92c407a0, sock_fd=46, interactive=false) at ../source3/smbd/process.c:3992
        trace_state = {frame = 0x7f5f93f901f0, smbd_idle_profstamp = 0}
        client = 0x7f5f9391de40
        sconn = 0x7f5f93000320
        xconn = 0x7f5f93ec6270
        locaddr = 0x7f5f9314ab20 "\220\267\001\224_\177"
        remaddr = 0x7f5f93afa550 "ipv4:192.168.0.109:62449"
        ret = 32607
        status = {v = 0}
        __FUNCTION__ = "smbd_process"
#23 0x00007f5f9182716b in smbd_accept_connection (ev=0x7f5f92c406b0, fde=0x7f5f9428da20, flags=1, private_data=0x7f5f94658be0) at ../source3/smbd/server.c:627
        status = {v = 0}
        s = 0x0
        msg_ctx = 0x7f5f92c407a0
        addr = {ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 16 times>, "\030FT\223_\177\000\000 \312\362\323\377\177\000\000\240\311\362\323\377\177\000\000\205f\034\221_\177\000\000\030FT\223_\177\000\000 \312\362\323\377\177\000\000;\000\000\000\000\000\000\000\354:\017\000\000\000\000\000@\312\362\323\377\177\000\000\243Wݎ_\177\000\000\031\253?U\000\000\000\000h\312\362\323\377\177\000"}
        in_addrlen = 16
        fd = 46
        pid = 0
        unique_id = 12802382382026209428
        __FUNCTION__ = "smbd_accept_connection"
#24 0x00007f5f8edd5d16 in run_events_poll (ev=0x7f5f92c406b0, pollrtn=1, pfds=0x7f5f92f77620, num_pfds=8) at ../source3/lib/events.c:257
        pfd = 0x7f5f92f77648
        flags = 1
        state = 0x7f5f92c416c0
        pollfd_idx = 0x7f5f93e09b00
        fde = 0x7f5f9428da20
        __FUNCTION__ = "run_events_poll"
#25 0x00007f5f8edd5fa5 in s3_event_loop_once (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../source3/lib/events.c:326
        state = 0x7f5f92c416c0
        timeout = 59999
        num_pfds = 8
        ret = 1
        poll_errno = 0
#26 0x00007f5f903a1539 in _tevent_loop_once (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:533
        ret = 0
        nesting_stack_ptr = 0x0
#27 0x00007f5f903a1783 in tevent_common_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:637
        ret = 0
#28 0x00007f5f903a184e in _tevent_loop_wait (ev=0x7f5f92c406b0, location=0x7f5f9182beca "../source3/smbd/server.c:985") at ../lib/tevent/tevent.c:656
No locals.
#29 0x00007f5f91827f81 in smbd_parent_loop (ev_ctx=0x7f5f92c406b0, parent=0x7f5f92c40920) at ../source3/smbd/server.c:985
        trace_state = {frame = 0x7f5f92c413a0}
        ret = 0
        __FUNCTION__ = "smbd_parent_loop"
#30 0x00007f5f918298df in main (argc=4, argv=0x7fffd3f2cfd8) at ../source3/smbd/server.c:1626
        is_daemon = true
        interactive = false
        Fork = false
        no_process_group = false
        log_stdout = false
        ports = 0x0
        profile_level = 0x0
        opt = -1
        pc = 0x7f5f92c31a30
        print_build_options = false
        long_options = {{longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f5f8dc4d3c0 <poptHelpOptions>, val = 0, descrip = 0x7f5f9182bfc9 "Help options:", argDescrip = 0x0}, {longName = 0x7f5f9182bfd7 "daemon", shortName = 68 'D', argInfo = 0, arg = 0x0, val = 1000, descrip = 0x7f5f9182bfde "Become a daemon (default)", argDescrip = 0x0}, {longName = 0x7f5f9182bff8 "interactive", shortName = 105 'i', argInfo = 0, arg = 0x0, val = 1001, descrip = 0x7f5f9182c008 "Run interactive (not a daemon)", argDescrip = 0x0}, {longName = 0x7f5f9182c027 "foreground", shortName = 70 'F', argInfo = 0, arg = 0x0, val = 1002, descrip = 0x7f5f9182c038 "Run daemon in foreground (for daemontools, etc.)", argDescrip = 0x0}, {longName = 0x7f5f9182c069 "no-process-group", shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 1003, descrip = 0x7f5f9182c080 "Don't create a new process group", argDescrip = 0x0}, {longName = 0x7f5f9182c0a1 "log-stdout", shortName = 83 'S', argInfo = 0, arg = 0x0, val = 1004, descrip = 0x7f5f9182c0ac "Log to stdout", argDescrip = 0x0}, {longName = 0x7f5f9182c0ba "build-options", shortName = 98 'b', argInfo = 0, arg = 0x0, val = 98, descrip = 0x7f5f9182c0c8 "Print build options", argDescrip = 0x0}, {longName = 0x7f5f9182c0dc "port", shortName = 112 'p', argInfo = 1, arg = 0x7fffd3f2cbc0, val = 0, descrip = 0x7f5f9182c0e1 "Listen on the specified ports", argDescrip = 0x0}, {longName = 0x7f5f9182c0ff "profiling-level", shortName = 80 'P', argInfo = 1, arg = 0x7fffd3f2cbc8, val = 0, descrip = 0x7f5f9182c10f "Set profiling level", argDescrip = 0x7f5f9182c123 "PROFILE_LEVEL"}, {longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x7f5f8f450380 <popt_common_samba>, val = 0, descrip = 0x7f5f9182c131 "Common samba options:", argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 0, descrip = 0x0, argDescrip = 0x0}}
        parent = 0x7f5f92c40920
        frame = 0x7f5f92c2f1a0
        status = {v = 0}
        ev_ctx = 0x7f5f92c406b0
        msg_ctx = 0x7f5f92c407a0
        server_id = {pid = 17147, task_id = 0, vnn = 4294967295, unique_id = 3072772009711624623}
        se = 0x7f5f92c4bcd0
        np_dir = 0x7f5f94183c30 "\320\023\034\223_\177"
        smbd_shim_fns = {cancel_pending_lock_requests_by_fid = 0x7f5f90d19618 <smbd_cancel_pending_lock_requests_by_fid>, send_stat_cache_delete_message = 0x7f5f90d23c49 <smbd_send_stat_cache_delete_message>, change_to_root_user = 0x7f5f90d00d32 <smbd_change_to_root_user>, become_authenticated_pipe_user = 0x7f5f90d00de8 <smbd_become_authenticated_pipe_user>, unbecome_authenticated_pipe_user = 0x7f5f90d00eda <smbd_unbecome_authenticated_pipe_user>, contend_level2_oplocks_begin = 0x7f5f90d95938 <smbd_contend_level2_oplocks_begin>, contend_level2_oplocks_end = 0x7f5f90d959ab <smbd_contend_level2_oplocks_end>, become_root = 0x7f5f90d010fa <smbd_become_root>, unbecome_root = 0x7f5f90d01122 <smbd_unbecome_root>, exit_server = 0x7f5f90d88fab <smbd_exit_server>, exit_server_cleanly = 0x7f5f90d88fc8 <smbd_exit_server_cleanly>}
        __FUNCTION__ = "main"
A debugging session is active.

        Inferior 1 [process 29529] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
Comment 12 Nick Semenkovich 2015-04-28 19:53:39 UTC
Created attachment 10995 [details]
packets right around crash

$ tcpdump -p -s 0 -w crashlog.pcap "host 192.168.0.109 and port 445"
Comment 13 chrno-sphered 2015-05-01 12:15:48 UTC
I'm getting this as well. The last working version for me is 4.1.9, everything since 4.2.0 makes smbd crash. I'm mainly accessing my arch linux x64 shares from a win8.1u1 PC, playing videos with mpc-hc. 
Samba is configured to use usershares following the arch wiki [1]

[2015/05/01 13:59:21.530609,  0] ../source3/lib/util.c:788(smb_panic_s3)
May 01 13:59:21 Zacate smbd[13949]: PANIC (pid 13949): internal error
May 01 13:59:21 Zacate smbd[13949]: [2015/05/01 13:59:21.535782,  0] ../source3/lib/util.c:899(log_stack_trace)
May 01 13:59:21 Zacate smbd[13949]: BACKTRACE: 29 stack frames:
May 01 13:59:21 Zacate smbd[13949]: #0 /usr/lib/libsmbconf.so.0(log_stack_trace+0x1a) [0x7f29ae6df23a]
May 01 13:59:21 Zacate smbd[13949]: #1 /usr/lib/libsmbconf.so.0(smb_panic_s3+0x20) [0x7f29ae6df310]
May 01 13:59:21 Zacate smbd[13949]: #2 /usr/lib/libsamba-util.so.0(smb_panic+0x2f) [0x7f29b052c0ff]
May 01 13:59:21 Zacate smbd[13949]: #3 /usr/lib/libsamba-util.so.0(+0x1a316) [0x7f29b052c316]
May 01 13:59:21 Zacate smbd[13949]: #4 /usr/lib/libpthread.so.0(+0x10740) [0x7f29b0754740]
May 01 13:59:21 Zacate smbd[13949]: #5 /usr/lib/libc.so.6(strlen+0x2a) [0x7f29ace0320a]
May 01 13:59:21 Zacate smbd[13949]: #6 /usr/lib/libsamba-util.so.0(push_ucs2_talloc+0x1e) [0x7f29b05249ae]
May 01 13:59:21 Zacate smbd[13949]: #7 /usr/lib/samba/libcliauth-samba4.so(E_md4hash+0x1e) [0x7f29aed6339e]
May 01 13:59:21 Zacate smbd[13949]: #8 /usr/lib/samba/libsmbd-base-samba4.so(create_volume_objectid+0x3a) [0x7f29b00d32fa]
May 01 13:59:21 Zacate smbd[13949]: #9 /usr/lib/samba/libsmbd-base-samba4.so(+0x1d944a) [0x7f29b01bd44a]
May 01 13:59:21 Zacate smbd[13949]: #10 /usr/lib/samba/libsmbd-base-samba4.so(smb_vfs_call_fsctl+0x47) [0x7f29b00f6f27]
May 01 13:59:21 Zacate smbd[13949]: #11 /usr/lib/samba/libsmbd-base-samba4.so(smb2_ioctl_filesys+0x81) [0x7f29b0126eb1]
May 01 13:59:21 Zacate smbd[13949]: #12 /usr/lib/samba/libsmbd-base-samba4.so(smbd_smb2_request_process_ioctl+0x560) [0x7f29b0126ae0]
May 01 13:59:21 Zacate smbd[13949]: #13 /usr/lib/samba/libsmbd-base-samba4.so(smbd_smb2_request_dispatch+0x762) [0x7f29b0119da2]
May 01 13:59:21 Zacate smbd[13949]: #14 /usr/lib/samba/libsmbd-base-samba4.so(smbd_smb2_request_dispatch_immediate+0x4f) [0x7f29b011a45f]
May 01 13:59:21 Zacate smbd[13949]: #15 /usr/lib/libtevent.so.0(tevent_common_loop_immediate+0xd4) [0x7f29ad12aad4]
May 01 13:59:21 Zacate smbd[13949]: #16 /usr/lib/libsmbconf.so.0(run_events_poll+0x3c) [0x7f29ae6f4b9c]
May 01 13:59:21 Zacate smbd[13949]: #17 /usr/lib/libsmbconf.so.0(+0x36e87) [0x7f29ae6f4e87]
May 01 13:59:21 Zacate smbd[13949]: #18 /usr/lib/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f29ad12a29d]
May 01 13:59:21 Zacate smbd[13949]: #19 /usr/lib/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f29ad12a43b]
May 01 13:59:21 Zacate smbd[13949]: #20 /usr/lib/samba/libsmbd-base-samba4.so(smbd_process+0x728) [0x7f29b0109298]
May 01 13:59:21 Zacate smbd[13949]: #21 /usr/bin/smbd(+0xb020) [0x7f29b0b90020]
May 01 13:59:21 Zacate smbd[13949]: #22 /usr/lib/libsmbconf.so.0(run_events_poll+0x167) [0x7f29ae6f4cc7]
May 01 13:59:21 Zacate smbd[13949]: #23 /usr/lib/libsmbconf.so.0(+0x36f27) [0x7f29ae6f4f27]
May 01 13:59:21 Zacate smbd[13949]: #24 /usr/lib/libtevent.so.0(_tevent_loop_once+0x8d) [0x7f29ad12a29d]
May 01 13:59:21 Zacate smbd[13949]: #25 /usr/lib/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7f29ad12a43b]
May 01 13:59:21 Zacate smbd[13949]: #26 /usr/bin/smbd(main+0x164f) [0x7f29b0b8c54f]
May 01 13:59:21 Zacate smbd[13949]: #27 /usr/lib/libc.so.6(__libc_start_main+0xf0) [0x7f29acda3800]
May 01 13:59:21 Zacate smbd[13949]: #28 /usr/bin/smbd(_start+0x29) [0x7f29b0b8c8e9]
May 01 13:59:21 Zacate smbd[13949]: [2015/05/01 13:59:21.555124,  0] ../source3/lib/dumpcore.c:318(dump_core)
May 01 13:59:21 Zacate smbd[13949]: dumping core in /var/log/samba/cores/smbd

And /var/log/samba/cores/smbd is empty. Using the official packages of archlinux.

[1] https://wiki.archlinux.org/index.php/Samba#Creating_usershare_path
Comment 14 Stefan Metzmacher 2015-05-01 13:41:55 UTC
(In reply to chrno-sphered from comment #13)

Your backtrace is different, please create a new bug report for it, thanks!
Comment 15 Stefan Metzmacher 2015-05-02 08:33:17 UTC
I think I found the problem.

A session reconnect (session setup with previous session id)
deletes a session without canceling pending requests.

I'm working on a fix...
Comment 16 Jeremy Allison 2015-05-02 19:56:04 UTC
Oh very cool Metze. Do you have a wireshark capture or reproducer, or did you find this by the power of logic (examining code :-) ?
Comment 17 Nick Semenkovich 2015-05-02 19:57:20 UTC
(In reply to Jeremy Allison from comment #16)

(FWIW, I attached a capture from when I saw this.)
Comment 18 Jeremy Allison 2015-05-02 19:58:44 UTC
I know but I didn't spot the problem when I looked at that :-). Maybe I should look again...
Comment 19 Stefan Metzmacher 2015-05-03 08:06:49 UTC
(In reply to Jeremy Allison from comment #18)

I didn't notice it in the capture either...

I first assumed a failing reauth would trigger it
and while debugging I found the comment "* TODO: cancel all outstanding requests on the session" in smbXsrv_session_close_loop().

Looking at the capture now, there's indeed a previous_session_id in frame
26.
Comment 20 Stefan Metzmacher 2015-05-03 08:10:16 UTC
Created attachment 11012 [details]
Work in progress patches (on v4-2-test)

This contains a reproducer (the smb2.notify.session-reconnect test)
and fixes, I need to port this to master and check the minimum set
for the backports.
Comment 21 Stefan Metzmacher 2015-05-03 08:11:09 UTC
(In reply to Stefan (metze) Metzmacher from comment #19)

BTW: this shows the fixes for https://bugzilla.samba.org/show_bug.cgi?id=10344
where incomplete...
Comment 22 Jeremy Allison 2015-05-04 15:17:00 UTC
Can't say I'm surprised (the fixes for https://bugzilla.samba.org/show_bug.cgi?id=10344 were incomplete... :-). That stuff was hellishly complex to try and get right :-).
Comment 23 Stefan Metzmacher 2015-05-07 09:05:58 UTC
Created attachment 11031 [details]
Patches for v4-2-test
Comment 24 Michael Adam 2015-05-07 13:55:58 UTC
Karo, please apply to 4.2.next
Comment 25 Stefan Metzmacher 2015-05-08 17:25:36 UTC
Created attachment 11037 [details]
Patches for v4-1-test
Comment 26 Jeremy Allison 2015-05-08 20:21:21 UTC
Comment on attachment 11037 [details]
Patches for v4-1-test

LGTM.
Comment 27 Karolin Seeger 2015-05-20 11:27:00 UTC
Pushed to autobuild-v4-[1|2]-test.
Comment 28 Karolin Seeger 2015-05-21 07:43:26 UTC
(In reply to Karolin Seeger from comment #27)
Pushed to both branches.
Closing out bug report.

Thanks!