Is it possible that you compile with -g, so that we get debug symbols? Also, your smb.conf, a network trace and a debug level 10 log of that failure would be helpful.

Thanks

I'll try to bisect this more tonight, and provide debug symbols. I'm using samba as a Windows 8 client PDC. Logins seem to work fine, but syncing offline files causes an immediate crash and makes samba unusable.

I've narrowed this down to a tighter range of commits (ran out of time with the last range of ~10 commits).


WORKING:

commit 237ec2fbac4c69f642eab481813350fb9568ebaf
Author: Volker Lendecke <vl@samba.org>
Date:   Tue Apr 9 12:15:02 2013 +0200

    libsmbclient: Use async cli_full_connection in python connection setup


BROKEN:

commit 41333f92d0c78e44a524f2248f5ae641ad59abf3
Author: Rusty Russell <rusty@rustcorp.com.au>
Date:   Thu Apr 11 17:12:16 2013 +0930

    ntdb: remove --disable-ntdb.

Additional bisecting puts in this range of ~4 commits:

WORKING:

commit 237ec2fbac4c69f642eab481813350fb9568ebaf
Author: Volker Lendecke <vl@samba.org>
Date:   Tue Apr 9 12:15:02 2013 +0200

    libsmbclient: Use async cli_full_connection in python connection setup


BROKEN:

commit 7a4dd845958f1411daa8031ca242987001ab2f26
Author: Jeremy Allison <jra@samba.org>
Date:   Wed Apr 10 16:30:10 2013 -0700

    Remove dependency on detection of HAVE_DIRFD for use of fdopendir().



/usr/local/samba/sbin/smbd: INTERNAL ERROR: Signal 11 in pid 22100 (4.1.0pre1-GIT-7a4dd84)
/usr/local/samba/sbin/smbd: Please read the Trouble-Shooting section of the Samba HOWTO
/usr/local/samba/sbin/smbd: ===============================================================
/usr/local/samba/sbin/smbd: PANIC (pid 22100): internal error
/usr/local/samba/sbin/smbd: BACKTRACE: 45 stack frames:
/usr/local/samba/sbin/smbd:  #0 /usr/local/samba/lib/libsmbconf.so.0(log_stack_trace+0x1f) [0x7f9e79cc8c6e]
/usr/local/samba/sbin/smbd:  #1 /usr/local/samba/lib/libsmbconf.so.0(smb_panic_s3+0x6c) [0x7f9e79cc8ac1]
/usr/local/samba/sbin/smbd:  #2 /usr/local/samba/lib/libsamba-util.so.0(smb_panic+0x28) [0x7f9e7b748c19]
/usr/local/samba/sbin/smbd:  #3 /usr/local/samba/lib/libsamba-util.so.0(+0x1c8fb) [0x7f9e7b7488fb]
/usr/local/samba/sbin/smbd:  #4 /usr/local/samba/lib/libsamba-util.so.0(+0x1c910) [0x7f9e7b748910]
/usr/local/samba/sbin/smbd:  #5 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfbd0) [0x7f9e7b977bd0]
/usr/local/samba/sbin/smbd:  #6 /usr/local/samba/lib/private/libsmbd_base.so(SeekDir+0x14) [0x7f9e7b248f61]
/usr/local/samba/sbin/smbd:  #7 /usr/local/samba/lib/private/libsmbd_base.so(dptr_SeekDir+0x27) [0x7f9e7b246abc]
/usr/local/samba/sbin/smbd:  #8 /usr/local/samba/lib/private/libsmbd_base.so(+0x1a2b82) [0x7f9e7b324b82]
/usr/local/samba/sbin/smbd:  #9 /usr/local/samba/lib/private/libsmbd_base.so(smbd_smb2_request_process_find+0x5e5) [0x7f9e7b32422f]
/usr/local/samba/sbin/smbd:  #10 /usr/local/samba/lib/private/libsmbd_base.so(smbd_smb2_request_dispatch+0x103d) [0x7f9e7b30a033]
/usr/local/samba/sbin/smbd:  #11 /usr/local/samba/lib/private/libsmbd_base.so(+0x18ac30) [0x7f9e7b30cc30]
/usr/local/samba/sbin/smbd:  #12 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a) [0x7f9e7af76a13]
/usr/local/samba/sbin/smbd:  #13 /usr/local/samba/lib/private/libtevent.so.0(+0x5a45) [0x7f9e7af76a45]
/usr/local/samba/sbin/smbd:  #14 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_done+0x25) [0x7f9e7af76a6c]
/usr/local/samba/sbin/smbd:  #15 /usr/local/samba/lib/private/libsmbd_base.so(+0x18a671) [0x7f9e7b30c671]
/usr/local/samba/sbin/smbd:  #16 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a) [0x7f9e7af76a13]
/usr/local/samba/sbin/smbd:  #17 /usr/local/samba/lib/private/libtevent.so.0(+0x5a45) [0x7f9e7af76a45]
/usr/local/samba/sbin/smbd:  #18 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_done+0x25) [0x7f9e7af76a6c]
/usr/local/samba/sbin/smbd:  #19 /usr/local/samba/lib/private/libsamba-sockets.so(+0xc39f) [0x7f9e79a9739f]
/usr/local/samba/sbin/smbd:  #20 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a) [0x7f9e7af76a13]
/usr/local/samba/sbin/smbd:  #21 /usr/local/samba/lib/private/libtevent.so.0(+0x5a45) [0x7f9e7af76a45]
/usr/local/samba/sbin/smbd:  #22 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_done+0x25) [0x7f9e7af76a6c]
/usr/local/samba/sbin/smbd:  #23 /usr/local/samba/lib/private/libsamba-sockets.so(+0xbe89) [0x7f9e79a96e89]
/usr/local/samba/sbin/smbd:  #24 /usr/local/samba/lib/private/libsamba-sockets.so(+0xc0b4) [0x7f9e79a970b4]
/usr/local/samba/sbin/smbd:  #25 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a) [0x7f9e7af76a13]
/usr/local/samba/sbin/smbd:  #26 /usr/local/samba/lib/private/libtevent.so.0(+0x5a45) [0x7f9e7af76a45]
/usr/local/samba/sbin/smbd:  #27 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_done+0x25) [0x7f9e7af76a6c]
/usr/local/samba/sbin/smbd:  #28 /usr/local/samba/lib/private/libsamba-sockets.so(+0xb3e7) [0x7f9e79a963e7]
/usr/local/samba/sbin/smbd:  #29 /usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a) [0x7f9e7af76a13]
/usr/local/samba/sbin/smbd:  #30 /usr/local/samba/lib/private/libtevent.so.0(+0x5a45) [0x7f9e7af76a45]
/usr/local/samba/sbin/smbd:  #31 /usr/local/samba/lib/private/libtevent.so.0(+0x5b63) [0x7f9e7af76b63]
/usr/local/samba/sbin/smbd:  #32 /usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_immediate+0x1f5) [0x7f9e7af75f78]
/usr/local/samba/sbin/smbd:  #33 /usr/local/samba/lib/libsmbconf.so.0(run_events_poll+0x56) [0x7f9e79ce9d77]
/usr/local/samba/sbin/smbd:  #34 /usr/local/samba/lib/libsmbconf.so.0(+0x4543b) [0x7f9e79cea43b]
/usr/local/samba/sbin/smbd:  #35 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_once+0xf4) [0x7f9e7af750b2]
/usr/local/samba/sbin/smbd:  #36 /usr/local/samba/lib/private/libsmbd_base.so(smbd_process+0x12ef) [0x7f9e7b2ea56f]
/usr/local/samba/sbin/smbd:  #37 /usr/local/samba/sbin/smbd() [0x409e66]
/usr/local/samba/sbin/smbd:  #38 /usr/local/samba/lib/libsmbconf.so.0(run_events_poll+0x55f) [0x7f9e79cea280]
/usr/local/samba/sbin/smbd:  #39 /usr/local/samba/lib/libsmbconf.so.0(+0x4554c) [0x7f9e79cea54c]
/usr/local/samba/sbin/smbd:  #40 /usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_once+0xf4) [0x7f9e7af750b2]
/usr/local/samba/sbin/smbd:  #41 /usr/local/samba/sbin/smbd() [0x40aad0]
/usr/local/samba/sbin/smbd:  #42 /usr/local/samba/sbin/smbd(main+0x16dd) [0x40c2fe]
/usr/local/samba/sbin/smbd:  #43 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f9e78537ea5]
/usr/local/samba/sbin/smbd:  #44 /usr/local/samba/sbin/smbd() [0x405589]
/usr/local/samba/sbin/smbd: dumping core in /usr/local/samba/var/cores/smbd

Created attachment 8809 [details]
crash core, compiled with ./configure --enable-debug ; make -j8 ; make install

crash core, compiled with ./configure --enable-debug ; make -j8 ; make install

Just to clarify, the coredump in comment #5 was compiled off of:

commit 7a4dd845958f1411daa8031ca242987001ab2f26
Author: Jeremy Allison <jra@samba.org>
Date:   Wed Apr 10 16:30:10 2013 -0700

    Remove dependency on detection of HAVE_DIRFD for use of fdopendir().


Compiled as:
./configure --enable-debug ; make -j8 ; make install

============================

237ec2fb works fine, so the problem has arisen in one of these five commits:

http://git.samba.org/?p=samba.git;a=commitdiff;h=7a4dd845958f1411daa8031ca242987001ab2f26
http://git.samba.org/?p=samba.git;a=commitdiff;h=7a4dd845958f1411daa8031ca242987001ab2f26
http://git.samba.org/?p=samba.git;a=commitdiff;h=0fe894fb89f4867e266bb04670a58101311e0234
http://git.samba.org/?p=samba.git;a=commitdiff;h=ea14c9443178da9ae6ccbe71e573156396f6f699
http://git.samba.org/?p=samba.git;a=commitdiff;h=e89ec641fc98ffd7f7193deb3728b0a284a093eb

(In reply to comment #5)
> Created attachment 8809 [details]
> crash core, compiled with ./configure --enable-debug ; make -j8 ; make install
> 
> crash core, compiled with ./configure --enable-debug ; make -j8 ; make install

The core file is useful on systems with your exact libraries, kernel and so on. On other systems you can't really dissect it. Can you do

gdb /usr/sbin/smbd <corefile>

(or wherever your smbd is), and do a "bt full" at the prompt and paste the output here? Also, the debug level 10 log and network trace might be helpful as well.

Thanks,

Volker

Yeah, we really need a backtrace with symbols in order to debug this further. Alternatively, can you explain *exactly* how you are reproducing this ?

Jeremy.

Just checking, you did do a complete "make clean" before testing this ? The change in question does change a structure size (although not one that should make an external difference to any VFS module - it's a completely internal interface).

Just to confirm, ensure you do a 'git clean -d -f -x' at the top of the tree and then reconfigure and make.

Otherwise the crash in SeekDir doesn't make sense, it isn't accessing anything that changed there.

Thanks,

Jeremy.

> Just to confirm, ensure you do a 'git clean -d -f -x' at the top of the tree
> and then reconfigure and make.


Yep, I do:

git checkout XXXXXXX
git clean -d -f -x
./configure --enable-debug
make -j8
make install


I'll double-check tonight that I did this every time I re-compiled, bisect the commits further, and provide a level 10 log and a gdb backtrace.

For a network log, is there a specific tcpdump command that will isolate samba traffic (or will samba generate network logs somehow)?

A network log doesn't help that much here. What will help are:

(a) a good backtrace with symbols (bt full).

What I do here is add "panic action = /bin/sleep 99999999"
to the [global] section of my smb.conf, then when I get a crash I attach
to the parent process of the sleep (which is the crashed smbd) and do a "bt full". That's the data we need.

(b). A *good* set of steps to reproduce. That means a full smb.conf, and *exact* steps on the Win8 clients that reproduces the problem.

Thanks,

Jeremy.

ping ! Can you get us the info we need asap ?

This problem is blocking another bug fix that addresses an issue that prevents many VFS module from working correctly in 4.0.x.

Jeremy.

(In reply to comment #12)
> ping ! Can you get us the info we need asap ?
> 
> This problem is blocking another bug fix that addresses an issue that prevents
> many VFS module from working correctly in 4.0.x.
> 
> Jeremy.

Yep -- sorry for the delay! Will be working on this tonight, around 8 p.m. EST.

I won't be able to get to this until tomorrow (I'll be out by 5pm PDT).

Jeremy.

Looks like this issue is caused by:

commit 0fe894fb89f4867e266bb04670a58101311e0234
Author: Jeremy Allison <jra@samba.org>
Date:   Wed Apr 10 16:29:03 2013 -0700

    Remove the "Ugly hack" that was the second use of dirfd().

    The destructor does all the resource deallocation needed.

    Signed-off-by: Jeremy Allison <jra@samba.org>
    Reviewed-by: Andreas Schneider <asn@samba.org>



The previous commit (ea14c9443178da9ae6ccbe71e573156396f6f699) works fine.

Created attachment 8818 [details]
smb.conf

Created attachment 8819 [details]
gdb: attach PID; bt full

Created attachment 8820 [details]
level 10 debug log

The above crashes/logs were from 0fe894fb89f4, built with:

git checkout 0fe894fb89f4
git clean -f -d -x
./configure --enable-debug
make -j8
make install



I have double-checked on multiple machines that the previous commit (ea14c9443178) doesn't have this panic.

Here's the "reproduction" per-se.

Samba4 is operating as a DC for 10 fully patched Windows 8 Pro clients. All the clients are /identical/ (same hardware, bought at same time from Dell).

These clients have mapped home directories, with offline files / windows file sync enabled. 


After a user logs in, the sync center always shows something like this:
http://imgur.com/a/rYkUQ#0

(Normally) If you click on "Sync All" (or wait for the next sync interval) the sync completes, and shows:
http://imgur.com/a/rYkUQ#1

However, starting with 0fe894 commit, the "Sync All" button (or any scheduled sync) causes the panic, and windows shows:
http://imgur.com/a/rYkUQ#2

The error details are vague:
http://imgur.com/a/rYkUQ#3


Eventually, Windows will show the home share as "Disconnected"
http://imgur.com/a/rYkUQ#4

If you wait a few minutes, eventually Windows will try to sync again (or give you the option to "Sync All") and the panic happens again.

From what I can tell, syncs never succeed.



Please let me know if you need anything else.

(In reply to comment #18)
> Created attachment 8820 [details]
> level 10 debug log

Unfortunately this is not a debug level 10 log of smbd, it is only a debug level 10 log of samba. This gives no clue what is going on in smbd, sorry

(In reply to comment #21)
> (In reply to comment #18)
> > Created attachment 8820 [details] [details]
> > level 10 debug log
> 
> Unfortunately this is not a debug level 10 log of smbd, it is only a debug
> level 10 log of samba. This gives no clue what is going on in smbd, sorry

I don't have a good grasp of the whole Samba architecture --  how can I specifically generate that log file?

If I do:

killall samba
smbd -i --debuglevel=10

Will clients be able to log on, etc (do I need to run any other processes)?

Should I set any additional flags (--option=server role check:inhibit=yes) ?

Ok, the backtrace shows a crash here:

        if (in_flags & SMB2_CONTINUE_FLAG_RESTART) {
                dptr_SeekDir(fsp->dptr, 0);
        }

the fsp is valid, but the underlying dptr->dir_hnd == NULl.

That shouldn't be able to happen :-).

I'll look at what logic case I must have missed.

Cheers,

Jeremy.

Ok found the logic error.

The smb2 find request has in_flags = 0x11 ==

SMB2_CONTINUE_FLAG_REOPEN|SMB2_CONTINUE_FLAG_RESTART

I missed nulling out a pointer in the destructor. Patch to follow.

Jeremy.

Created attachment 8821 [details]
git-am fix for master.

Please test this fix, it should address the issue.

Once you've confirmed I'll get it into master, and then wrapped up with the previous fix for 4.0.next.

Cheers,

Jeremy.

Thanks so much for your help!

I'll test & verify this patch late tonight (after 8 EST)

No problem, thanks so much for providing the backtrace that showed where I'd screwed up the previous patch !

Doing the bisect work to discover what patch this was also deserves a medal for going above and beyond the call of duty :-).

Cheers,

Jeremy.

Patch works -- thanks again! (And thanks for the work you've all put into Samba over the years!)

Bug fix is attached to the patchset for bug:

https://bugzilla.samba.org/show_bug.cgi?id=9778

so we'll close both there,

Jeremy.

*** Bug 9849 has been marked as a duplicate of this bug. ***