Bug 14645 - rpcclient NetFileEnum and net rpc file both cause lock order violation: brlock.tdb, share_entries.tdb
Summary: rpcclient NetFileEnum and net rpc file both cause lock order violation: brloc...
Status: RESOLVED DUPLICATE of bug 14355
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Clustering (show other bugs)
Version: 4.12.3
Hardware: All Linux
: P5 normal (vote)
Target Milestone: ---
Assignee: Volker Lendecke
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-02-18 16:57 UTC by isaac
Modified: 2021-02-25 17:08 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description isaac 2021-02-18 16:57:37 UTC
running samba 4.12.3 as a domain member in clustered mode with ctdb 4.12.3

NetFileEnum rpc is having problems. Other things seem to work, including
NetShareEnum.

calling `rpclient -c NetFileEnum` produces this result

> cli_api_pipe failed: NT_STATUS_CONNECTION_DISCONNECTED
> result was WERR_NETNAME_DELETED

`smbstatus -L` returns a list of locked files with no errors.
`net rpc file` fails with no error printed, but the same crash as below.

looking at the logs and I think the smb process is crashing:

[2021/02/16 19:37:34.389710,  2] ../../source3/auth/auth.c:329(
auth_check_ntlm_password)
  check_ntlm_password:  authentication for user [isaac.stone] ->
[isaac.stone] -> [SOMDEV\isaac.stone] succeeded
[2021/02/16 19:37:34.402327,  0] ../../lib/dbwrap/dbwrap.c:156(
dbwrap_lock_order_lock)
  dbwrap_lock_order_lock: Lock order violation: Trying brlock.tdb at 2
while share_entries.tdb at 3 is locked
[2021/02/16 19:37:34.402448,  0] ../../lib/dbwrap/dbwrap.c:129(
debug_lock_order)
  lock order:  1:<none> 2:<none> 3:share_entries.tdb 4:<none>
[2021/02/16 19:37:34.402465,  0] ../../source3/lib/util.c:830(smb_panic_s3)
  PANIC (pid 1653800): lock order violation
[2021/02/16 19:37:34.403131,  0] ../../lib/util/fault.c:265(log_stack_trace)
  BACKTRACE: 49 stack frames:
   #0 /lib64/libsamba-util.so.0(log_stack_trace+0x34) [0x7fe835aa6264]
   #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x29) [0x7fe835501639]
   #2 /lib64/libsamba-util.so.0(smb_panic+0x31) [0x7fe835aa6361]
   #3 /usr/lib64/samba/libdbwrap-samba4.so(+0x34c0) [0x7fe8336244c0]
   #4 /usr/lib64/samba/libdbwrap-samba4.so(+0x361b) [0x7fe83362461b]
   #5 /usr/lib64/samba/libsmbd-base-samba4.so(brl_get_locks+0x5f)
[0x7fe8355d2e4f]
   #6 /usr/lib64/samba/libsmbd-base-samba4.so(+0x157614) [0x7fe835695614]
   #7 /usr/lib64/samba/libsmbd-base-samba4.so(+0x98792) [0x7fe8355d6792]
   #8 /usr/lib64/samba/libsmbd-base-samba4.so(+0x9a44b) [0x7fe8355d844b]
   #9 /usr/lib64/samba/libdbwrap-samba4.so(dbwrap_do_locked+0x106)
[0x7fe833624ce6]
   #10 /usr/lib64/samba/libsmbd-base-samba4.so(share_mode_forall_entries+0x5a)
[0x7fe8355dad6a]
   #11 /usr/lib64/samba/libsmbd-base-samba4.so(+0x9ce68) [0x7fe8355dae68]
   #12 /usr/lib64/samba/libsmbd-base-samba4.so(+0x98cc8) [0x7fe8355d6cc8]
   #13 /lib64/libsmbconf.so.0(+0x34078) [0x7fe8354dd078]
   #14 /usr/lib64/samba/libsamba-cluster-support-samba4.so(+0x39f3)
[0x7fe83529e9f3]
   #15 /usr/lib64/samba/libsamba-cluster-support-samba4.so(ctdbd_traverse+0x130)
[0x7fe8352a6b80]
   #16 /usr/lib64/samba/libsamba-cluster-support-samba4.so(+0x4192)
[0x7fe83529f192]
   #17 /usr/lib64/samba/libsamba-cluster-support-samba4.so(+0x435f)
[0x7fe83529f35f]
   #18 /usr/lib64/samba/libdbwrap-samba4.so(dbwrap_traverse_read+0xb)
[0x7fe83362488b]
   #19 /lib64/libsmbconf.so.0(+0x3511d) [0x7fe8354de11d]
   #20 /usr/lib64/samba/libdbwrap-samba4.so(dbwrap_traverse_read+0xb)
[0x7fe83362488b]
   #21 /usr/lib64/samba/libsmbd-base-samba4.so(share_mode_forall+0x46)
[0x7fe8355da9f6]
   #22 /usr/lib64/samba/libsmbd-base-samba4.so(share_entry_forall+0x47)
[0x7fe8355daa67]
   #23 /usr/lib64/samba/libsmbd-base-samba4.so(_srvsvc_NetFileEnum+0xa1)
[0x7fe835697301]
   #24 /usr/lib64/samba/libsmbd-base-samba4.so(+0x163ca6) [0x7fe8356a1ca6]
   #25 /usr/lib64/samba/libsmbd-base-samba4.so(+0x23e86a) [0x7fe83577c86a]
   #26 /usr/lib64/samba/libsmbd-base-samba4.so(process_complete_pdu+0x3ca)
[0x7fe83577da5a]
   #27 /usr/lib64/samba/libsmbd-base-samba4.so(dcerpc_ncacn_packet_process+0xf1)
[0x7fe83564b3b1]
   #28 /lib64/libdcerpc-binding.so.0(+0xd12f) [0x7fe83324712f]
   #29 /usr/lib64/samba/libsamba-sockets-samba4.so(+0x6c7d) [0x7fe835495c7d]
   #30 /usr/lib64/samba/libsamba-sockets-samba4.so(+0x5a0f) [0x7fe835494a0f]
   #31 /lib64/libtevent.so.0(tevent_common_invoke_immediate_handler+0x139)
[0x7fe83488c969]
   #32 /lib64/libtevent.so.0(tevent_common_loop_immediate+0x27)
[0x7fe83488c997]
   #33 /lib64/libtevent.so.0(+0xd8af) [0x7fe8348928af]
   #34 /lib64/libtevent.so.0(+0xbadb) [0x7fe834890adb]
   #35 /lib64/libtevent.so.0(_tevent_loop_once+0x95) [0x7fe83488bc05]
   #36 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fe83488beab]
   #37 /lib64/libtevent.so.0(+0xba6b) [0x7fe834890a6b]
   #38 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x7bf)
[0x7fe8357350ef]
   #39 /usr/sbin/smbd(+0xdd20) [0x55c4b6dabd20]
   #40 /lib64/libtevent.so.0(tevent_common_invoke_fd_handler+0x83)
[0x7fe83488c533]
   #41 /lib64/libtevent.so.0(+0xdacf) [0x7fe834892acf]
   #42 /lib64/libtevent.so.0(+0xbadb) [0x7fe834890adb]
   #43 /lib64/libtevent.so.0(_tevent_loop_once+0x95) [0x7fe83488bc05]
   #44 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fe83488beab]
   #45 /lib64/libtevent.so.0(+0xba6b) [0x7fe834890a6b]
   #46 /usr/sbin/smbd(main+0x1bd2) [0x55c4b6da62c2]
   #47 /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fe8342c57b3]
   #48 /usr/sbin/smbd(_start+0x2e) [0x55c4b6da667e]
[2021/02/16 19:37:34.403657,  0] ../../source3/lib/dumpcore.c:318(dump_core)
  coredump is handled by helper binary specified at
/proc/sys/kernel/core_pattern

Here is our smb.conf:

#======================= Global Settings ==============================
=======
[global]
netbios name = C25-USW2
realm = SOMDEV.LOCAL
workgroup = SOMDEV

security = ads

clustering = yes
fake oplocks = no

log level = 2

idmap config * : backend = tdb
idmap config * : range = 3000-7999
idmap config SOMDEV : backend = rid
idmap config SOMDEV : range = 10000-999999

dedicated keytab file = /etc/krb5.keytab
kerberos method = secrets and keytab

map acl inherit = yes

winbind refresh tickets = yes

# disables printing:
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes

bind interfaces only = yes
interfaces = lo eth0

#============================ Share Definitions
==============================
[SF]
path = "/mnt/usw2/SF"
writeable = yes
browsable = yes
fileid:algorithm = fsname
vfs objects = fileid acl_xattr shadow_copy2
acl_xattr:ignore system acls = yes
shadow:mountpoint = /mnt/usw2
shadow:snapdir = /snapshots/usw2
shadow:snapsharepath = SF
shadow:format = %Y-%m-%dT%H:%M:%SZ
[LA]
path = "/mnt/usw2/LA"
writeable = yes
browsable = yes
fileid:algorithm = fsname
vfs objects = fileid acl_xattr shadow_copy2
acl_xattr:ignore system acls = yes
shadow:mountpoint = /mnt/usw2
shadow:snapdir = /snapshots/usw2
shadow:snapsharepath = LA
shadow:format = %Y-%m-%dT%H:%M:%SZ

-----

I am suspicious this might be a bug:

>  dbwrap_lock_order_lock: Lock order violation: Trying brlock.tdb at 2
>  while share_entries.tdb at 3 is locked
>  lock order:  1:<none> 2:<none> 3:share_entries.tdb 4:<none>
>  PANIC (pid 1654263): lock order violation
Comment 1 Jeremy Allison 2021-02-22 22:22:53 UTC
Can you get me a stack backtrace with line numbers ? That would really help in tracking this one down so I can see the exact code path that leads to the crash.

I think this code has changed in 4.13.x and above.
Comment 2 isaac 2021-02-23 00:48:16 UTC
(In reply to Jeremy Allison from comment #1)

How might I get line numbers into the stack trace?

Trying to get a build of latest samba up and running, but even though we are on redhat for a while we are still waiting on budget to approve support accounts, and samba build has dependencies that require RHEL accounts to access....

yeah anyway probably a couple weeks before I can do a build and test of the latest samba. Will update the ticket as soon as I can
Comment 3 Jeremy Allison 2021-02-23 02:10:57 UTC
Install the symbols packages onto the system. Any more help you can give us will be most appreciated !
Comment 4 Martin Schwenke 2021-02-23 03:09:52 UTC
This looks to be fixed in master:

[root@testn1 ~]# net rpc file -U 'SAMBADOM\administrator%p@ssw0rd'

Enumerating open files on remote server:


FileId  Opened by            Perms  Locks  Path
------  ---------            -----  -----  ---- 
263520269 SAMBADOM\user1       0x03   0      C:\clusterfs\data\motd


[root@testn1 ~]# smbd --version
Version 4.15.0pre1-GIT-5177d07

I'm spinning up a 4.12.12 (pre-release) cluster to ensure I can recreate this in 4.12.
Comment 5 Martin Schwenke 2021-02-23 06:49:04 UTC
I think my test is invalid:

[root@testn1 ~]# net rpc file -U 'SAMBADOM\administrator%p@ssw0rd'

Enumerating open files on remote server:


FileId  Opened by            Perms  Locks  Path
------  ---------            -----  -----  ---- 
-567279614 SAMBADOM\user1       0x03   0      C:\clusterfs\data\motd


[root@testn1 ~]# smbd --version
Version 4.12.12-GIT-237a51d


Perhaps there needs to be a byte-range lock?  Trying that, but mutexes...
Comment 6 Ralph Böhme 2021-02-23 06:51:04 UTC

*** This bug has been marked as a duplicate of bug 14355 ***
Comment 7 Martin Schwenke 2021-02-23 07:11:21 UTC
Thanks Ralph!

So, fixed in 4.12.6
Comment 8 isaac 2021-02-25 17:08:43 UTC
Just to confirm I was finally able to do a build from source and everything worked flawlessly (after I got SELinux to behave)

Thanks for the help, and thanks for making samba so great!