Bug 13465 - testparm crashes with PANIC: messaging not initialized on SLES 12 SP3
testparm crashes with PANIC: messaging not initialized on SLES 12 SP3
Status: RESOLVED FIXED
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Clustering
4.8.2
x86 Linux
: P5 major
: ---
Assigned To: Karolin Seeger
Samba QA Contact
:
: 13515 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-06-06 13:24 UTC by Christian Meyer
Modified: 2018-10-11 10:11 UTC (History)
8 users (show)

See Also:


Attachments
Backtrace testparm (4.35 KB, text/plain)
2018-06-06 13:24 UTC, Christian Meyer
no flags Details
testparm_dbg_output (13.48 KB, text/plain)
2018-06-29 05:55 UTC, fettfoen
no flags Details
Possible patch for master (1.87 KB, patch)
2018-07-05 12:43 UTC, Ralph Böhme
no flags Details
Patch for 4.9 backported from master (65.90 KB, patch)
2018-10-04 10:10 UTC, Ralph Böhme
cs: review+
Details
Patch for 4.8 backported from master (65.46 KB, patch)
2018-10-04 10:10 UTC, Ralph Böhme
cs: review+
Details
Patch for 4.7 backported from master (64.74 KB, patch)
2018-10-04 10:11 UTC, Ralph Böhme
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Christian Meyer 2018-06-06 13:24:16 UTC
Created attachment 14224 [details]
Backtrace testparm

Dear Samba Team,

we’re facing a problem with CTDB/Samba using the registry support. We’re using SUSE Linux Enterprise Server 12 SP3,
Samba/CTDB Version is: 4.8.2-git.30.690aa93c1892.1-SUSE-SLE_12-x86_64

CTDB is up and running, but if we run testparm the program crashes with the message „PANIC: messaging not initialized“
Please find attached a backtrace and below our smb.conf and ctdbd.conf 

Please don't hesitate to contact me if you need more information.

Kind regards

Christian

##### smb.conf #######
# See smb.conf.example for a more detailed config file or
# read the smb.conf manpage.
# Run 'testparm' to verify the config is correct after
# you modified it.

[global]
log level = 10

# Only these both parameters are neccessary, if runnnig samba with CTDB
clustering = yes
dbwrap_tdb_mutexes:* = true
ctdb:registry.tdb = yes
#config backend = registry
include = registry
#######################

### ctdb.conf #########
CTDB_RECOVERY_LOCK=/run/gluster/shared_storage/ctdb.lock
CTDB_NODES=/etc/ctdb/nodes
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses

CTDB_MANAGES_SAMBA=yes

# Only when using samba-vfs-glusterfs!
# CTDB_SAMBA_SKIP_SHARE_CHECK=yes

#######################
Comment 1 Amitay Isaacs 2018-06-29 01:00:08 UTC
Hi Ralph, I wrongly assigned to the tools component.  Now it's all yours. :-)
Comment 2 fettfoen 2018-06-29 05:55:47 UTC
Created attachment 14264 [details]
testparm_dbg_output
Comment 3 fettfoen 2018-06-29 05:55:53 UTC
Hello,

i can confirm this issue. I upgraded my CTDB Cluster from SAMBA 4.7.7/8 to SAMBA 4.8.3.

I'm also using registry support. CTDB is up and running, but throws some panic messages by using the "testparm" command.

SAMBA is running on Ubuntu 18.04.

smb.conf

[global]
 clustering = yes
 include =registry

ctdbd.conf

CTDB_RECOVERY_LOCK=/cephfuse/.lockdir/ctdb.lock
CTDB_SAMBA_SKIP_SHARE_CHECK=no
CTDB_NODES=/usr/local/samba/etc/ctdb/nodes2
CTDB_PUBLIC_ADDRESSES=/usr/local/samba/etc/ctdb/public_addresses

That's it! Please have look at my attachment. I added some hopefull helpfull debug information!
Comment 4 Ralph Böhme 2018-07-05 12:43:11 UTC
Created attachment 14281 [details]
Possible patch for master

Can someone test this patch? Fixes it for me on my test devel cluster.

Patch should apply to 4.8 as well.
Comment 5 fettfoen 2018-07-05 13:57:19 UTC
Hello,

i tested your patch in samba 4.8.3. 

It works!

Thanks for fix!

Regards
Micha
Comment 6 Christof Schmitt 2018-07-09 20:12:37 UTC
Thanks for the patch. I am also looking into
running Samba 4.8 in a cluster. While this
bugzilla only mentions testparm, the same
problem also exists with at least smbcacls,
smbcquotas, smbclient, rpcclient and sharesec.
We probably should fix the issue for all tools.
Comment 7 Ralph Böhme 2018-07-09 20:24:42 UTC
See the linked bug under "See also" :).

Fixing this is tricky, I have a WIP branch running in CI but I keep getting it wrong:
<https://gitlab.com/samba-team/devel/samba/pipelines/25426325>
Comment 8 Christof Schmitt 2018-07-09 21:54:24 UTC
I tried patching a few more tools based on your patches:
https://gitlab.com/samba-team/devel/samba/commits/christofschmitt-4-8-cluster
https://gitlab.com/samba-team/devel/samba/pipelines/25431943
Comment 9 Ralph Böhme 2018-07-10 10:32:28 UTC
(In reply to Christof Schmitt from comment #8)
Thanks! I modified your patches so messaging is initialized after popt processing is done. This is needed to ensure get_dyn_config() picks up smb.conf given via commandline argument.

Also the pdbedit backbox test needed a fix to pass the testenv smb.conf to pdbedit in the test.

Currently running: https://gitlab.com/samba-team/devel/samba/pipelines/25462386
Comment 10 Ralph Böhme 2018-07-10 10:34:33 UTC
*** Bug 13515 has been marked as a duplicate of this bug. ***
Comment 11 Ralph Böhme 2018-07-10 10:38:14 UTC
List of affected tools fixed by the patch for this bug:

pdbedit
rpcclient
sharesec
smbcacls
smbclient
smbcquotas
smbstatus
smbtree
smbpasswd
testparm
Comment 13 Martin Schwenke 2018-10-04 00:02:12 UTC
Just wondering if this is ready to go upstream and be backported?
Comment 14 Ralph Böhme 2018-10-04 09:30:47 UTC
(In reply to Martin Schwenke from comment #13)
As discussed on the ML, there's still an unresolved issue with smbtorture.

https://lists.samba.org/archive/samba-technical/2018-September/130177.html
Comment 15 Ralph Böhme 2018-10-04 09:42:54 UTC
(In reply to Ralph Böhme from comment #14)
Created bug 13640 to track the issue with smbtorture.

I propose we proceed with backporting with what we have and closing this bug. The new bug 13640 can then be used to track the remaining issue with smbtorture.
Comment 16 Ralph Böhme 2018-10-04 10:10:18 UTC
Created attachment 14511 [details]
Patch for 4.9 backported from master
Comment 17 Ralph Böhme 2018-10-04 10:10:51 UTC
Created attachment 14512 [details]
Patch for 4.8 backported from master
Comment 18 Ralph Böhme 2018-10-04 10:11:23 UTC
Created attachment 14513 [details]
Patch for 4.7 backported from master
Comment 19 Ralph Böhme 2018-10-04 10:27:22 UTC
Comment on attachment 14513 [details]
Patch for 4.7 backported from master

The problem is 4.8 upwards...
Comment 20 Ralph Böhme 2018-10-05 05:32:13 UTC
Reassigning to Karolin for inclusion in 4.9 and 4.8. 4.7 is not affected by this.
Comment 21 Karolin Seeger 2018-10-09 09:24:11 UTC
(In reply to Ralph Böhme from comment #20)
Pushed to autobuild-v4-{9,8-}test.
Comment 22 Karolin Seeger 2018-10-11 10:11:51 UTC
(In reply to Karolin Seeger from comment #21)
Pushed to both branches.
Closing out bug report.

Thanks!