Since commit 10d15c9e5dfe4e8595d0b322c96f474fc7078f46, CTDB uses Samba's file logging. When the log file is (re)opened, stderr is dup()ed to the log file. This overrides CTDB's capture of stderr, so that stderr in subprocesses will automatically be captured and logged. This means subprocesses that usefprintf(stderr, ...) now have the output written into the log file without a debug header. Moreover, the CTDB code that logs any captured output calls dbgtext() rather than DEBUG() so, even if the initial problem can be worked around, these captured entries are logged without a header. This means it is not possible to sort logs to get a cluster-wide view. Before the switch to Samba's file logging, CTDB uses a callback that always added a debug header. A workaround is needed in Samba's file logging (in debug.c) and the relevant CTDB logging code needs to be changes to use DEBUG().
This bug was referenced in samba master: a8091bd0c565a3f14542731e642319dbb68b4786 1596a3e84babb8fdd86af0c4b98906b309be7907 88f35cf86285b7a818282d5f465711de66dfad59 e752f841e682cc571006c09249b03d82aea5f8cd
Created attachment 17367 [details] Patch for v4-16-test Cherry picks cleanly into v4-16-test. Smoke tested with: $ VALGRIND="valgrind -q" ./tests/run_tests.sh tests/INTEGRATION/simple/cluster.016.reclock_move_lock_dir.sh [test passes] $ ./tests/local_daemons.sh tests/var/INTEGRATION/simple/ print-log all | less [messages from ctdb_mutex_fcntl_helper are correctly logged]
Pushed to autobuild-v4-16-test.
This bug was referenced in samba v4-16-test: a393eab06a74e3b8f32eee33d11bef510e137bf6 79706765035fc1815795d61bcfb3e0d937b08f5d c4e176e46cd4a8392cc1de08ad90d62b3c114079 959d37e72cc539cf76fb99dbe8fc99ae32e59b63
Closing out bug report. Thanks!
This bug was referenced in samba v4-16-stable (Release samba-4.16.3): a393eab06a74e3b8f32eee33d11bef510e137bf6 79706765035fc1815795d61bcfb3e0d937b08f5d c4e176e46cd4a8392cc1de08ad90d62b3c114079 959d37e72cc539cf76fb99dbe8fc99ae32e59b63