Since commit 10d15c9e5dfe4e8595d0b322c96f474fc7078f46, CTDB uses Samba's file logging. When the log file is (re)opened, stderr is dup()ed to the log file. This overrides CTDB's capture of stderr, so that stderr in subprocesses will automatically be captured and logged. This means subprocesses that usefprintf(stderr, ...) now have the output written into the log file without a debug header. Moreover, the CTDB code that logs any captured output calls dbgtext() rather than DEBUG() so, even if the initial problem can be worked around, these captured entries are logged without a header. This means it is not possible to sort logs to get a cluster-wide view.
Before the switch to Samba's file logging, CTDB uses a callback that always added a debug header.
A workaround is needed in Samba's file logging (in debug.c) and the relevant CTDB logging code needs to be changes to use DEBUG().
This bug was referenced in samba master:
Created attachment 17367 [details]
Patch for v4-16-test
Cherry picks cleanly into v4-16-test.
Smoke tested with:
$ VALGRIND="valgrind -q" ./tests/run_tests.sh tests/INTEGRATION/simple/cluster.016.reclock_move_lock_dir.sh
$ ./tests/local_daemons.sh tests/var/INTEGRATION/simple/ print-log all | less
[messages from ctdb_mutex_fcntl_helper are correctly logged]
Pushed to autobuild-v4-16-test.
This bug was referenced in samba v4-16-test:
Closing out bug report.
This bug was referenced in samba v4-16-stable (Release samba-4.16.3):