It seems valuable to have a possibility to set the maximum size of the ctdb log file (e.g. inherit from the Samba's "max log size" directive or set via environment variable in /etc/sysconfig/ctdb). In case of debuggigng a problem the logfiles can grow quickly to undesirable sizes yet it is sometimes necessary to wait until the problem debugged reproduces.
Thanks for filing this bug report! It is not only the log size. but we first need infrastructure for reopening the logs at all. An alternative to "max log size" which makes sense for samba with all its possibbly very many log files would be to simply react with some "reopen_logs" to a HUP signal and rely on an external logrotate mechanism to trigger the actual rotation. I just talked to ronnie today about this problem. We need some solution like this! Meanwhile, you can make ctdb log to syslog by running ctdbd with "--syslog". This has automatic log rotation.. I just committed a patch to make this configurable via the sysconfig/ctdb file with the new variable CTDB_SYSLOG=[yes/no] which defaults to "no". This will be in 1.0.91 if I am not mistaken. Cheers - Michael
I'm going to steal this bug to track the lack of log rotation in CTDB. It can be worked around in all currently supported versions by using CTDB_LOGGING=syslog (or a variant) to log to syslog, which supports log rotation. CTDB's logging currently uses the Samba debug module, but uses the callback mechanism instead of any of the built-in ones. Notes about the different types of logging supported by CTDB: * file Direct to file with no re-open mechanism, which is required for rotation. * syslog Has obvious implementation using syslog(3). Additional support for non-blocking/lossy logging via either non-blocking logging to /dev/log or UDP. So, instead of supporting log rotation directly in CTDB, the approach here would be to switch to using Samba's debug.[ch] file logging directly. Then we get the rotation for free and we have even less duplicated code. Blocking this is CTDB's use of debug_extra, where the string value of that variable is prefixed to every log message. I think that we should be able to replace this by using logging classes... but this is quite a bit of work so would need a chunk of time. If we fix the debug_extra issue then we could also port any desired non-blocking support into debug.[ch] and drop the CTDB logging entirely. However, this would be a secondary effort.
Moving this to CTDB component of Samba.
Just an update because I received an email asking about the status of this... The good news here is that debug_extra is gone. I would still like to do 2 things: 1. Fix the problem where Samba's file logging is not atomic The header and content are logged with separate writes. This means that 2 processes can each log a header and then leach log the content. This means that you can't be sure which header matches which content. This can be fixed using writev(). I started doing this a long time ago but never got back it. 2. Add an option to log in syslog format This is a nice, well known format that we've been using for CTDB's file logging for a long time. There is one minor dependency on this: tests/local_daemons.sh print-log command. Then we should be good to go...
Log reopening to support rotation has been enabled since Samba 4.16. Does anyone object to closing this? It should also be relatively easy to add "max log size" to ctdb.conf...