While running ctdb/tests/INTEGRATION/simple/cluster.015.reclock_remove_lock.sh this is seen: 2026-10-08T05:57:19.726102+00:00 node.0 ctdbd[2655369]: ctdb_mutex_fcntl_helper: lock lost - lock file "tests/var/INTEGRATION/simple/sh ared/.ctdb/cluster.lock" open failed (ret=2) 2026-10-08T05:57:19.726937+00:00 node.0 ctdb-recoverd[2655437]: Cluster lock helper terminated 2026-10-08T05:57:19.726971+00:00 node.0 ctdb-recoverd[2655437]: Start election 2026-10-08T05:57:19.727053+00:00 node.0 ctdbd[2655369]: Recovery mode set to ACTIVE 2026-10-08T05:57:19.727071+00:00 node.1 ctdbd[2655392]: Recovery mode set to ACTIVE 2026-10-08T05:57:19.727071+00:00 node.2 ctdbd[2655407]: Recovery mode set to ACTIVE 2026-10-08T05:57:19.727161+00:00 node.0 ctdb-recoverd[2655437]: Attempting to take cluster lock (! /m/martins/aMASTER/b1443461/ctdb/ctd b/bin/ctdb_mutex_fcntl_helper tests/var/INTEGRATION/simple/shared/.ctdb/cluster.lock 5) 2026-10-08T05:57:19.727198+00:00 node.1 ctdb-recoverd[2655480]: Attempting to take cluster lock (! /m/martins/aMASTER/b1443461/ctdb/ctd b/bin/ctdb_mutex_fcntl_helper tests/var/INTEGRATION/simple/shared/.ctdb/cluster.lock 5) 2026-10-08T05:57:19.727212+00:00 node.2 ctdb-recoverd[2655503]: Attempting to take cluster lock (! /m/martins/aMASTER/b1443461/ctdb/ctd b/bin/ctdb_mutex_fcntl_helper tests/var/INTEGRATION/simple/shared/.ctdb/cluster.lock 5) 2026-10-08T05:57:19.727885+00:00 node.0 ctdb-recoverd[2655437]: Received leader broadcast, leader=0 2026-10-08T05:57:19.731562+00:00 node.0 ctdb-recoverd[2655437]: Unable to take cluster lock - contention 2026-10-08T05:57:19.731579+00:00 node.2 ctdb-recoverd[2655503]: Unable to take cluster lock - contention 2026-10-08T05:57:19.731587+00:00 node.1 ctdb-recoverd[2655480]: Cluster lock taken successfully 2026-10-08T05:57:19.731602+00:00 node.1 ctdb-recoverd[2655480]: Took cluster lock, leader=1 2026-10-08T05:57:19.731976+00:00 node.0 ctdb-recoverd[2655437]: Node:0 was in recovery mode. Start recovery process 2026-10-08T05:57:19.732006+00:00 node.0 ctdb-recoverd[2655437]: Node:1 was in recovery mode. Start recovery process 2026-10-08T05:57:19.732022+00:00 node.0 ctdb-recoverd[2655437]: Node:2 was in recovery mode. Start recovery process 2026-10-08T05:57:19.732029+00:00 node.0 ctdb-recoverd[2655437]: do_recovery: Starting do_recovery 2026-10-08T05:57:19.732035+00:00 node.1 ctdb-recoverd[2655480]: Node:1 was in recovery mode. Start recovery process 2026-10-08T05:57:19.732050+00:00 node.1 ctdb-recoverd[2655480]: Node:0 was in recovery mode. Start recovery process 2026-10-08T05:57:19.732059+00:00 node.1 ctdb-recoverd[2655480]: Node:2 was in recovery mode. Start recovery process 2026-10-08T05:57:19.732062+00:00 node.0 ctdb-recoverd[2655437]: Cluster lock not held - abort recovery, ban node 2026-10-08T05:57:19.732063+00:00 node.1 ctdb-recoverd[2655480]: do_recovery: Starting do_recovery 2026-10-08T05:57:19.732067+00:00 node.0 ctdb-recoverd[2655437]: Banning node 0 for 300 seconds 2026-10-08T05:57:19.732082+00:00 node.0 ctdbd[2655369]: SET BAN STATE 2026-10-08T05:57:19.732090+00:00 node.0 ctdbd[2655369]: Banning this node for 300 seconds 2026-10-08T05:57:19.732094+00:00 node.0 ctdbd[2655369]: Making node INACTIVE 2026-10-08T05:57:19.732096+00:00 node.1 ctdb-recoverd[2655480]: do_recovery: Recovery initiated due to problem with node 0 This happens because, although the node does not hold the cluster lock, this_node_is_leader() succeeds because rec->leader == rec->pnn is still true, so the node can continue with recovery. This happens even though node 1 has taken the cluster lock and is leader. There are 2 possible solutions: 1. Set rec->leader = CTDB_UNKNOWN_PNN before starting the election when the cluster lock helper terminates... and any similar circumstances. 2. Have this_node_is_leader() also check cluster_lock_held() if cluster_lock_enabled() is true. (1) is simpler provided all the similar cases are found. (2) pollutes the simple concept of the leader node with election implementation details. If this were properly modularised then the generic this_node_is_leader() could also check, say, rec->election_implementation->this_node_is_leader() as well. Marked this as being in 4.24.7 because it is in all supported versions and this is the earliest branch we will fix it in.
This bug was referenced in samba master: a01ade983b71640332798a97e7d6bcaa454b78ca
Created attachment 19280 [details] Patch for v4.25-test, v4-24-test Cherry-picks cleanly into v4-25-test. ./tests/INTEGRATION/simple/cluster.015.reclock_remove_lock.sh passes, so no likely regression. Almost impossible to test the actual race.
Comment on attachment 19280 [details] Patch for v4.25-test, v4-24-test Adding both original reviewers (thanks to both of you). Can whoever reviews first please delete the other, since we don't need 2 reviews. Thanks! :-)
I've updated the description of the attached (unchanged) patch to also mark it for v4-24-test (thanks @anoopcs). As with v4-25-test... It cherry-picks cleanly into v4-24-test. ./tests/INTEGRATION/simple/cluster.015.reclock_remove_lock.sh passes, so no likely regression. Some history has been lost here since nothing indicates the patch wasn't previously targeted at 4.24. @anoopcs, I thought it indicated that you had withdrawn your review while I sorted out the mess, but that might just have been stale state showing in my browser. Either way, if you're not happy that it shows +1 for your review then please consider rechecking. Either way, sorry for the noise...
Re-assigning for inclusion in 4.25 and 4.24.
This bug was referenced in samba v4-24-test: cace7d01ce30fca55e6151a0137d06bfec30d7bc
This bug was referenced in samba v4-24-stable (Release samba-4.24.8): cace7d01ce30fca55e6151a0137d06bfec30d7bc
This bug was referenced in samba v4-25-test: 8177022c4c0a27642946d02568b783007a904ae4
This bug was referenced in samba v4-25-stable (Release samba-4.25.1): 8177022c4c0a27642946d02568b783007a904ae4