Bug 16273 - A node can be banned because it tries to run recovery after losing the cluster lock
Summary: A node can be banned because it tries to run recovery after losing the cluste...
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: CTDB (show other bugs)
Version: 4.24.7
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Samba release manager
QA Contact: Samba QA Contact
URL: https://gitlab.com/samba-team/samba/-...
Keywords:
Depends on:
Blocks:
 
Reported: 2026-10-08 10:25 UTC by Martin Schwenke
Modified: 2026-10-23 15:39 UTC (History)
3 users (show)

See Also:


Attachments
Patch for v4.25-test, v4-24-test (5.03 KB, patch)
2026-10-09 22:06 UTC, Martin Schwenke
anoopcs: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Schwenke 2026-10-08 10:25:19 UTC
While running
ctdb/tests/INTEGRATION/simple/cluster.015.reclock_remove_lock.sh
this is seen:

2026-10-08T05:57:19.726102+00:00 node.0 ctdbd[2655369]: ctdb_mutex_fcntl_helper: lock lost - lock file "tests/var/INTEGRATION/simple/sh
ared/.ctdb/cluster.lock" open failed (ret=2)
2026-10-08T05:57:19.726937+00:00 node.0 ctdb-recoverd[2655437]: Cluster lock helper terminated
2026-10-08T05:57:19.726971+00:00 node.0 ctdb-recoverd[2655437]: Start election
2026-10-08T05:57:19.727053+00:00 node.0 ctdbd[2655369]: Recovery mode set to ACTIVE
2026-10-08T05:57:19.727071+00:00 node.1 ctdbd[2655392]: Recovery mode set to ACTIVE
2026-10-08T05:57:19.727071+00:00 node.2 ctdbd[2655407]: Recovery mode set to ACTIVE
2026-10-08T05:57:19.727161+00:00 node.0 ctdb-recoverd[2655437]: Attempting to take cluster lock (! /m/martins/aMASTER/b1443461/ctdb/ctd
b/bin/ctdb_mutex_fcntl_helper tests/var/INTEGRATION/simple/shared/.ctdb/cluster.lock 5)
2026-10-08T05:57:19.727198+00:00 node.1 ctdb-recoverd[2655480]: Attempting to take cluster lock (! /m/martins/aMASTER/b1443461/ctdb/ctd
b/bin/ctdb_mutex_fcntl_helper tests/var/INTEGRATION/simple/shared/.ctdb/cluster.lock 5)
2026-10-08T05:57:19.727212+00:00 node.2 ctdb-recoverd[2655503]: Attempting to take cluster lock (! /m/martins/aMASTER/b1443461/ctdb/ctd
b/bin/ctdb_mutex_fcntl_helper tests/var/INTEGRATION/simple/shared/.ctdb/cluster.lock 5)
2026-10-08T05:57:19.727885+00:00 node.0 ctdb-recoverd[2655437]: Received leader broadcast, leader=0
2026-10-08T05:57:19.731562+00:00 node.0 ctdb-recoverd[2655437]: Unable to take cluster lock - contention
2026-10-08T05:57:19.731579+00:00 node.2 ctdb-recoverd[2655503]: Unable to take cluster lock - contention
2026-10-08T05:57:19.731587+00:00 node.1 ctdb-recoverd[2655480]: Cluster lock taken successfully
2026-10-08T05:57:19.731602+00:00 node.1 ctdb-recoverd[2655480]: Took cluster lock, leader=1
2026-10-08T05:57:19.731976+00:00 node.0 ctdb-recoverd[2655437]: Node:0 was in recovery mode. Start recovery process
2026-10-08T05:57:19.732006+00:00 node.0 ctdb-recoverd[2655437]: Node:1 was in recovery mode. Start recovery process
2026-10-08T05:57:19.732022+00:00 node.0 ctdb-recoverd[2655437]: Node:2 was in recovery mode. Start recovery process
2026-10-08T05:57:19.732029+00:00 node.0 ctdb-recoverd[2655437]: do_recovery: Starting do_recovery
2026-10-08T05:57:19.732035+00:00 node.1 ctdb-recoverd[2655480]: Node:1 was in recovery mode. Start recovery process
2026-10-08T05:57:19.732050+00:00 node.1 ctdb-recoverd[2655480]: Node:0 was in recovery mode. Start recovery process
2026-10-08T05:57:19.732059+00:00 node.1 ctdb-recoverd[2655480]: Node:2 was in recovery mode. Start recovery process
2026-10-08T05:57:19.732062+00:00 node.0 ctdb-recoverd[2655437]: Cluster lock not held - abort recovery, ban node
2026-10-08T05:57:19.732063+00:00 node.1 ctdb-recoverd[2655480]: do_recovery: Starting do_recovery
2026-10-08T05:57:19.732067+00:00 node.0 ctdb-recoverd[2655437]: Banning node 0 for 300 seconds
2026-10-08T05:57:19.732082+00:00 node.0 ctdbd[2655369]: SET BAN STATE
2026-10-08T05:57:19.732090+00:00 node.0 ctdbd[2655369]: Banning this node for 300 seconds
2026-10-08T05:57:19.732094+00:00 node.0 ctdbd[2655369]: Making node INACTIVE
2026-10-08T05:57:19.732096+00:00 node.1 ctdb-recoverd[2655480]: do_recovery: Recovery initiated due to problem with node 0

This happens because, although the node does not hold the cluster lock,
this_node_is_leader() succeeds because rec->leader == rec->pnn is still
true, so the node can continue with recovery.  This happens even
though node 1 has taken the cluster lock and is leader.

There are 2 possible solutions:

1. Set rec->leader = CTDB_UNKNOWN_PNN before starting the election
   when the cluster lock helper terminates... and any similar
   circumstances.

2. Have this_node_is_leader() also check cluster_lock_held() if
   cluster_lock_enabled() is true.

(1) is simpler provided all the similar cases are found.

(2) pollutes the simple concept of the leader node with election
implementation details.  If this were properly modularised then
the generic this_node_is_leader() could also check, say,
rec->election_implementation->this_node_is_leader() as well.

Marked this as being in 4.24.7 because it is in all supported
versions and this is the earliest branch we will fix it in.
Comment 1 Samba QA Contact 2026-10-09 18:48:03 UTC
This bug was referenced in samba master:

a01ade983b71640332798a97e7d6bcaa454b78ca
Comment 2 Martin Schwenke 2026-10-09 22:06:44 UTC
Created attachment 19280 [details]
Patch for v4.25-test, v4-24-test

Cherry-picks cleanly into v4-25-test.  ./tests/INTEGRATION/simple/cluster.015.reclock_remove_lock.sh passes, so no likely regression.  Almost impossible to test the actual race.
Comment 3 Martin Schwenke 2026-10-09 22:08:10 UTC
Comment on attachment 19280 [details]
Patch for v4.25-test, v4-24-test

Adding both original reviewers (thanks to both of you).  Can whoever reviews first please delete the other, since we don't need 2 reviews.  Thanks!  :-)
Comment 4 Martin Schwenke 2026-10-10 11:02:37 UTC
I've updated the description of the attached (unchanged) patch to also mark it for v4-24-test (thanks @anoopcs).

As with v4-25-test...

It cherry-picks cleanly into v4-24-test.  ./tests/INTEGRATION/simple/cluster.015.reclock_remove_lock.sh passes, so no likely regression.

Some history has been lost here since nothing indicates the patch wasn't previously targeted at 4.24.

@anoopcs, I thought it indicated that you had withdrawn your review while I 
sorted out the mess, but that might just have been stale state showing in my 
browser.  Either way, if you're not happy that it shows +1 for your review then
please consider rechecking.

Either way, sorry for the noise...
Comment 5 Anoop C S 2026-10-10 11:49:04 UTC
Re-assigning for inclusion in 4.25 and 4.24.
Comment 6 Samba QA Contact 2026-10-10 20:51:04 UTC
This bug was referenced in samba v4-24-test:

cace7d01ce30fca55e6151a0137d06bfec30d7bc
Comment 7 Samba QA Contact 2026-10-12 19:26:41 UTC
This bug was referenced in samba v4-24-stable (Release samba-4.24.8):

cace7d01ce30fca55e6151a0137d06bfec30d7bc
Comment 8 Samba QA Contact 2026-10-13 19:15:05 UTC
This bug was referenced in samba v4-25-test:

8177022c4c0a27642946d02568b783007a904ae4
Comment 9 Samba QA Contact 2026-10-23 15:39:25 UTC
This bug was referenced in samba v4-25-stable (Release samba-4.25.1):

8177022c4c0a27642946d02568b783007a904ae4