Bug 12434 - ctdb recovery helper crash when low on disk space
Summary: ctdb recovery helper crash when low on disk space
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: CTDB (show other bugs)
Version: 4.5.0
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-11-18 01:03 UTC by Steve French
Modified: 2016-11-23 11:14 UTC (History)
3 users (show)

See Also:


Attachments
Patches for v4-4 (1.09 KB, patch)
2016-11-22 00:08 UTC, Amitay Isaacs
martins: review+
sfrench: review+
Details
Patches for v4-5 (1.09 KB, patch)
2016-11-22 00:08 UTC, Amitay Isaacs
martins: review+
sfrench: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Steve French 2016-11-18 01:03:11 UTC
our testers were doing a recovery scenario which involved
repeated reboots of the system (running Samba 4.5 - latest 4.5-stable
branch) which runs the ctdb recovery master, and hit repeated crashes
in ctdb_recovery_helper.  The exact location they couldn't give
me because the core files seemed corrupt.

Key part of reproduction scenario seemed to be that the system was
running of disk space (at least in /var).

and could be related with inability to create recovery files

2016/11/17 00:50:20.350797 [recoverd: 3800]: recovery: failed to
create recovery db /var/lib/ctdb/state/recdb.ctdb.tdb
2016/11/17 00:50:20.351082 [recoverd: 3800]: recovery: failed to
create recovery db /var/lib/ctdb/state/recdb.account_policy.tdb
2016/11/17 00:50:20.351691 [recoverd: 3800]: recovery: failed to
create recovery db /var/lib/ctdb/state/recdb.group_mapping.tdb
2016/11/17 00:50:20.352250 [recoverd: 3800]: recovery: failed to
create recovery db /var/lib/ctdb/state/recdb.passdb.tdb
2016/11/17 00:50:20.354377 [recoverd: 3800]: recovery: failed to
create recovery db /var/lib/ctdb/state/recdb.secrets.tdb
Comment 1 Steve French 2016-11-18 06:34:36 UTC
testing a fix for this now
Comment 2 Amitay Isaacs 2016-11-22 00:08:25 UTC
Created attachment 12680 [details]
Patches for v4-4
Comment 3 Amitay Isaacs 2016-11-22 00:08:44 UTC
Created attachment 12681 [details]
Patches for v4-5
Comment 4 Martin Schwenke 2016-11-22 05:22:33 UTC
Hi Karolin,

This is ready for 4.4. and 4.5.

Thanks!
Comment 5 Karolin Seeger 2016-11-22 09:02:49 UTC
(In reply to Martin Schwenke from comment #4)
Pushed to autobuild-v4-{5,4}-test.
Comment 6 Karolin Seeger 2016-11-23 11:14:17 UTC
(In reply to Karolin Seeger from comment #5)
Pushed to both branches..
Closing out bug report.

Thanks!