Attachment 10432 Details for Bug 10934 – this is the node which banned log when other node reboot,all nodes have same configure.

this is the node which banned log when other node reboot,all nodes have same configure.

log.ctdb (text/plain), 733.79 KB, created by 郭成峰 on 2014-11-14 04:51:44 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: 郭成峰

Created: 2014-11-14 04:51:44 UTC

Size: 733.79 KB

patch

obsolete

>2014/10/25 14:46:36.253271 [ 1370]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/25 14:46:36.254020 [ 1370]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 14:46:36.254038 [ 1370]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 16:20:44.553577 [ 1404]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/25 16:20:44.553694 [ 1404]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 16:20:44.553707 [ 1404]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 17:48:34.171983 [ 1582]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/25 17:48:34.172075 [ 1582]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 17:48:34.172088 [ 1582]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 18:19:11.715748 [ 1559]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/25 18:19:11.715876 [ 1559]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 18:19:11.715890 [ 1559]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 20:38:12.602655 [ 1357]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/25 20:38:12.602748 [ 1357]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/25 20:38:12.602762 [ 1357]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 11:09:01.636613 [ 1366]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/27 11:09:01.636716 [ 1366]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 11:09:01.636730 [ 1366]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 11:41:25.818698 [ 1376]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/27 11:41:25.818793 [ 1376]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 11:41:25.818807 [ 1376]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 14:33:48.484936 [ 8746]: Starting CTDBD (Version 2.5.3) as PID: 8746
>2014/10/27 14:33:48.757756 [ 8746]: Freeze priority 1
>2014/10/27 14:33:48.775278 [ 8746]: Freeze priority 2
>2014/10/27 14:33:48.775676 [ 8746]: Freeze priority 3
>2014/10/27 14:33:48.803510 [ 8746]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/27 14:33:48.807169 [ 8746]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/27 14:33:48.810707 [ 8746]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/27 14:33:48.933404 [ 8746]: Freeze priority 1
>2014/10/27 14:33:48.933477 [ 8746]: Freeze priority 2
>2014/10/27 14:33:48.933531 [ 8746]: Freeze priority 3
>2014/10/27 14:33:49.306475 [ 8746]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/27 14:33:52.438828 [recoverd: 8928]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/27 14:33:52.438926 [ 8746]: Freeze priority 1
>2014/10/27 14:33:52.439000 [ 8746]: Freeze priority 2
>2014/10/27 14:33:52.439065 [ 8746]: Freeze priority 3
>2014/10/27 14:33:57.664734 [ 8746]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/27 14:33:57.680496 [ 8746]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/27 14:33:57.696523 [ 8746]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/27 14:33:57.711980 [ 8746]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/27 14:33:57.727396 [ 8746]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/27 14:33:57.760824 [ 8746]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/27 14:33:57.778460 [ 8746]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/27 14:33:57.788110 [ 8746]: Freeze priority 1
>2014/10/27 14:33:57.789290 [ 8746]: Freeze priority 2
>2014/10/27 14:33:57.790215 [ 8746]: Freeze priority 3
>2014/10/27 14:33:57.950336 [ 8746]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/27 14:33:59.296884 [ 8746]: Thawing priority 1
>2014/10/27 14:33:59.296919 [ 8746]: Release freeze handler for prio 1
>2014/10/27 14:33:59.296961 [ 8746]: Thawing priority 2
>2014/10/27 14:33:59.296981 [ 8746]: Release freeze handler for prio 2
>2014/10/27 14:33:59.297009 [ 8746]: Thawing priority 3
>2014/10/27 14:33:59.297027 [ 8746]: Release freeze handler for prio 3
>2014/10/27 14:34:17.186801 [ 8746]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/27 14:34:18.419310 [ 8746]: Freeze priority 1
>2014/10/27 14:34:18.420561 [ 8746]: Freeze priority 2
>2014/10/27 14:34:18.421533 [ 8746]: Freeze priority 3
>2014/10/27 14:35:08.978918 [ 8746]: DB Attach to database ctdb.tdb deferred for client with pid:11885 since node is in recovery mode.
>2014/10/27 14:35:13.299547 [ 8746]: Event script '00.ctdb startup ' timed out after 60.0s, count: 0, pid: 10371
>2014/10/27 14:35:13.299605 [ 8746]: startup event failed
>2014/10/27 14:35:15.847649 [ 8746]: pnn 2 Invalid reqid 107 in ctdb_reply_control
>2014/10/27 14:35:15.847688 [ 8746]: pnn 2 Invalid reqid 117 in ctdb_reply_control
>2014/10/27 14:35:18.300311 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:18.300362 [ 8746]: Unable to launch startup event script
>2014/10/27 14:35:23.300633 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:23.300687 [ 8746]: Unable to launch startup event script
>2014/10/27 14:35:27.134015 [ 8746]: pnn 2 Invalid reqid 135 in ctdb_reply_control
>2014/10/27 14:35:28.301700 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:28.301738 [ 8746]: Unable to launch startup event script
>2014/10/27 14:35:33.301813 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:33.301862 [ 8746]: Unable to launch startup event script
>2014/10/27 14:35:38.302097 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:38.302148 [ 8746]: Unable to launch startup event script
>2014/10/27 14:35:43.303245 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:43.303289 [ 8746]: Unable to launch startup event script
>2014/10/27 14:35:46.670637 [ 8746]: pnn 2 Invalid reqid 157 in ctdb_reply_control
>2014/10/27 14:35:48.303745 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:48.303787 [ 8746]: Unable to launch startup event script
>2014/10/27 14:35:53.304773 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:53.304837 [ 8746]: Unable to launch startup event script
>2014/10/27 14:35:58.305306 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:35:58.305350 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:03.305562 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:03.305599 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:05.617439 [ 8746]: Freeze priority 1
>2014/10/27 14:36:05.618316 [ 8746]: Freeze priority 2
>2014/10/27 14:36:05.619245 [ 8746]: Freeze priority 3
>2014/10/27 14:36:08.306461 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:08.306511 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:13.300559 [ 8746]: ctdb_kill: trying to kill(10371, 9) a process that does not exist
>2014/10/27 14:36:13.306683 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:13.306723 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:18.307569 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:18.307622 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:19.312060 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:36:19.312100 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:36:19.312135 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 14:36:19.312149 [recoverd: 8928]: Async wait failed - fail_count=1
>2014/10/27 14:36:19.312162 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 14:36:19.312208 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 14:36:23.308311 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:23.308359 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:28.308846 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:28.308900 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:33.309870 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:33.309911 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:38.310508 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:38.310550 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:43.311223 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:43.311266 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:44.999377 [ 8746]: pnn 2 Invalid reqid 183 in ctdb_reply_control
>2014/10/27 14:36:48.312297 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:48.312338 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:53.313194 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:53.313231 [ 8746]: Unable to launch startup event script
>2014/10/27 14:36:58.314054 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:36:58.314094 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:03.314426 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:03.314463 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:05.621025 [ 8746]: Event script '01.reclock startrecovery ' timed out after 10.2s, count: 0, pid: 15118
>2014/10/27 14:37:05.621067 [ 8746]: Ignoring hung script for  call 3
>2014/10/27 14:37:05.637662 [ 8746]: Freeze priority 1
>2014/10/27 14:37:05.638022 [ 8746]: Freeze priority 2
>2014/10/27 14:37:05.638354 [ 8746]: Freeze priority 3
>2014/10/27 14:37:08.315363 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:08.315406 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:13.315531 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:13.315581 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:18.316524 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:18.316565 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:23.316954 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:23.317007 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:28.317789 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:28.317859 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:33.318425 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:33.318461 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:38.319129 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:38.319193 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:43.319607 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:43.319668 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:47.996188 [ 8746]: Hung-script: ===== Start of hung script debug for PID="15118", event="startrecovery" =====
>2014/10/27 14:37:47.996231 [ 8746]: Hung-script: pstree -p -a 15118:
>2014/10/27 14:37:48.016425 [ 8746]: Hung-script: 
>2014/10/27 14:37:48.018469 [ 8746]: Hung-script: ---- ctdb scriptstatus startrecovery: ----
>2014/10/27 14:37:48.019791 [ 8746]: Hung-script: 2 scripts were executed last startrecovery cycle
>2014/10/27 14:37:48.019869 [ 8746]: Hung-script: 00.ctdb              Status:OK    Duration:49.798 Mon Oct 27 14:36:05 2014
>2014/10/27 14:37:48.019903 [ 8746]: Hung-script: 01.reclock           Status:OK    Duration:-1414391815.419 Mon Oct 27 14:36:55 2014
>2014/10/27 14:37:48.020084 [ 8746]: Hung-script: ===== End of hung script debug for PID="15118", event="startrecovery" =====
>2014/10/27 14:37:48.020563 [ 8746]: ctdb_kill: trying to kill(15118, 9) a process that does not exist
>2014/10/27 14:37:48.144677 [ 8746]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/27 14:37:48.320081 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:37:48.320124 [ 8746]: Unable to launch startup event script
>2014/10/27 14:37:49.272439 [ 8746]: Thawing priority 1
>2014/10/27 14:37:49.272496 [ 8746]: Release freeze handler for prio 1
>2014/10/27 14:37:49.272539 [ 8746]: Thawing priority 2
>2014/10/27 14:37:49.272556 [ 8746]: Release freeze handler for prio 2
>2014/10/27 14:37:49.272593 [ 8746]: Thawing priority 3
>2014/10/27 14:37:49.272604 [ 8746]: Release freeze handler for prio 3
>2014/10/27 14:37:57.358439 [ 8746]: Recovery daemon ping timeout. Count : 0
>2014/10/27 14:37:57.359647 [recoverd: 8928]: ctdb_control error: 'ctdb_control timed out'
>2014/10/27 14:37:57.359693 [recoverd: 8928]: ctdb_control error: 'ctdb_control timed out'
>2014/10/27 14:37:57.359707 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 14:37:57.359717 [recoverd: 8928]: Async wait failed - fail_count=1
>2014/10/27 14:37:57.359727 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 14:37:57.359737 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 14:38:14.325504 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:38:14.325562 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:38:14.325586 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 14:38:14.325599 [recoverd: 8928]: Async wait failed - fail_count=1
>2014/10/27 14:38:14.325611 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 14:38:14.325624 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 14:38:47.373307 [ 8746]: pnn 2 Invalid reqid 553 in ctdb_reply_control
>2014/10/27 14:38:53.320762 [ 8746]: Event script '01.reclock startup ' timed out after 19.7s, count: 0, pid: 18127
>2014/10/27 14:38:53.320814 [ 8746]: startup event failed
>2014/10/27 14:39:07.211179 [ 8746]: Event script '00.ctdb ipreallocated ' timed out after 60.0s, count: 0, pid: 17319
>2014/10/27 14:39:07.211234 [ 8746]:  "ipreallocated" event script failed (status -62)
>2014/10/27 14:39:07.211247 [ 8746]: Banning this node for 30 seconds
>2014/10/27 14:39:07.211259 [ 8746]: Freeze priority 1
>2014/10/27 14:39:07.211395 [ 8746]: Freeze priority 2
>2014/10/27 14:39:07.211578 [ 8746]: Freeze priority 3
>2014/10/27 14:39:27.374013 [ 8746]: Event script '00.ctdb recovered ' timed out after 60.0s, count: 0, pid: 17950
>2014/10/27 14:39:27.374058 [ 8746]: Ignoring hung script for  call 4
>2014/10/27 14:39:37.211760 [ 8746]: Banning timedout
>2014/10/27 14:39:45.530034 [ 8746]: Hung-script: ===== Start of hung script debug for PID="18127", event="startup" =====
>2014/10/27 14:39:45.530082 [ 8746]: Hung-script: pstree -p -a 18127:
>2014/10/27 14:39:45.531213 [ 8746]: DB Attach to database ctdb.tdb deferred for client with pid:20292 since node is in recovery mode.
>2014/10/27 14:39:45.551972 [ 8746]: Hung-script: 
>2014/10/27 14:39:45.553712 [ 8746]: Hung-script: ---- ctdb scriptstatus startup: ----
>2014/10/27 14:39:45.554805 [ 8746]: Hung-script: 2 scripts were executed last startup cycle
>2014/10/27 14:39:45.554860 [ 8746]: Hung-script: 00.ctdb              Status:OK    Duration:40.325 Mon Oct 27 14:37:53 2014
>2014/10/27 14:39:45.554873 [ 8746]: Hung-script: 01.reclock           Status:TIMEDOUT    Mon Oct 27 14:38:33 2014
>2014/10/27 14:39:45.554882 [ 8746]: Hung-script:    OUTPUT:
>2014/10/27 14:39:45.555015 [ 8746]: Hung-script: ===== End of hung script debug for PID="18127", event="startup" =====
>2014/10/27 14:39:45.555469 [ 8746]: ctdb_kill: trying to kill(18127, 9) a process that does not exist
>2014/10/27 14:39:58.321988 [ 8746]: Event script '00.ctdb startup ' timed out after 60.0s, count: 0, pid: 18798
>2014/10/27 14:39:58.322028 [ 8746]: startup event failed
>2014/10/27 14:40:03.322925 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:03.322964 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:04.489585 [ 8746]: Hung-script: ===== Start of hung script debug for PID="17950", event="recovered" =====
>2014/10/27 14:40:04.489632 [ 8746]: Hung-script: pstree -p -a 17950:
>2014/10/27 14:40:04.509927 [ 8746]: Hung-script: 
>2014/10/27 14:40:04.511773 [ 8746]: Hung-script: ---- ctdb scriptstatus recovered: ----
>2014/10/27 14:40:04.512873 [ 8746]: Hung-script: 1 scripts were executed last recovered cycle
>2014/10/27 14:40:04.512972 [ 8746]: Hung-script: 00.ctdb              Status:OK    Duration:-1414391907.374 Mon Oct 27 14:38:27 2014
>2014/10/27 14:40:04.513138 [ 8746]: Hung-script: ===== End of hung script debug for PID="17950", event="recovered" =====
>2014/10/27 14:40:04.513529 [ 8746]: ctdb_kill: trying to kill(17950, 9) a process that does not exist
>2014/10/27 14:40:07.212833 [ 8746]: ctdb_kill: trying to kill(17319, 9) a process that does not exist
>2014/10/27 14:40:08.323350 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:08.323389 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:13.323485 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:13.323526 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:17.408840 [ 8746]: Freeze priority 1
>2014/10/27 14:40:17.409180 [ 8746]: Freeze priority 2
>2014/10/27 14:40:17.409499 [ 8746]: Freeze priority 3
>2014/10/27 14:40:18.324194 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:18.324242 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:23.324713 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:23.324754 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:28.325330 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:28.325383 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:29.341740 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:40:29.341784 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:40:29.341833 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 14:40:29.341853 [recoverd: 8928]: Async wait failed - fail_count=1
>2014/10/27 14:40:29.341885 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 14:40:29.341897 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 14:40:33.325580 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:33.325619 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:36.370006 [ 8746]: Hung-script: ===== Start of hung script debug for PID="18798", event="startup" =====
>2014/10/27 14:40:36.370073 [ 8746]: Hung-script: pstree -p -a 18798:
>2014/10/27 14:40:36.389563 [ 8746]: Hung-script: ctdb_event_help,18798 47 44 /etc/ctdb/events.d/00.ctdb startup
>2014/10/27 14:40:36.389600 [ 8746]: Hung-script:   `-00.ctdb,18799 /etc/ctdb/events.d/00.ctdb startup
>2014/10/27 14:40:36.389620 [ 8746]: Hung-script:       `-ctdb,20292 attach ctdb.tdb persistent
>2014/10/27 14:40:36.391450 [ 8746]: Hung-script: ---- ctdb scriptstatus startup: ----
>2014/10/27 14:40:36.392503 [ 8746]: Hung-script: 1 scripts were executed last startup cycle
>2014/10/27 14:40:36.392582 [ 8746]: Hung-script: 00.ctdb              Status:TIMEDOUT    Mon Oct 27 14:38:58 2014
>2014/10/27 14:40:36.392612 [ 8746]: Hung-script:    OUTPUT:
>2014/10/27 14:40:36.392757 [ 8746]: Hung-script: ===== End of hung script debug for PID="18798", event="startup" =====
>2014/10/27 14:40:38.326545 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:38.326606 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:43.327570 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:43.327629 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:48.328321 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:48.328376 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:53.328507 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:53.328544 [ 8746]: Unable to launch startup event script
>2014/10/27 14:40:55.951831 [ 8746]: pnn 2 Invalid reqid 946 in ctdb_reply_control
>2014/10/27 14:40:58.328733 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:40:58.328786 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:03.328929 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:03.328984 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:08.329575 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:08.329635 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:13.330085 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:13.330128 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:17.410721 [ 8746]: Event script '01.reclock startrecovery ' timed out after 19.0s, count: 0, pid: 22264
>2014/10/27 14:41:17.410762 [ 8746]: Ignoring hung script for  call 3
>2014/10/27 14:41:18.331065 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:18.331101 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:23.332182 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:23.332220 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:28.332477 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:28.332514 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:33.332972 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:33.333012 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:34.731784 [ 8746]: Hung-script: ===== Start of hung script debug for PID="22264", event="startrecovery" =====
>2014/10/27 14:41:34.731864 [ 8746]: Hung-script: pstree -p -a 22264:
>2014/10/27 14:41:34.754781 [ 8746]: Hung-script: ctdb_event_help,22264 43 40 /etc/ctdb/events.d/01.reclock startrecovery
>2014/10/27 14:41:34.754834 [ 8746]: Hung-script:   `-01.reclock,22265 /etc/ctdb/events.d/01.reclock startrecovery
>2014/10/27 14:41:34.756684 [ 8746]: Hung-script: ---- ctdb scriptstatus startrecovery: ----
>2014/10/27 14:41:34.757792 [ 8746]: Hung-script: 2 scripts were executed last startrecovery cycle
>2014/10/27 14:41:34.757895 [ 8746]: Hung-script: 00.ctdb              Status:OK    Duration:41.033 Mon Oct 27 14:40:17 2014
>2014/10/27 14:41:34.757930 [ 8746]: Hung-script: 01.reclock           Status:OK    Duration:-1414392058.443 Mon Oct 27 14:40:58 2014
>2014/10/27 14:41:34.758094 [ 8746]: Hung-script: ===== End of hung script debug for PID="22264", event="startrecovery" =====
>2014/10/27 14:41:38.333598 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:38.333646 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:43.334282 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:43.334326 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:48.334960 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:48.335002 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:53.335097 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:53.335135 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:58.335444 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:41:58.335494 [ 8746]: Unable to launch startup event script
>2014/10/27 14:41:59.473019 [ 8746]: Recovery daemon ping timeout. Count : 0
>2014/10/27 14:42:03.336053 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:03.336092 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:08.336534 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:08.336576 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:08.840861 [recoverd: 8928]: ctdb_control error: 'ctdb_control timed out'
>2014/10/27 14:42:08.840924 [recoverd: 8928]: client/ctdb_client.c:1535 ctdb_control for getnodes failed ret:-1 res:-1
>2014/10/27 14:42:08.840940 [recoverd: 8928]: server/ctdb_recoverd.c:3739 Unable to get nodemap from recovery master 0
>2014/10/27 14:42:13.336880 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:13.336918 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:14.354230 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:42:14.354272 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:42:14.354288 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 14:42:14.354313 [recoverd: 8928]: Async wait failed - fail_count=1
>2014/10/27 14:42:14.354333 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 14:42:14.354348 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 14:42:17.410733 [ 8746]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS
>2014/10/27 14:42:18.337586 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:18.337618 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:23.338484 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:23.338522 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:27.160216 [ 8746]: Freeze priority 1
>2014/10/27 14:42:27.161408 [ 8746]: Freeze priority 2
>2014/10/27 14:42:27.162366 [ 8746]: Freeze priority 3
>2014/10/27 14:42:28.339564 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:28.339609 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:33.340401 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:33.340438 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:37.194220 [ 8746]: Freeze priority 1
>2014/10/27 14:42:37.195016 [ 8746]: Freeze priority 2
>2014/10/27 14:42:37.195707 [ 8746]: Freeze priority 3
>2014/10/27 14:42:38.340718 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:38.340753 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:43.341599 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:43.341638 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:48.341980 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:48.342035 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:53.342879 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:53.342921 [ 8746]: Unable to launch startup event script
>2014/10/27 14:42:58.343809 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:42:58.343859 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:03.344517 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:03.344557 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:08.344781 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:08.344833 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:13.345570 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:13.345607 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:14.359872 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:43:14.359912 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:43:14.359927 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 14:43:14.359938 [recoverd: 8928]: Async wait failed - fail_count=1
>2014/10/27 14:43:14.359949 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 14:43:14.359960 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 14:43:18.346502 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:18.346540 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:19.360178 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:43:19.360216 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:43:19.360230 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 14:43:19.360241 [recoverd: 8928]: Async wait failed - fail_count=1
>2014/10/27 14:43:19.360250 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 14:43:19.360275 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 14:43:19.361452 [ 8746]: Freeze priority 1
>2014/10/27 14:43:19.361641 [ 8746]: Freeze priority 2
>2014/10/27 14:43:19.361753 [ 8746]: Freeze priority 3
>2014/10/27 14:43:22.366203 [recoverd: 8928]: Taking out recovery lock from recovery daemon
>2014/10/27 14:43:22.366244 [recoverd: 8928]: Take the recovery lock
>2014/10/27 14:43:23.347255 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:23.347295 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:28.348177 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:28.348215 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:33.348573 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:33.348605 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:37.197173 [ 8746]: Event script '00.ctdb startrecovery ' timed out after 60.0s, count: 0, pid: 25254
>2014/10/27 14:43:37.197212 [ 8746]: Ignoring hung script for  call 3
>2014/10/27 14:43:38.349397 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:38.349437 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:43.349727 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:43.349765 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:45.977333 [ 8746]: pnn 2 Invalid reqid 1267 in ctdb_reply_control
>2014/10/27 14:43:48.350156 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:48.350191 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:52.092445 [ 8746]: Freeze priority 1
>2014/10/27 14:43:52.093804 [ 8746]: Freeze priority 2
>2014/10/27 14:43:52.094877 [ 8746]: Freeze priority 3
>2014/10/27 14:43:53.350335 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:53.350376 [ 8746]: Unable to launch startup event script
>2014/10/27 14:43:55.101949 [ 8746]: Freeze priority 1
>2014/10/27 14:43:55.102316 [ 8746]: Freeze priority 2
>2014/10/27 14:43:55.102751 [ 8746]: Freeze priority 3
>2014/10/27 14:43:58.109464 [ 8746]: Freeze priority 1
>2014/10/27 14:43:58.110147 [ 8746]: Freeze priority 2
>2014/10/27 14:43:58.110693 [ 8746]: Freeze priority 3
>2014/10/27 14:43:58.351027 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:43:58.351073 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:03.033628 [ 8746]: High RECLOCK latency 40.667294s for operation recd reclock
>2014/10/27 14:44:03.036598 [ 8746]: Hung-script: ===== Start of hung script debug for PID="25254", event="startrecovery" =====
>2014/10/27 14:44:03.036644 [ 8746]: Hung-script: pstree -p -a 25254:
>2014/10/27 14:44:03.037394 [ 8746]: Freeze priority 1
>2014/10/27 14:44:03.037496 [ 8746]: Freeze priority 2
>2014/10/27 14:44:03.037577 [ 8746]: Freeze priority 3
>2014/10/27 14:44:03.059250 [ 8746]: Hung-script: 
>2014/10/27 14:44:03.061101 [ 8746]: Hung-script: ---- ctdb scriptstatus startrecovery: ----
>2014/10/27 14:44:03.062361 [ 8746]: Hung-script: 1 scripts were executed last startrecovery cycle
>2014/10/27 14:44:03.062459 [ 8746]: Hung-script: 00.ctdb              Status:OK    Duration:-1414392157.197 Mon Oct 27 14:42:37 2014
>2014/10/27 14:44:03.062670 [ 8746]: Hung-script: ===== End of hung script debug for PID="25254", event="startrecovery" =====
>2014/10/27 14:44:03.063163 [ 8746]: ctdb_kill: trying to kill(25254, 9) a process that does not exist
>2014/10/27 14:44:03.351886 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:03.351914 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:08.352174 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:08.352218 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:13.352731 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:13.352779 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:18.353248 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:18.353295 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:22.365471 [ 8746]: Recovery daemon ping timeout. Count : 0
>2014/10/27 14:44:23.353654 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:23.353695 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:28.353847 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:28.353904 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:30.727645 [ 8746]: pnn 2 Invalid reqid 1080 in ctdb_reply_control
>2014/10/27 14:44:30.727740 [ 8746]: pnn 2 Invalid reqid 1112 in ctdb_reply_control
>2014/10/27 14:44:30.727794 [ 8746]: pnn 2 Invalid reqid 1123 in ctdb_reply_control
>2014/10/27 14:44:30.727834 [ 8746]: pnn 2 Invalid reqid 1139 in ctdb_reply_control
>2014/10/27 14:44:30.727852 [ 8746]: pnn 2 Invalid reqid 1154 in ctdb_reply_control
>2014/10/27 14:44:33.354099 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:33.354134 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:38.354974 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:38.355041 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:43.355657 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:43.355707 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:48.355858 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:48.355917 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:53.356397 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:53.356440 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:58.357058 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:44:58.357111 [ 8746]: Unable to launch startup event script
>2014/10/27 14:44:59.380501 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:44:59.380546 [recoverd: 8928]: ctdb_control error: 'node is disconnected'
>2014/10/27 14:44:59.380563 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=52
>2014/10/27 14:44:59.380576 [recoverd: 8928]: Async wait failed - fail_count=1
>2014/10/27 14:44:59.380588 [recoverd: 8928]: client/ctdb_client.c:3027 Unable to update nodeflags on remote nodes
>2014/10/27 14:44:59.380606 [recoverd: 8928]: server/ctdb_recoverd.c:873 Unable to update nodeflags on remote nodes
>2014/10/27 14:44:59.380618 [recoverd: 8928]: server/ctdb_recoverd.c:1857 Unable to update flags on all nodes for node 2
>2014/10/27 14:44:59.381616 [ 8746]: Freeze priority 1
>2014/10/27 14:44:59.381787 [ 8746]: Freeze priority 2
>2014/10/27 14:44:59.381985 [ 8746]: Freeze priority 3
>2014/10/27 14:45:02.384545 [recoverd: 8928]: Taking out recovery lock from recovery daemon
>2014/10/27 14:45:02.384589 [recoverd: 8928]: Take the recovery lock
>2014/10/27 14:45:03.358068 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:03.358105 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:08.358215 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:08.358255 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:13.358454 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:13.358499 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:18.359185 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:18.359223 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:23.359378 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:23.359411 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:28.360026 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:28.360078 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:33.361105 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:33.361142 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:38.362182 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:38.362232 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:43.362338 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:43.362407 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:45.068022 [recoverd: 8928]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile'
>2014/10/27 14:45:45.068077 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 14:45:45.068166 [ 8746]: Banning this node for 30 seconds
>2014/10/27 14:45:48.363137 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:48.363188 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:53.363291 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:53.363333 [ 8746]: Unable to launch startup event script
>2014/10/27 14:45:58.363651 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:45:58.363702 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:03.364135 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:03.364166 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:08.364364 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:08.364417 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:13.365225 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:13.365270 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:15.068727 [ 8746]: Banning timedout
>2014/10/27 14:46:15.108171 [recoverd: 8928]: Taking out recovery lock from recovery daemon
>2014/10/27 14:46:15.108213 [recoverd: 8928]: Take the recovery lock
>2014/10/27 14:46:15.110044 [recoverd: 8928]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile'
>2014/10/27 14:46:15.110078 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 14:46:15.110145 [ 8746]: Banning this node for 30 seconds
>2014/10/27 14:46:18.366197 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:18.366239 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:23.366844 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:23.366893 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:25.268013 [ 8746]: tcp/tcp_connect.c:211Failed to bind socket Cannot assign requested address(99)
>2014/10/27 14:46:25.268112 [ 8746]: tcp/tcp_connect.c:211Failed to bind socket Cannot assign requested address(99)
>2014/10/27 14:46:28.367972 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:28.368009 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:33.368470 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:33.368499 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:38.369315 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:38.369370 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:43.370283 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:43.370333 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:45.110631 [ 8746]: Banning timedout
>2014/10/27 14:46:45.150500 [recoverd: 8928]: Taking out recovery lock from recovery daemon
>2014/10/27 14:46:45.150541 [recoverd: 8928]: Take the recovery lock
>2014/10/27 14:46:48.371229 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:48.371273 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:53.371388 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:53.371435 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:58.371644 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:46:58.371695 [ 8746]: Unable to launch startup event script
>2014/10/27 14:46:59.382834 [ 8746]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS
>2014/10/27 14:47:03.372386 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:03.372423 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:08.372660 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:08.372714 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:09.289023 [recoverd: 8928]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected)
>2014/10/27 14:47:09.289061 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 14:47:09.289152 [ 8746]: Banning this node for 30 seconds
>2014/10/27 14:47:13.373525 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:13.373568 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:18.373767 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:18.373828 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:23.374751 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:23.374793 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:28.375507 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:28.375555 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:33.376581 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:33.376628 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:38.377172 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:38.377236 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:39.289419 [ 8746]: Banning timedout
>2014/10/27 14:47:39.336834 [recoverd: 8928]: Taking out recovery lock from recovery daemon
>2014/10/27 14:47:39.336883 [recoverd: 8928]: Take the recovery lock
>2014/10/27 14:47:39.337112 [recoverd: 8928]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected)
>2014/10/27 14:47:39.337127 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 14:47:39.337157 [ 8746]: Banning this node for 30 seconds
>2014/10/27 14:47:43.377914 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:43.377983 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:48.378941 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:48.378987 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:53.379370 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:53.379410 [ 8746]: Unable to launch startup event script
>2014/10/27 14:47:58.379968 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:47:58.380005 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:03.380810 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:03.380862 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:08.381875 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:08.381921 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:09.338166 [ 8746]: Banning timedout
>2014/10/27 14:48:09.381036 [recoverd: 8928]: Taking out recovery lock from recovery daemon
>2014/10/27 14:48:09.381066 [recoverd: 8928]: Take the recovery lock
>2014/10/27 14:48:09.381290 [recoverd: 8928]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected)
>2014/10/27 14:48:09.381314 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 14:48:09.381339 [ 8746]: Banning this node for 30 seconds
>2014/10/27 14:48:13.382172 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:13.382232 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:18.382447 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:18.382495 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:23.383002 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:23.383046 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:28.383141 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:28.383180 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:33.384171 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:33.384219 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:38.384919 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:38.384976 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:39.382289 [ 8746]: Banning timedout
>2014/10/27 14:48:39.423250 [recoverd: 8928]: Taking out recovery lock from recovery daemon
>2014/10/27 14:48:39.423280 [recoverd: 8928]: Take the recovery lock
>2014/10/27 14:48:39.423557 [recoverd: 8928]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected)
>2014/10/27 14:48:39.423573 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 14:48:39.423630 [ 8746]: Banning this node for 30 seconds
>2014/10/27 14:48:43.385296 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:43.385346 [ 8746]: Unable to launch startup event script
>2014/10/27 14:48:48.385973 [ 8746]: Refusing to run event scripts call 'startup' while in recovery
>2014/10/27 14:48:48.386017 [ 8746]: Unable to launch startup event script
>2014/10/27 14:50:43.926813 [ 1649]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/27 14:50:43.926913 [ 1649]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 14:50:43.926927 [ 1649]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 15:05:41.521877 [ 1686]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/27 15:05:41.521977 [ 1686]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 15:05:41.521992 [ 1686]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/27 16:17:47.005708 [ 1701]: Starting CTDBD (Version 2.5.3) as PID: 1701
>2014/10/27 16:17:47.729112 [ 1701]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/27 16:17:47.744014 [ 1701]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/27 16:17:47.759088 [ 1701]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/27 16:17:47.774176 [ 1701]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/27 16:17:47.788893 [ 1701]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/27 16:17:47.803150 [ 1701]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/27 16:17:47.817911 [ 1701]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/27 16:17:47.817981 [ 1701]: Freeze priority 1
>2014/10/27 16:17:47.832442 [ 1701]: Freeze priority 2
>2014/10/27 16:17:47.832827 [ 1701]: Freeze priority 3
>2014/10/27 16:17:47.922136 [ 1701]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/27 16:17:47.926218 [ 1701]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/27 16:17:47.929747 [ 1701]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/27 16:17:48.073929 [ 1701]: Freeze priority 1
>2014/10/27 16:17:48.074022 [ 1701]: Freeze priority 2
>2014/10/27 16:17:48.074079 [ 1701]: Freeze priority 3
>2014/10/27 16:17:51.578941 [recoverd: 1929]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/27 16:17:51.579031 [ 1701]: Freeze priority 1
>2014/10/27 16:17:51.579095 [ 1701]: Freeze priority 2
>2014/10/27 16:17:51.579149 [ 1701]: Freeze priority 3
>2014/10/27 16:17:55.078318 [ 1701]: Freeze priority 1
>2014/10/27 16:17:55.079577 [ 1701]: Freeze priority 2
>2014/10/27 16:17:55.080468 [ 1701]: Freeze priority 3
>2014/10/27 16:18:55.095036 [ 1701]: Freeze priority 1
>2014/10/27 16:18:55.095408 [ 1701]: Freeze priority 2
>2014/10/27 16:18:55.095945 [ 1701]: Freeze priority 3
>2014/10/27 16:18:59.902815 [ 1701]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/27 16:19:00.120810 [ 1701]: Freeze priority 1
>2014/10/27 16:19:00.121516 [ 1701]: Freeze priority 2
>2014/10/27 16:19:00.122113 [ 1701]: Freeze priority 3
>2014/10/27 16:19:00.267888 [ 1701]: server/ctdb_freeze.c:344 recovery transaction cancelled called
>2014/10/27 16:19:01.179939 [ 1701]: Freeze priority 1
>2014/10/27 16:19:01.180091 [ 1701]: Freeze priority 2
>2014/10/27 16:19:01.180195 [ 1701]: Freeze priority 3
>2014/10/27 16:19:04.185123 [recoverd: 1929]: Taking out recovery lock from recovery daemon
>2014/10/27 16:19:04.185181 [recoverd: 1929]: Take the recovery lock
>2014/10/27 16:19:04.189852 [ 1701]: Freeze priority 1
>2014/10/27 16:19:04.189923 [ 1701]: Freeze priority 2
>2014/10/27 16:19:04.189971 [ 1701]: Freeze priority 3
>2014/10/27 16:19:05.462859 [ 1701]: Thawing priority 1
>2014/10/27 16:19:05.462919 [ 1701]: Release freeze handler for prio 1
>2014/10/27 16:19:05.462948 [ 1701]: Thawing priority 2
>2014/10/27 16:19:05.462960 [ 1701]: Release freeze handler for prio 2
>2014/10/27 16:19:05.462979 [ 1701]: Thawing priority 3
>2014/10/27 16:19:05.462990 [ 1701]: Release freeze handler for prio 3
>2014/10/27 16:19:05.797116 [recoverd: 1929]: Resetting ban count to 0 for all nodes
>2014/10/27 16:19:19.767718 [ 1701]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/27 16:19:19.812846 [recoverd: 1929]: Trigger takeoverrun
>2014/10/27 16:19:20.056781 [ 1701]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/27 16:19:20.068021 [ 1701]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/27 16:19:20.089715 [ 1701]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/27 16:19:20.284702 [ 1701]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/27 16:19:22.616634 [ 1701]: Node became HEALTHY. Ask recovery master 2 to perform ip reallocation
>2014/10/27 16:19:22.816971 [recoverd: 1929]: Public IP '10.10.10.184' is not assigned and we could serve it
>2014/10/27 16:19:22.817032 [recoverd: 1929]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/27 16:19:22.817043 [recoverd: 1929]: Public IP '10.10.10.182' is not assigned and we could serve it
>2014/10/27 16:19:22.817123 [recoverd: 1929]: Trigger takeoverrun
>2014/10/27 16:19:23.136642 [ 1701]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/27 16:19:31.153481 [ 1701]: Freeze priority 1
>2014/10/27 16:19:31.154651 [ 1701]: Freeze priority 2
>2014/10/27 16:19:31.155601 [ 1701]: Freeze priority 3
>2014/10/27 16:19:34.171634 [ 1701]: Freeze priority 1
>2014/10/27 16:19:34.171931 [ 1701]: Freeze priority 2
>2014/10/27 16:19:34.172190 [ 1701]: Freeze priority 3
>2014/10/27 16:19:35.433529 [ 1701]: Thawing priority 1
>2014/10/27 16:19:35.433605 [ 1701]: Release freeze handler for prio 1
>2014/10/27 16:19:35.433651 [ 1701]: Thawing priority 2
>2014/10/27 16:19:35.433669 [ 1701]: Release freeze handler for prio 2
>2014/10/27 16:19:35.433695 [ 1701]: Thawing priority 3
>2014/10/27 16:19:35.433710 [ 1701]: Release freeze handler for prio 3
>2014/10/27 16:19:35.933309 [ 1701]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/27 16:19:57.391601 [ 1701]: Freeze priority 1
>2014/10/27 16:19:57.392859 [ 1701]: Freeze priority 2
>2014/10/27 16:19:57.393904 [ 1701]: Freeze priority 3
>2014/10/27 16:19:57.395119 [ 1701]: Monitoring event was cancelled
>2014/10/27 16:19:57.395162 [ 1701]: server/eventscript.c:569 Sending SIGTERM to child pid:9150
>2014/10/27 16:20:19.795485 [ 1701]: pnn 2 Invalid reqid 2224 in ctdb_reply_control
>2014/10/27 16:20:19.795614 [ 1701]: pnn 2 Invalid reqid 2238 in ctdb_reply_control
>2014/10/27 16:20:43.251237 [ 1701]: pnn 2 Invalid reqid 2248 in ctdb_reply_control
>2014/10/27 16:20:59.603323 [ 1701]: pnn 2 Invalid reqid 2268 in ctdb_reply_control
>2014/10/27 16:20:59.603442 [ 1701]: pnn 2 Invalid reqid 2295 in ctdb_reply_control
>2014/10/27 16:21:43.252883 [ 1701]: Recovery daemon ping timeout. Count : 0
>2014/10/27 16:21:57.395393 [ 1701]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS
>2014/10/27 16:21:58.099995 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:21:58.100135 [recoverd: 1929]: client/ctdb_client.c:1535 ctdb_control for getnodes failed ret:-1 res:-1
>2014/10/27 16:21:58.100154 [recoverd: 1929]: server/ctdb_recoverd.c:1183 Unable to get nodemap from remote node 1
>2014/10/27 16:21:58.100169 [recoverd: 1929]: Unable to update local flags
>2014/10/27 16:21:58.102467 [recoverd: 1929]: Taking out recovery lock from recovery daemon
>2014/10/27 16:21:58.102501 [recoverd: 1929]: Take the recovery lock
>2014/10/27 16:22:27.003298 [ 1701]: High RECLOCK latency 28.900705s for operation recd reclock
>2014/10/27 16:22:27.009836 [ 1701]: Freeze priority 1
>2014/10/27 16:22:27.010121 [ 1701]: Freeze priority 2
>2014/10/27 16:22:27.010331 [ 1701]: Freeze priority 3
>2014/10/27 16:22:27.041138 [ 1701]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1
>2014/10/27 16:22:58.100362 [ 1701]: Recovery daemon ping timeout. Count : 0
>2014/10/27 16:23:27.192843 [recoverd: 1929]: client/ctdb_client.c:1015 control timed out. reqid:1572 opcode:52 dstnode:1
>2014/10/27 16:23:27.192928 [recoverd: 1929]: client/ctdb_client.c:1126 ctdb_control_recv failed
>2014/10/27 16:23:27.192941 [recoverd: 1929]: Async operation failed with state 3, opcode:52
>2014/10/27 16:23:27.192958 [recoverd: 1929]: Async wait failed - fail_count=1
>2014/10/27 16:23:27.192968 [recoverd: 1929]: client/ctdb_client.c:3027 Unable to update nodeflags on remote nodes
>2014/10/27 16:23:27.192977 [recoverd: 1929]: server/ctdb_recoverd.c:873 Unable to update nodeflags on remote nodes
>2014/10/27 16:23:27.193000 [recoverd: 1929]: server/ctdb_recoverd.c:1857 Unable to update flags on all nodes for node 0
>2014/10/27 16:23:27.193039 [recoverd: 1929]: client/ctdb_client.c:962 reqid 1572 not found
>2014/10/27 16:23:33.109264 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:23:33.109308 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:23:33.109322 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 16:23:33.109331 [recoverd: 1929]: Async wait failed - fail_count=1
>2014/10/27 16:23:33.109340 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 16:23:33.109349 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 16:23:33.110597 [recoverd: 1929]: Taking out recovery lock from recovery daemon
>2014/10/27 16:23:33.110612 [recoverd: 1929]: Take the recovery lock
>2014/10/27 16:23:33.119358 [ 1701]: Freeze priority 1
>2014/10/27 16:23:33.119656 [ 1701]: Freeze priority 2
>2014/10/27 16:23:33.119911 [ 1701]: Freeze priority 3
>2014/10/27 16:23:34.385979 [ 1701]: Thawing priority 1
>2014/10/27 16:23:34.386033 [ 1701]: Release freeze handler for prio 1
>2014/10/27 16:23:34.386067 [ 1701]: Thawing priority 2
>2014/10/27 16:23:34.386096 [ 1701]: Release freeze handler for prio 2
>2014/10/27 16:23:34.386120 [ 1701]: Thawing priority 3
>2014/10/27 16:23:34.386134 [ 1701]: Release freeze handler for prio 3
>2014/10/27 16:23:34.394937 [recoverd: 1929]: Inconsistent IP allocation - node 0 thinks 10.10.10.183 is held by node 0 while it is assigned to node 2
>2014/10/27 16:23:34.394984 [recoverd: 1929]: Trigger IP reallocation
>2014/10/27 16:23:34.395239 [recoverd: 1929]: Inconsistent IP allocation - node 2 thinks 10.10.10.183 is held by node 0 while it is assigned to node 2
>2014/10/27 16:23:34.395268 [recoverd: 1929]: Trigger IP reallocation
>2014/10/27 16:23:34.690047 [ 1701]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/27 16:23:34.963319 [recoverd: 1929]: Resetting ban count to 0 for all nodes
>2014/10/27 16:23:58.004451 [ 1701]: High RECLOCK latency 1.151293s for operation recd reclock
>2014/10/27 16:24:43.116578 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:24:43.116674 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:24:43.116689 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 16:24:43.116698 [recoverd: 1929]: Async wait failed - fail_count=1
>2014/10/27 16:24:43.116707 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 16:24:43.116730 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 16:24:45.331407 [ 1701]: pnn 2 Invalid reqid 2979 in ctdb_reply_control
>2014/10/27 16:25:43.124079 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:25:43.124167 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:25:43.124183 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 16:25:43.124207 [recoverd: 1929]: Async wait failed - fail_count=1
>2014/10/27 16:25:43.124216 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 16:25:43.124227 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 16:25:45.875583 [ 1701]: pnn 2 Invalid reqid 3497 in ctdb_reply_control
>2014/10/27 16:25:46.427968 [ 1701]: pnn 2 Invalid reqid 3402 in ctdb_reply_control
>2014/10/27 16:25:53.998288 [ 1701]: Event script '00.ctdb monitor ' timed out after 60.0s, count: 0, pid: 20796
>2014/10/27 16:25:57.155173 [ 1701]: pnn 2 Invalid reqid 3761 in ctdb_reply_control
>2014/10/27 16:25:59.092489 [ 1701]: High RECLOCK latency 1.936301s for operation recd reclock
>2014/10/27 16:26:59.093872 [ 1701]: Recovery daemon ping timeout. Count : 0
>2014/10/27 16:27:08.999500 [ 1701]: Event script '10.interface monitor ' timed out after 23.1s, count: 1, pid: 25018
>2014/10/27 16:27:33.269145 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:27:33.269233 [recoverd: 1929]: client/ctdb_client.c:1535 ctdb_control for getnodes failed ret:-1 res:-1
>2014/10/27 16:27:33.269249 [recoverd: 1929]: server/ctdb_recoverd.c:1183 Unable to get nodemap from remote node 0
>2014/10/27 16:27:33.269264 [recoverd: 1929]: Unable to update local flags
>2014/10/27 16:27:33.272001 [recoverd: 1929]: Taking out recovery lock from recovery daemon
>2014/10/27 16:27:33.272032 [recoverd: 1929]: Take the recovery lock
>2014/10/27 16:28:09.000169 [ 1701]: ctdb_kill: trying to kill(25018, 9) a process that does not exist
>2014/10/27 16:28:09.267471 [ 1701]: pnn 2 Invalid reqid 4444 in ctdb_reply_control
>2014/10/27 16:28:24.000749 [ 1701]: Event script '00.ctdb monitor ' timed out after 60.0s, count: 2, pid: 26144
>2014/10/27 16:28:33.270336 [ 1701]: Recovery daemon ping timeout. Count : 0
>2014/10/27 16:29:24.001811 [ 1701]: ctdb_kill: trying to kill(26144, 9) a process that does not exist
>2014/10/27 16:29:27.827980 [recoverd: 1929]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile'
>2014/10/27 16:29:27.828038 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 16:29:27.828116 [ 1701]: Banning this node for 30 seconds
>2014/10/27 16:29:27.828153 [ 1701]: Freeze priority 1
>2014/10/27 16:29:27.828324 [ 1701]: Freeze priority 2
>2014/10/27 16:29:27.828451 [ 1701]: Freeze priority 3
>2014/10/27 16:29:27.828600 [ 1701]: Monitoring event was cancelled
>2014/10/27 16:29:27.828630 [ 1701]: server/eventscript.c:569 Sending SIGTERM to child pid:29275
>2014/10/27 16:29:33.270926 [ 1701]: Recovery daemon ping timeout. Count : 1
>2014/10/27 16:29:42.829357 [ 1701]: Skip monitoring since databases are frozen
>2014/10/27 16:29:57.828410 [ 1701]: Banning timedout
>2014/10/27 16:29:57.829484 [ 1701]: Skip monitoring since databases are frozen
>2014/10/27 16:30:12.830161 [ 1701]: Skip monitoring since databases are frozen
>2014/10/27 16:30:15.074050 [ 1701]: Freeze priority 1
>2014/10/27 16:30:15.075124 [ 1701]: Freeze priority 2
>2014/10/27 16:30:15.076194 [ 1701]: Freeze priority 3
>2014/10/27 16:30:18.091334 [ 1701]: Freeze priority 1
>2014/10/27 16:30:18.091621 [ 1701]: Freeze priority 2
>2014/10/27 16:30:18.091884 [ 1701]: Freeze priority 3
>2014/10/27 16:30:27.828367 [recoverd: 1929]: ctdb_control error: 'ctdb_control timed out'
>2014/10/27 16:30:27.828440 [recoverd: 1929]: client/ctdb_client.c:4643 ctdb_control for set ban state failed
>2014/10/27 16:30:27.828451 [recoverd: 1929]: server/ctdb_recoverd.c:176 Failed to ban node 2
>2014/10/27 16:30:27.829074 [ 1701]: Event script '00.ctdb releaseip bond1 10.10.10.184 24' timed out after 60.0s, count: 3, pid: 30855
>2014/10/27 16:30:27.829105 [ 1701]: Ignoring hung script for bond1 10.10.10.184 24 call 6
>2014/10/27 16:31:13.391289 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:31:13.391371 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:31:13.391386 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 16:31:13.391397 [recoverd: 1929]: Async wait failed - fail_count=1
>2014/10/27 16:31:13.391418 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 16:31:13.391428 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 16:31:18.092892 [ 1701]: Event script '00.ctdb startrecovery ' timed out after 60.0s, count: 3, pid: 32555
>2014/10/27 16:31:18.092986 [ 1701]: Ignoring hung script for  call 3
>2014/10/27 16:31:27.829584 [ 1701]: ctdb_kill: trying to kill(30855, 9) a process that does not exist
>2014/10/27 16:31:27.829722 [ 1701]: Event script '00.ctdb releaseip bond1 10.10.10.182 24' timed out after 60.0s, count: 3, pid: 369
>2014/10/27 16:31:27.829737 [ 1701]: Ignoring hung script for bond1 10.10.10.182 24 call 6
>2014/10/27 16:31:27.829895 [ 1701]: pnn 2 Invalid reqid 4948 in ctdb_reply_control
>2014/10/27 16:32:00.659414 [ 1701]: pnn 2 Invalid reqid 4069 in ctdb_reply_control
>2014/10/27 16:32:00.659482 [ 1701]: pnn 2 Invalid reqid 4075 in ctdb_reply_control
>2014/10/27 16:32:00.659509 [ 1701]: pnn 2 Invalid reqid 4129 in ctdb_reply_control
>2014/10/27 16:32:00.659521 [ 1701]: pnn 2 Invalid reqid 4189 in ctdb_reply_control
>2014/10/27 16:32:13.392539 [ 1701]: Recovery daemon ping timeout. Count : 0
>2014/10/27 16:32:13.392682 [recoverd: 1929]: ctdb_control error: 'ctdb_control timed out'
>2014/10/27 16:32:13.392722 [recoverd: 1929]: ctdb_control error: 'ctdb_control timed out'
>2014/10/27 16:32:13.392743 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 16:32:13.392759 [recoverd: 1929]: Async wait failed - fail_count=1
>2014/10/27 16:32:13.392784 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 16:32:13.392804 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 16:32:18.092956 [ 1701]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS
>2014/10/27 16:32:58.452549 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:32:58.452589 [ 1701]: tcp/tcp_connect.c:211Failed to bind socket Cannot assign requested address(99)
>2014/10/27 16:32:58.452623 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:32:58.452641 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 16:32:58.452652 [recoverd: 1929]: Async wait failed - fail_count=1
>2014/10/27 16:32:58.452663 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 16:32:58.452674 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 16:33:13.454963 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:33:13.454973 [ 1701]: tcp/tcp_connect.c:211Failed to bind socket Cannot assign requested address(99)
>2014/10/27 16:33:13.455031 [recoverd: 1929]: ctdb_control error: 'node is disconnected'
>2014/10/27 16:33:13.455048 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/27 16:33:13.455060 [recoverd: 1929]: Async wait failed - fail_count=1
>2014/10/27 16:33:13.455070 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/27 16:33:13.455081 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/27 16:33:13.456016 [ 1701]: Freeze priority 1
>2014/10/27 16:33:13.456174 [ 1701]: Freeze priority 2
>2014/10/27 16:33:13.456329 [ 1701]: Freeze priority 3
>2014/10/27 16:33:16.461003 [recoverd: 1929]: Taking out recovery lock from recovery daemon
>2014/10/27 16:33:16.461070 [recoverd: 1929]: Take the recovery lock
>2014/10/27 16:34:10.057443 [recoverd: 1929]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected)
>2014/10/27 16:34:10.057482 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 16:34:10.057582 [ 1701]: Banning this node for 30 seconds
>2014/10/27 16:34:40.058417 [ 1701]: Banning timedout
>2014/10/27 16:34:40.102905 [recoverd: 1929]: Taking out recovery lock from recovery daemon
>2014/10/27 16:34:40.102957 [recoverd: 1929]: Take the recovery lock
>2014/10/27 16:34:40.103175 [recoverd: 1929]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected)
>2014/10/27 16:34:40.103198 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 16:34:40.103228 [ 1701]: Banning this node for 30 seconds
>2014/10/27 16:35:10.103415 [ 1701]: Banning timedout
>2014/10/27 16:35:10.148526 [recoverd: 1929]: Taking out recovery lock from recovery daemon
>2014/10/27 16:35:10.148596 [recoverd: 1929]: Take the recovery lock
>2014/10/27 16:35:10.148995 [recoverd: 1929]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected)
>2014/10/27 16:35:10.149021 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 16:35:10.149069 [ 1701]: Banning this node for 30 seconds
>2014/10/27 16:35:13.457079 [ 1701]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS
>2014/10/27 16:35:40.149181 [ 1701]: Banning timedout
>2014/10/27 16:35:40.198703 [recoverd: 1929]: Taking out recovery lock from recovery daemon
>2014/10/27 16:35:40.198768 [recoverd: 1929]: Take the recovery lock
>2014/10/27 16:35:40.199153 [recoverd: 1929]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected)
>2014/10/27 16:35:40.199168 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/27 16:35:40.199199 [ 1701]: Banning this node for 30 seconds
>2014/10/27 16:35:55.380365 [ 1701]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/27 16:35:55.391833 [ 1701]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 11:42:50.796318 [ 1555]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/28 11:42:50.796425 [ 1555]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 11:42:50.796440 [ 1555]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 13:43:34.009786 [18275]: Starting CTDBD (Version 2.5.3) as PID: 18275
>2014/10/28 13:43:34.878010 [18275]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/28 13:43:34.895656 [18275]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/28 13:43:34.914100 [18275]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/28 13:43:34.931442 [18275]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/28 13:43:34.948996 [18275]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/28 13:43:34.966424 [18275]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/28 13:43:34.983027 [18275]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/28 13:43:34.983078 [18275]: Freeze priority 1
>2014/10/28 13:43:34.994663 [18275]: Freeze priority 2
>2014/10/28 13:43:34.995095 [18275]: Freeze priority 3
>2014/10/28 13:43:35.084334 [18275]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/28 13:43:35.088408 [18275]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/28 13:43:35.091910 [18275]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/28 13:43:35.219371 [18275]: Freeze priority 1
>2014/10/28 13:43:35.219451 [18275]: Freeze priority 2
>2014/10/28 13:43:35.219495 [18275]: Freeze priority 3
>2014/10/28 13:43:35.742105 [18275]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/28 13:43:35.808236 [18275]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/28 13:43:39.226456 [recoverd:18517]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/28 13:43:39.226592 [18275]: Freeze priority 1
>2014/10/28 13:43:39.226664 [18275]: Freeze priority 2
>2014/10/28 13:43:39.226723 [18275]: Freeze priority 3
>2014/10/28 13:43:43.175153 [18275]: Freeze priority 1
>2014/10/28 13:43:43.176249 [18275]: Freeze priority 2
>2014/10/28 13:43:43.177111 [18275]: Freeze priority 3
>2014/10/28 13:43:43.319778 [18275]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/28 13:43:43.320559 [18275]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/28 13:43:44.405392 [18275]: Thawing priority 1
>2014/10/28 13:43:44.405449 [18275]: Release freeze handler for prio 1
>2014/10/28 13:43:44.405483 [18275]: Thawing priority 2
>2014/10/28 13:43:44.405511 [18275]: Release freeze handler for prio 2
>2014/10/28 13:43:44.405534 [18275]: Thawing priority 3
>2014/10/28 13:43:44.405559 [18275]: Release freeze handler for prio 3
>2014/10/28 13:43:58.747418 [18275]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/28 13:43:59.012218 [18275]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 13:43:59.023752 [18275]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 13:43:59.045309 [18275]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/28 13:43:59.250982 [18275]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/28 13:43:59.256986 [recoverd:18517]: Trigger takeoverrun
>2014/10/28 13:44:01.726411 [18275]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/28 13:44:02.179387 [18275]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/28 15:07:35.793781 [18275]: Freeze priority 1
>2014/10/28 15:07:35.794785 [18275]: Freeze priority 2
>2014/10/28 15:07:35.795716 [18275]: Freeze priority 3
>2014/10/28 15:07:38.894209 [18275]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 15:07:38.905980 [18275]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 15:10:36.264082 [ 1560]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/28 15:10:36.264188 [ 1560]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 15:10:36.264204 [ 1560]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 15:14:37.877527 [ 8024]: Starting CTDBD (Version 2.5.3) as PID: 8024
>2014/10/28 15:14:39.319788 [ 8024]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/28 15:14:39.337086 [ 8024]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/28 15:14:39.355193 [ 8024]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/28 15:14:39.372606 [ 8024]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/28 15:14:39.390014 [ 8024]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/28 15:14:39.406603 [ 8024]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/28 15:14:39.422159 [ 8024]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/28 15:14:39.422200 [ 8024]: Freeze priority 1
>2014/10/28 15:14:39.437229 [ 8024]: Freeze priority 2
>2014/10/28 15:14:39.437799 [ 8024]: Freeze priority 3
>2014/10/28 15:14:39.534743 [ 8024]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/28 15:14:39.538506 [ 8024]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/28 15:14:39.544025 [ 8024]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/28 15:14:39.664832 [ 8024]: Freeze priority 1
>2014/10/28 15:14:39.664919 [ 8024]: Freeze priority 2
>2014/10/28 15:14:39.664973 [ 8024]: Freeze priority 3
>2014/10/28 15:14:43.170726 [recoverd: 8268]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/28 15:14:43.170817 [ 8024]: Freeze priority 1
>2014/10/28 15:14:43.170881 [ 8024]: Freeze priority 2
>2014/10/28 15:14:43.170936 [ 8024]: Freeze priority 3
>2014/10/28 15:14:46.839971 [ 8024]: Freeze priority 1
>2014/10/28 15:14:46.841126 [ 8024]: Freeze priority 2
>2014/10/28 15:14:46.841848 [ 8024]: Freeze priority 3
>2014/10/28 15:14:50.931074 [ 8024]: Thawing priority 1
>2014/10/28 15:14:50.931113 [ 8024]: Release freeze handler for prio 1
>2014/10/28 15:14:50.931143 [ 8024]: Thawing priority 2
>2014/10/28 15:14:50.931164 [ 8024]: Release freeze handler for prio 2
>2014/10/28 15:14:50.931191 [ 8024]: Thawing priority 3
>2014/10/28 15:14:50.931210 [ 8024]: Release freeze handler for prio 3
>2014/10/28 15:14:55.089867 [ 8024]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/28 15:15:01.293444 [ 8024]: Freeze priority 1
>2014/10/28 15:15:01.294714 [ 8024]: Freeze priority 2
>2014/10/28 15:15:01.295674 [ 8024]: Freeze priority 3
>2014/10/28 15:15:04.469018 [ 8024]: Thawing priority 1
>2014/10/28 15:15:04.469065 [ 8024]: Release freeze handler for prio 1
>2014/10/28 15:15:04.469096 [ 8024]: Thawing priority 2
>2014/10/28 15:15:04.469116 [ 8024]: Release freeze handler for prio 2
>2014/10/28 15:15:04.469147 [ 8024]: Thawing priority 3
>2014/10/28 15:15:04.469166 [ 8024]: Release freeze handler for prio 3
>2014/10/28 15:15:19.493509 [recoverd: 8268]: Trigger takeoverrun
>2014/10/28 15:15:19.668661 [ 8024]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/28 15:15:20.159380 [ 8024]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 15:15:20.174343 [ 8024]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 15:15:20.207130 [ 8024]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/28 15:15:20.380708 [ 8024]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/28 15:15:23.955123 [ 8024]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/28 15:15:24.451979 [ 8024]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/28 15:15:35.457606 [ 8024]: Freeze priority 1
>2014/10/28 15:15:35.458734 [ 8024]: Freeze priority 2
>2014/10/28 15:15:35.459571 [ 8024]: Freeze priority 3
>2014/10/28 15:15:39.595930 [ 8024]: Thawing priority 1
>2014/10/28 15:15:39.595970 [ 8024]: Release freeze handler for prio 1
>2014/10/28 15:15:39.596001 [ 8024]: Thawing priority 2
>2014/10/28 15:15:39.596020 [ 8024]: Release freeze handler for prio 2
>2014/10/28 15:15:39.596049 [ 8024]: Thawing priority 3
>2014/10/28 15:15:39.596067 [ 8024]: Release freeze handler for prio 3
>2014/10/28 15:20:23.960958 [ 8024]: Freeze priority 1
>2014/10/28 15:20:23.965481 [ 8024]: Freeze priority 1
>2014/10/28 15:20:29.341908 [ 8024]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=13440 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=13440 =====
>===== Start of debug locks PID=14593 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>2014/10/28 15:20:44.342724 [ 8024]: Skip monitoring since databases are frozen
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=14593 =====
>===== Start of debug locks PID=15012 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=15012 =====
>2014/10/28 15:20:59.343680 [ 8024]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=15456 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=15456 =====
>===== Start of debug locks PID=15948 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>2014/10/28 15:21:14.344067 [ 8024]: Skip monitoring since databases are frozen
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=15948 =====
>2014/10/28 15:21:23.964033 [ 8024]: Freeze priority 1
>2014/10/28 15:21:23.964085 [ 8024]: Recovery daemon ping timeout. Count : 0
>2014/10/28 15:21:23.966262 [recoverd: 8268]: ctdb_control error: 'ctdb_control timed out'
>2014/10/28 15:21:23.966310 [recoverd: 8268]: ctdb_control error: 'ctdb_control timed out'
>2014/10/28 15:21:23.966349 [recoverd: 8268]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/28 15:21:23.966367 [recoverd: 8268]: Failed to freeze node 1 during recovery. Set it as ban culprit for 3 credits
>2014/10/28 15:21:23.966388 [recoverd: 8268]: ctdb_control error: 'ctdb_control timed out'
>2014/10/28 15:21:23.966401 [recoverd: 8268]: ctdb_control error: 'ctdb_control timed out'
>2014/10/28 15:21:23.966417 [recoverd: 8268]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/28 15:21:23.966429 [recoverd: 8268]: Failed to freeze node 2 during recovery. Set it as ban culprit for 3 credits
>2014/10/28 15:21:23.966444 [recoverd: 8268]: Async wait failed - fail_count=2
>2014/10/28 15:21:23.966457 [recoverd: 8268]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/28 15:21:23.966472 [recoverd: 8268]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/28 15:21:23.968027 [ 8024]: Freeze priority 1
>===== Start of debug locks PID=16375 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=16375 =====
>2014/10/28 15:21:29.344420 [ 8024]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=16809 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=16809 =====
>===== Start of debug locks PID=17179 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>2014/10/28 15:21:44.345209 [ 8024]: Skip monitoring since databases are frozen
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=17179 =====
>===== Start of debug locks PID=17435 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=17435 =====
>2014/10/28 15:21:59.345909 [ 8024]: Skip monitoring since databases are frozen
>2014/10/28 15:22:00.150854 [ 8024]: pnn 2 Invalid reqid 293987 in ctdb_reply_control
>===== Start of debug locks PID=17694 =====
>13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W
>12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124
>13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329
>13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=12416 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13064 -----
>#0  0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f40493041b4 in smbd_accept_connection ()
>#25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f4049300d01 in main ()
>----- Stack trace for PID=13069 -----
>#0  0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541
>#6  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537
>#9  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537
>#11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=17694 =====
>2014/10/28 15:22:12.191506 [ 8024]: pnn 2 Invalid reqid 293988 in ctdb_reply_control
>2014/10/28 15:22:12.191593 [ 8024]: Freeze priority 2
>2014/10/28 15:22:12.191600 [recoverd: 8268]: ctdb_control error: 'ctdb_control to disconnected node'
>2014/10/28 15:22:12.191658 [recoverd: 8268]: ctdb_control error: 'ctdb_control to disconnected node'
>2014/10/28 15:22:12.191687 [recoverd: 8268]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/28 15:22:12.191709 [recoverd: 8268]: Failed to freeze node 1 during recovery. Set it as ban culprit for 3 credits
>2014/10/28 15:22:12.192248 [recoverd: 8268]: Async wait failed - fail_count=1
>2014/10/28 15:22:12.192277 [recoverd: 8268]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/28 15:22:12.192297 [recoverd: 8268]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/28 15:22:12.192847 [recoverd: 8268]: ctdb_control error: 'ctdb_control to disconnected node'
>2014/10/28 15:22:12.192880 [recoverd: 8268]: client/ctdb_client.c:4643 ctdb_control for set ban state failed
>2014/10/28 15:22:12.192892 [recoverd: 8268]: server/ctdb_recoverd.c:176 Failed to ban node 1
>2014/10/28 15:22:12.193011 [ 8024]: Freeze priority 1
>2014/10/28 15:22:12.193081 [ 8024]: Freeze priority 2
>2014/10/28 15:22:12.193155 [ 8024]: Freeze priority 3
>2014/10/28 15:22:15.197554 [recoverd: 8268]: Taking out recovery lock from recovery daemon
>2014/10/28 15:22:15.197587 [recoverd: 8268]: Take the recovery lock
>2014/10/28 15:22:15.199791 [ 8024]: Freeze priority 1
>2014/10/28 15:22:15.199846 [ 8024]: Freeze priority 2
>2014/10/28 15:22:15.199886 [ 8024]: Freeze priority 3
>2014/10/28 15:22:15.599021 [ 8024]: 60.nfs: /etc/ctdb/functions: line 37: /etc/sysconfig/ctdb: Software caused connection abort
>2014/10/28 15:22:16.134484 [ 8024]: 00.ctdb: /etc/ctdb/functions: line 37: /etc/sysconfig/ctdb: Software caused connection abort
>2014/10/28 15:22:16.134629 [ 8024]: server/ctdb_recover.c:971 startrecovery event script failed (status 1)
>2014/10/28 15:22:16.134712 [ 8024]: pnn 2 Invalid reqid 294481 in ctdb_reply_control
>2014/10/28 15:22:16.262335 [ 8024]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 15:22:16.273200 [ 8024]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 15:29:17.403745 [ 1545]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/28 15:29:17.403897 [ 1545]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 15:29:17.403911 [ 1545]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 15:33:18.798488 [ 7997]: Starting CTDBD (Version 2.5.3) as PID: 7997
>2014/10/28 15:33:19.888102 [ 7997]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/28 15:33:19.902848 [ 7997]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/28 15:33:19.917427 [ 7997]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/28 15:33:19.931290 [ 7997]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/28 15:33:19.946036 [ 7997]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/28 15:33:19.960934 [ 7997]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/28 15:33:19.976015 [ 7997]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/28 15:33:19.976155 [ 7997]: Freeze priority 1
>2014/10/28 15:33:19.991610 [ 7997]: Freeze priority 2
>2014/10/28 15:33:19.991981 [ 7997]: Freeze priority 3
>2014/10/28 15:33:20.080995 [ 7997]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/28 15:33:20.084774 [ 7997]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/28 15:33:20.088310 [ 7997]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/28 15:33:20.217395 [ 7997]: Freeze priority 1
>2014/10/28 15:33:20.217476 [ 7997]: Freeze priority 2
>2014/10/28 15:33:20.217530 [ 7997]: Freeze priority 3
>2014/10/28 15:33:23.803848 [recoverd: 8280]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/28 15:33:23.803931 [ 7997]: Freeze priority 1
>2014/10/28 15:33:23.803991 [ 7997]: Freeze priority 2
>2014/10/28 15:33:23.804043 [ 7997]: Freeze priority 3
>2014/10/28 15:33:27.057437 [ 7997]: Freeze priority 1
>2014/10/28 15:33:27.058229 [ 7997]: Freeze priority 2
>2014/10/28 15:33:27.058763 [ 7997]: Freeze priority 3
>2014/10/28 15:33:27.200565 [ 7997]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/28 15:33:32.275443 [ 7997]: Handling event took 4 seconds!
>2014/10/28 15:33:32.278537 [ 7997]: Thawing priority 1
>2014/10/28 15:33:32.278614 [ 7997]: Release freeze handler for prio 1
>2014/10/28 15:33:32.278652 [ 7997]: Thawing priority 2
>2014/10/28 15:33:32.278673 [ 7997]: Release freeze handler for prio 2
>2014/10/28 15:33:32.278700 [ 7997]: Thawing priority 3
>2014/10/28 15:33:32.278720 [ 7997]: Release freeze handler for prio 3
>2014/10/28 15:33:47.282595 [ 7997]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/28 15:33:47.301208 [recoverd: 8280]: Trigger takeoverrun
>2014/10/28 15:33:48.048369 [ 7997]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 15:33:48.061804 [ 7997]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 15:33:48.090275 [ 7997]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/28 15:33:48.278527 [ 7997]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/28 15:33:50.717806 [ 7997]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/28 15:33:52.156153 [ 7997]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/28 15:34:37.729290 [ 7997]: Freeze priority 1
>2014/10/28 15:34:37.730337 [ 7997]: Freeze priority 2
>2014/10/28 15:34:37.731151 [ 7997]: Freeze priority 3
>2014/10/28 15:34:38.982118 [ 7997]: Thawing priority 1
>2014/10/28 15:34:38.982171 [ 7997]: Release freeze handler for prio 1
>2014/10/28 15:34:38.982213 [ 7997]: Thawing priority 2
>2014/10/28 15:34:38.982234 [ 7997]: Release freeze handler for prio 2
>2014/10/28 15:34:38.982269 [ 7997]: Thawing priority 3
>2014/10/28 15:34:38.982288 [ 7997]: Release freeze handler for prio 3
>2014/10/28 15:55:31.160893 [ 7997]: Freeze priority 1
>2014/10/28 15:55:31.162055 [ 7997]: Freeze priority 2
>2014/10/28 15:55:31.163045 [ 7997]: Freeze priority 3
>2014/10/28 15:55:34.186193 [recoverd: 8280]: Taking out recovery lock from recovery daemon
>2014/10/28 15:55:34.186227 [recoverd: 8280]: Take the recovery lock
>2014/10/28 15:55:34.189109 [ 7997]: Freeze priority 1
>2014/10/28 15:55:34.189170 [ 7997]: Freeze priority 2
>2014/10/28 15:55:34.189213 [ 7997]: Freeze priority 3
>2014/10/28 15:55:36.499864 [ 7997]: Thawing priority 1
>2014/10/28 15:55:36.499904 [ 7997]: Release freeze handler for prio 1
>2014/10/28 15:55:36.499940 [ 7997]: Thawing priority 2
>2014/10/28 15:55:36.499961 [ 7997]: Release freeze handler for prio 2
>2014/10/28 15:55:36.499996 [ 7997]: Thawing priority 3
>2014/10/28 15:55:36.500015 [ 7997]: Release freeze handler for prio 3
>2014/10/28 15:55:36.500991 [ 7997]: server/ctdb_call.c:1005 reqid 1253773 not found
>2014/10/28 15:55:36.501025 [ 7997]: server/ctdb_call.c:1005 reqid 1253774 not found
>2014/10/28 15:55:36.848367 [ 7997]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/28 15:55:37.023172 [recoverd: 8280]: Resetting ban count to 0 for all nodes
>2014/10/28 15:55:47.875052 [ 7997]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 15:55:47.885594 [ 7997]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 15:58:38.248954 [ 1546]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/28 15:58:38.249114 [ 1546]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 15:58:38.249134 [ 1546]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 16:02:39.394901 [ 8039]: Starting CTDBD (Version 2.5.3) as PID: 8039
>2014/10/28 16:02:40.284154 [ 8039]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/28 16:02:40.297936 [ 8039]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/28 16:02:40.312620 [ 8039]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/28 16:02:40.326540 [ 8039]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/28 16:02:40.340425 [ 8039]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/28 16:02:40.354386 [ 8039]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/28 16:02:40.368231 [ 8039]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/28 16:02:40.368260 [ 8039]: Freeze priority 1
>2014/10/28 16:02:40.379777 [ 8039]: Freeze priority 2
>2014/10/28 16:02:40.380139 [ 8039]: Freeze priority 3
>2014/10/28 16:02:40.469492 [ 8039]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/28 16:02:40.473611 [ 8039]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/28 16:02:40.477296 [ 8039]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/28 16:02:40.597178 [ 8039]: Freeze priority 1
>2014/10/28 16:02:40.597253 [ 8039]: Freeze priority 2
>2014/10/28 16:02:40.597307 [ 8039]: Freeze priority 3
>2014/10/28 16:02:44.102517 [recoverd: 8274]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/28 16:02:44.102611 [ 8039]: Freeze priority 1
>2014/10/28 16:02:44.102677 [ 8039]: Freeze priority 2
>2014/10/28 16:02:44.102738 [ 8039]: Freeze priority 3
>2014/10/28 16:02:47.460062 [ 8039]: Freeze priority 1
>2014/10/28 16:02:47.460793 [ 8039]: Freeze priority 2
>2014/10/28 16:02:47.461545 [ 8039]: Freeze priority 3
>2014/10/28 16:02:47.598360 [ 8039]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/28 16:02:49.141359 [ 8039]: Thawing priority 1
>2014/10/28 16:02:49.141406 [ 8039]: Release freeze handler for prio 1
>2014/10/28 16:02:49.141436 [ 8039]: Thawing priority 2
>2014/10/28 16:02:49.141464 [ 8039]: Release freeze handler for prio 2
>2014/10/28 16:02:49.141488 [ 8039]: Thawing priority 3
>2014/10/28 16:02:49.141504 [ 8039]: Release freeze handler for prio 3
>2014/10/28 16:03:03.627551 [recoverd: 8274]: Trigger takeoverrun
>2014/10/28 16:03:04.038618 [ 8039]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/28 16:03:04.379270 [ 8039]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 16:03:04.390200 [ 8039]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 16:03:04.413027 [ 8039]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/28 16:03:04.601594 [ 8039]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/28 16:03:07.054322 [ 8039]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/28 16:03:07.900699 [ 8039]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/28 16:03:30.492139 [ 8039]: Freeze priority 1
>2014/10/28 16:03:30.493255 [ 8039]: Freeze priority 2
>2014/10/28 16:03:30.494283 [ 8039]: Freeze priority 3
>2014/10/28 16:03:32.412399 [ 8039]: Thawing priority 1
>2014/10/28 16:03:32.412454 [ 8039]: Release freeze handler for prio 1
>2014/10/28 16:03:32.412487 [ 8039]: Thawing priority 2
>2014/10/28 16:03:32.412515 [ 8039]: Release freeze handler for prio 2
>2014/10/28 16:03:32.412540 [ 8039]: Thawing priority 3
>2014/10/28 16:03:32.412555 [ 8039]: Release freeze handler for prio 3
>2014/10/28 17:08:24.484388 [ 8039]: Freeze priority 1
>2014/10/28 17:08:24.494222 [ 8039]: Freeze priority 1
>2014/10/28 17:08:39.119651 [ 8039]: Skip monitoring since databases are frozen
>2014/10/28 17:08:54.120233 [ 8039]: Skip monitoring since databases are frozen
>2014/10/28 17:09:09.120357 [ 8039]: Skip monitoring since databases are frozen
>2014/10/28 17:09:24.120973 [ 8039]: Skip monitoring since databases are frozen
>2014/10/28 17:09:24.487003 [ 8039]: Freeze priority 1
>2014/10/28 17:09:24.493148 [ 8039]: Recovery daemon ping timeout. Count : 0
>2014/10/28 17:09:24.495347 [recoverd: 8274]: ctdb_control error: 'ctdb_control timed out'
>2014/10/28 17:09:24.495393 [recoverd: 8274]: ctdb_control error: 'ctdb_control timed out'
>2014/10/28 17:09:24.495417 [recoverd: 8274]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/28 17:09:24.495433 [recoverd: 8274]: Failed to freeze node 1 during recovery. Set it as ban culprit for 3 credits
>2014/10/28 17:09:24.495450 [recoverd: 8274]: Async wait failed - fail_count=1
>2014/10/28 17:09:24.495465 [recoverd: 8274]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/28 17:09:24.495480 [recoverd: 8274]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/28 17:09:24.497084 [ 8039]: Freeze priority 1
>2014/10/28 17:09:39.121758 [ 8039]: Skip monitoring since databases are frozen
>2014/10/28 17:09:54.122453 [ 8039]: Skip monitoring since databases are frozen
>2014/10/28 17:09:54.578568 [ 8039]: pnn 2 Invalid reqid 2929975 in ctdb_reply_control
>2014/10/28 17:09:54.578688 [ 8039]: Freeze priority 2
>2014/10/28 17:09:54.578900 [ 8039]: Freeze priority 2
>2014/10/28 17:09:54.579710 [ 8039]: Freeze priority 3
>2014/10/28 17:09:54.579978 [ 8039]: Freeze priority 3
>2014/10/28 17:09:57.586366 [recoverd: 8274]: Taking out recovery lock from recovery daemon
>2014/10/28 17:09:57.586420 [recoverd: 8274]: Take the recovery lock
>2014/10/28 17:09:57.590009 [ 8039]: Freeze priority 1
>2014/10/28 17:09:57.590081 [ 8039]: Freeze priority 2
>2014/10/28 17:09:57.590148 [ 8039]: Freeze priority 3
>2014/10/28 17:09:59.683656 [ 8039]: Thawing priority 1
>2014/10/28 17:09:59.683701 [ 8039]: Release freeze handler for prio 1
>2014/10/28 17:09:59.683729 [ 8039]: Thawing priority 2
>2014/10/28 17:09:59.683746 [ 8039]: Release freeze handler for prio 2
>2014/10/28 17:09:59.683771 [ 8039]: Thawing priority 3
>2014/10/28 17:09:59.683794 [ 8039]: Release freeze handler for prio 3
>2014/10/28 17:09:59.895066 [ 8039]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/28 17:09:59.895186 [ 8039]: pnn 2 Invalid reqid 2930556 in ctdb_reply_control
>2014/10/28 17:09:59.914099 [ 8039]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/28 17:09:59.914154 [ 8039]: pnn 2 Invalid reqid 2930554 in ctdb_reply_control
>2014/10/28 17:10:00.428347 [ 8039]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 17:10:00.438986 [ 8039]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 17:17:03.248575 [ 1546]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/28 17:17:03.248671 [ 1546]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 17:17:03.248685 [ 1546]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 17:21:05.044759 [ 8316]: Starting CTDBD (Version 2.5.3) as PID: 8316
>2014/10/28 17:21:05.942401 [ 8316]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/28 17:21:05.956168 [ 8316]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/28 17:21:05.970671 [ 8316]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/28 17:21:05.984520 [ 8316]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/28 17:21:05.998347 [ 8316]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/28 17:21:06.012493 [ 8316]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/28 17:21:06.026478 [ 8316]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/28 17:21:06.026510 [ 8316]: Freeze priority 1
>2014/10/28 17:21:06.038113 [ 8316]: Freeze priority 2
>2014/10/28 17:21:06.038456 [ 8316]: Freeze priority 3
>2014/10/28 17:21:06.127527 [ 8316]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/28 17:21:06.131659 [ 8316]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/28 17:21:06.135335 [ 8316]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/28 17:21:06.262555 [ 8316]: Freeze priority 1
>2014/10/28 17:21:06.262639 [ 8316]: Freeze priority 2
>2014/10/28 17:21:06.262693 [ 8316]: Freeze priority 3
>2014/10/28 17:21:09.767515 [recoverd: 8583]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/28 17:21:09.767619 [ 8316]: Freeze priority 1
>2014/10/28 17:21:09.767690 [ 8316]: Freeze priority 2
>2014/10/28 17:21:09.767751 [ 8316]: Freeze priority 3
>2014/10/28 17:21:13.719252 [ 8316]: Freeze priority 1
>2014/10/28 17:21:13.719969 [ 8316]: Freeze priority 2
>2014/10/28 17:21:13.720494 [ 8316]: Freeze priority 3
>2014/10/28 17:21:13.863229 [ 8316]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/28 17:21:19.116982 [ 8316]: Handling event took 5 seconds!
>2014/10/28 17:21:19.118635 [ 8316]: Thawing priority 1
>2014/10/28 17:21:19.118677 [ 8316]: Release freeze handler for prio 1
>2014/10/28 17:21:19.118707 [ 8316]: Thawing priority 2
>2014/10/28 17:21:19.118726 [ 8316]: Release freeze handler for prio 2
>2014/10/28 17:21:19.118752 [ 8316]: Thawing priority 3
>2014/10/28 17:21:19.118770 [ 8316]: Release freeze handler for prio 3
>2014/10/28 17:21:33.437609 [ 8316]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/28 17:21:33.759486 [ 8316]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 17:21:33.770638 [ 8316]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 17:21:33.809196 [ 8316]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/28 17:21:33.960678 [ 8316]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/28 17:21:34.140581 [recoverd: 8583]: Trigger takeoverrun
>2014/10/28 17:21:36.422132 [ 8316]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/28 17:21:37.260465 [ 8316]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/28 17:22:25.861996 [ 8316]: Freeze priority 1
>2014/10/28 17:22:25.862907 [ 8316]: Freeze priority 2
>2014/10/28 17:22:25.863695 [ 8316]: Freeze priority 3
>2014/10/28 17:22:27.310243 [ 8316]: Thawing priority 1
>2014/10/28 17:22:27.310288 [ 8316]: Release freeze handler for prio 1
>2014/10/28 17:22:27.310327 [ 8316]: Thawing priority 2
>2014/10/28 17:22:27.310354 [ 8316]: Release freeze handler for prio 2
>2014/10/28 17:22:27.310401 [ 8316]: Thawing priority 3
>2014/10/28 17:22:27.310420 [ 8316]: Release freeze handler for prio 3
>2014/10/28 17:22:37.814884 [ 8316]: Freeze priority 1
>2014/10/28 17:22:37.815985 [ 8316]: Freeze priority 2
>2014/10/28 17:22:37.816997 [ 8316]: Freeze priority 3
>2014/10/28 17:22:39.114387 [ 8316]: Thawing priority 1
>2014/10/28 17:22:39.114435 [ 8316]: Release freeze handler for prio 1
>2014/10/28 17:22:39.114478 [ 8316]: Thawing priority 2
>2014/10/28 17:22:39.114501 [ 8316]: Release freeze handler for prio 2
>2014/10/28 17:22:39.114540 [ 8316]: Thawing priority 3
>2014/10/28 17:22:39.114564 [ 8316]: Release freeze handler for prio 3
>2014/10/28 17:28:55.784892 [ 8316]: Freeze priority 1
>2014/10/28 17:28:56.704515 [ 8316]: Freeze priority 1
>2014/10/28 17:29:03.997358 [ 8316]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=5989 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=5989 =====
>===== Start of debug locks PID=6398 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=6398 =====
>2014/10/28 17:29:18.997981 [ 8316]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=6760 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=6760 =====
>2014/10/28 17:29:33.998876 [ 8316]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=7107 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=7107 =====
>===== Start of debug locks PID=7540 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=7540 =====
>2014/10/28 17:29:48.999822 [ 8316]: Skip monitoring since databases are frozen
>2014/10/28 17:29:55.787913 [ 8316]: Freeze priority 1
>===== Start of debug locks PID=7922 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=7922 =====
>2014/10/28 17:29:56.703260 [ 8316]: Recovery daemon ping timeout. Count : 0
>2014/10/28 17:29:56.705448 [recoverd: 8583]: ctdb_control error: 'ctdb_control timed out'
>2014/10/28 17:29:56.705497 [recoverd: 8583]: ctdb_control error: 'ctdb_control timed out'
>2014/10/28 17:29:56.705516 [recoverd: 8583]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/28 17:29:56.705529 [recoverd: 8583]: Failed to freeze node 2 during recovery. Set it as ban culprit for 3 credits
>2014/10/28 17:29:56.705544 [recoverd: 8583]: Async wait failed - fail_count=1
>2014/10/28 17:29:56.705574 [recoverd: 8583]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/28 17:29:56.705589 [recoverd: 8583]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/28 17:29:56.707222 [ 8316]: Freeze priority 1
>2014/10/28 17:30:03.999970 [ 8316]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=8331 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=8331 =====
>===== Start of debug locks PID=8699 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=8699 =====
>2014/10/28 17:30:19.000450 [ 8316]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=9081 =====
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347
>5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348
>5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W
>----- Stack trace for PID=5255 -----
>#0  0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so
>#12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so
>#13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0
>#21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#24 0x00007f701a38e1b4 in smbd_accept_connection ()
>#25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f701a38ad01 in main ()
>----- Stack trace for PID=5256 -----
>#0  0x00007fef801f6094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541
>#16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=9081 =====
>2014/10/28 17:30:30.275173 [ 8316]: pnn 2 Invalid reqid 431614 in ctdb_reply_control
>2014/10/28 17:30:30.275591 [ 8316]: Freeze priority 2
>2014/10/28 17:30:30.275595 [recoverd: 8583]: ctdb_control error: 'ctdb_control to disconnected node'
>2014/10/28 17:30:30.275634 [recoverd: 8583]: ctdb_control error: 'ctdb_control to disconnected node'
>2014/10/28 17:30:30.275660 [recoverd: 8583]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/28 17:30:30.275679 [recoverd: 8583]: Failed to freeze node 1 during recovery. Set it as ban culprit for 3 credits
>2014/10/28 17:30:30.276313 [recoverd: 8583]: Async wait failed - fail_count=1
>2014/10/28 17:30:30.276346 [recoverd: 8583]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/28 17:30:30.276359 [recoverd: 8583]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/28 17:30:30.277177 [ 8316]: Freeze priority 1
>2014/10/28 17:30:30.277245 [ 8316]: Freeze priority 2
>2014/10/28 17:30:30.277297 [ 8316]: Freeze priority 3
>2014/10/28 17:30:32.040647 [ 8316]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 17:30:32.051958 [ 8316]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 17:37:34.604030 [ 1520]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/28 17:37:34.604130 [ 1520]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 17:37:34.604147 [ 1520]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/28 17:41:33.996789 [ 7969]: Starting CTDBD (Version 2.5.3) as PID: 7969
>2014/10/28 17:41:35.055980 [ 7969]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/28 17:41:35.071983 [ 7969]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/28 17:41:35.087387 [ 7969]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/28 17:41:35.101489 [ 7969]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/28 17:41:35.116062 [ 7969]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/28 17:41:35.130037 [ 7969]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/28 17:41:35.144014 [ 7969]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/28 17:41:35.144046 [ 7969]: Freeze priority 1
>2014/10/28 17:41:35.277767 [ 7969]: Freeze priority 2
>2014/10/28 17:41:35.278214 [ 7969]: Freeze priority 3
>2014/10/28 17:41:35.293204 [ 7969]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/28 17:41:35.296994 [ 7969]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/28 17:41:35.300297 [ 7969]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/28 17:41:35.421222 [ 7969]: Freeze priority 1
>2014/10/28 17:41:35.421297 [ 7969]: Freeze priority 2
>2014/10/28 17:41:35.421350 [ 7969]: Freeze priority 3
>2014/10/28 17:41:38.696426 [ 7969]: Freeze priority 1
>2014/10/28 17:41:38.697151 [ 7969]: Freeze priority 2
>2014/10/28 17:41:38.697707 [ 7969]: Freeze priority 3
>2014/10/28 17:41:38.923315 [recoverd: 8207]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/28 17:41:38.923415 [ 7969]: Freeze priority 1
>2014/10/28 17:41:38.923487 [ 7969]: Freeze priority 2
>2014/10/28 17:41:38.923551 [ 7969]: Freeze priority 3
>2014/10/28 17:41:43.342809 [ 7969]: Handling event took 4 seconds!
>2014/10/28 17:41:43.345296 [ 7969]: Thawing priority 1
>2014/10/28 17:41:43.345324 [ 7969]: Release freeze handler for prio 1
>2014/10/28 17:41:43.345349 [ 7969]: Thawing priority 2
>2014/10/28 17:41:43.345376 [ 7969]: Release freeze handler for prio 2
>2014/10/28 17:41:43.345398 [ 7969]: Thawing priority 3
>2014/10/28 17:41:43.345413 [ 7969]: Release freeze handler for prio 3
>2014/10/28 17:41:46.346931 [recoverd: 8207]: Trigger takeoverrun
>2014/10/28 17:41:53.943869 [ 7969]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/28 17:41:53.950835 [ 7969]: Freeze priority 1
>2014/10/28 17:41:53.951905 [ 7969]: Freeze priority 2
>2014/10/28 17:41:53.952769 [ 7969]: Freeze priority 3
>2014/10/28 17:41:56.524977 [ 7969]: Thawing priority 1
>2014/10/28 17:41:56.525033 [ 7969]: Release freeze handler for prio 1
>2014/10/28 17:41:56.525076 [ 7969]: Thawing priority 2
>2014/10/28 17:41:56.525098 [ 7969]: Release freeze handler for prio 2
>2014/10/28 17:41:56.525141 [ 7969]: Thawing priority 3
>2014/10/28 17:41:56.525161 [ 7969]: Release freeze handler for prio 3
>2014/10/28 17:42:11.545493 [recoverd: 8207]: Trigger takeoverrun
>2014/10/28 17:42:12.439831 [ 7969]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/28 17:42:12.838121 [ 7969]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/28 17:42:12.851341 [ 7969]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/28 17:42:12.881606 [ 7969]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/28 17:42:13.101211 [ 7969]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/28 17:42:15.606644 [ 7969]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/28 17:42:16.423117 [ 7969]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/28 17:43:00.998137 [ 7969]: Freeze priority 1
>2014/10/28 17:43:00.999122 [ 7969]: Freeze priority 2
>2014/10/28 17:43:00.999922 [ 7969]: Freeze priority 3
>2014/10/28 17:43:05.301283 [ 7969]: Thawing priority 1
>2014/10/28 17:43:05.301323 [ 7969]: Release freeze handler for prio 1
>2014/10/28 17:43:05.301356 [ 7969]: Thawing priority 2
>2014/10/28 17:43:05.301388 [ 7969]: Release freeze handler for prio 2
>2014/10/28 17:43:05.301417 [ 7969]: Thawing priority 3
>2014/10/28 17:43:05.301437 [ 7969]: Release freeze handler for prio 3
>2014/10/28 17:43:15.821487 [ 7969]: Freeze priority 1
>2014/10/28 17:43:15.822320 [ 7969]: Freeze priority 2
>2014/10/28 17:43:15.823048 [ 7969]: Freeze priority 3
>2014/10/28 17:43:17.254028 [ 7969]: Thawing priority 1
>2014/10/28 17:43:17.254081 [ 7969]: Release freeze handler for prio 1
>2014/10/28 17:43:17.254128 [ 7969]: Thawing priority 2
>2014/10/28 17:43:17.254152 [ 7969]: Release freeze handler for prio 2
>2014/10/28 17:43:17.254203 [ 7969]: Thawing priority 3
>2014/10/28 17:43:17.254224 [ 7969]: Release freeze handler for prio 3
>2014/10/28 17:46:28.001501 [ 7969]: No event for 19 seconds!
>2014/10/29 09:05:41.508845 [ 7969]: Freeze priority 1
>2014/10/29 09:05:41.510285 [ 7969]: Freeze priority 2
>2014/10/29 09:05:41.511314 [ 7969]: Freeze priority 3
>2014/10/29 09:05:44.663963 [recoverd: 8207]: Taking out recovery lock from recovery daemon
>2014/10/29 09:05:44.664012 [recoverd: 8207]: Take the recovery lock
>2014/10/29 09:05:44.675410 [ 7969]: Freeze priority 1
>2014/10/29 09:05:44.675705 [ 7969]: Freeze priority 2
>2014/10/29 09:05:44.675938 [ 7969]: Freeze priority 3
>2014/10/29 09:05:47.314017 [ 7969]: Thawing priority 1
>2014/10/29 09:05:47.314063 [ 7969]: Release freeze handler for prio 1
>2014/10/29 09:05:47.314102 [ 7969]: Thawing priority 2
>2014/10/29 09:05:47.314128 [ 7969]: Release freeze handler for prio 2
>2014/10/29 09:05:47.314171 [ 7969]: Thawing priority 3
>2014/10/29 09:05:47.314191 [ 7969]: Release freeze handler for prio 3
>2014/10/29 09:05:47.831604 [recoverd: 8207]: Resetting ban count to 0 for all nodes
>2014/10/29 09:06:00.838421 [recoverd: 8207]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 2 vs 1
>2014/10/29 09:06:00.838476 [recoverd: 8207]: Taking out recovery lock from recovery daemon
>2014/10/29 09:06:00.838491 [recoverd: 8207]: Take the recovery lock
>2014/10/29 09:06:00.843691 [ 7969]: Freeze priority 1
>2014/10/29 09:06:00.844501 [ 7969]: Freeze priority 2
>2014/10/29 09:06:00.845107 [ 7969]: Freeze priority 3
>2014/10/29 09:06:02.815223 [ 7969]: Thawing priority 1
>2014/10/29 09:06:02.815277 [ 7969]: Release freeze handler for prio 1
>2014/10/29 09:06:02.815308 [ 7969]: Thawing priority 2
>2014/10/29 09:06:02.815340 [ 7969]: Release freeze handler for prio 2
>2014/10/29 09:06:02.815373 [ 7969]: Thawing priority 3
>2014/10/29 09:06:02.815394 [ 7969]: Release freeze handler for prio 3
>2014/10/29 09:06:03.175310 [ 7969]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 09:06:03.336407 [recoverd: 8207]: Resetting ban count to 0 for all nodes
>2014/10/29 09:06:13.236166 [ 7969]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1
>2014/10/29 09:06:13.583318 [ 7969]: 50.samba: Redirecting to /bin/systemctl stop  smb.service
>2014/10/29 09:06:13.627326 [ 7969]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 09:06:13.638506 [ 7969]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 09:15:59.869061 [32406]: Starting CTDBD (Version 2.5.3) as PID: 32406
>2014/10/29 09:16:00.238118 [32406]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 09:16:00.261772 [32406]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 09:16:00.275908 [32406]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 09:16:00.290135 [32406]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 09:16:00.304308 [32406]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 09:16:00.318257 [32406]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 09:16:00.332054 [32406]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 09:16:00.332096 [32406]: Freeze priority 1
>2014/10/29 09:16:00.332622 [32406]: Freeze priority 2
>2014/10/29 09:16:00.332962 [32406]: Freeze priority 3
>2014/10/29 09:16:00.347767 [32406]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 09:16:00.351006 [32406]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 09:16:00.354266 [32406]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 09:16:00.434704 [32406]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 09:16:00.475130 [32406]: Freeze priority 1
>2014/10/29 09:16:00.475205 [32406]: Freeze priority 2
>2014/10/29 09:16:00.475259 [32406]: Freeze priority 3
>2014/10/29 09:16:00.678341 [32406]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 09:16:03.980660 [recoverd:32690]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 09:16:03.980748 [32406]: Freeze priority 1
>2014/10/29 09:16:03.980824 [32406]: Freeze priority 2
>2014/10/29 09:16:03.980888 [32406]: Freeze priority 3
>2014/10/29 09:16:09.761421 [32406]: Freeze priority 1
>2014/10/29 09:16:09.762200 [32406]: Freeze priority 2
>2014/10/29 09:16:09.762914 [32406]: Freeze priority 3
>2014/10/29 09:16:09.897037 [32406]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 09:16:09.897736 [32406]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 09:16:11.963652 [32406]: Thawing priority 1
>2014/10/29 09:16:11.963721 [32406]: Release freeze handler for prio 1
>2014/10/29 09:16:11.963770 [32406]: Thawing priority 2
>2014/10/29 09:16:11.963791 [32406]: Release freeze handler for prio 2
>2014/10/29 09:16:11.963819 [32406]: Thawing priority 3
>2014/10/29 09:16:11.963838 [32406]: Release freeze handler for prio 3
>2014/10/29 09:16:25.767088 [32406]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 09:16:25.956298 [32406]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 09:16:25.969449 [32406]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 09:16:26.000345 [32406]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 09:16:26.178695 [32406]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 09:16:26.513799 [recoverd:32690]: Trigger takeoverrun
>2014/10/29 09:16:28.445157 [32406]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/29 09:16:29.790563 [32406]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 10:04:18.714985 [32406]: Freeze priority 1
>2014/10/29 10:04:18.746248 [32406]: Freeze priority 2
>2014/10/29 10:04:18.747418 [32406]: Freeze priority 3
>2014/10/29 10:04:20.477745 [32406]: Thawing priority 1
>2014/10/29 10:04:20.477775 [32406]: Release freeze handler for prio 1
>2014/10/29 10:04:20.477802 [32406]: Thawing priority 2
>2014/10/29 10:04:20.477813 [32406]: Release freeze handler for prio 2
>2014/10/29 10:04:20.477833 [32406]: Thawing priority 3
>2014/10/29 10:04:20.477843 [32406]: Release freeze handler for prio 3
>2014/10/29 10:04:20.479079 [32406]: server/ctdb_call.c:1005 reqid 69953 not found
>2014/10/29 10:04:20.479116 [32406]: server/ctdb_call.c:1005 reqid 69954 not found
>2014/10/29 10:04:20.834705 [32406]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 10:14:39.873806 [32406]: Freeze priority 1
>2014/10/29 10:14:39.884064 [32406]: Freeze priority 2
>2014/10/29 10:14:39.884976 [32406]: Freeze priority 3
>2014/10/29 10:14:43.279029 [32406]: Thawing priority 1
>2014/10/29 10:14:43.279070 [32406]: Release freeze handler for prio 1
>2014/10/29 10:14:43.279103 [32406]: Thawing priority 2
>2014/10/29 10:14:43.279137 [32406]: Release freeze handler for prio 2
>2014/10/29 10:14:43.279166 [32406]: Thawing priority 3
>2014/10/29 10:14:43.279196 [32406]: Release freeze handler for prio 3
>2014/10/29 10:14:53.699000 [32406]: 10.interface: Killing TCP connection 10.10.10.206:49595 10.10.10.184:445
>2014/10/29 10:14:53.711739 [32406]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.184
>2014/10/29 10:14:53.721091 [32406]: 10.interface: Re-adding secondary address 10.10.10.183/24 to dev bond1
>2014/10/29 10:14:54.105661 [32406]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 11:18:08.638677 [32406]: Freeze priority 1
>2014/10/29 11:18:08.639781 [32406]: Freeze priority 2
>2014/10/29 11:18:08.640775 [32406]: Freeze priority 3
>2014/10/29 11:18:11.878323 [recoverd:32690]: Taking out recovery lock from recovery daemon
>2014/10/29 11:18:11.878356 [recoverd:32690]: Take the recovery lock
>2014/10/29 11:18:11.880882 [32406]: Freeze priority 1
>2014/10/29 11:18:11.880939 [32406]: Freeze priority 2
>2014/10/29 11:18:11.880979 [32406]: Freeze priority 3
>2014/10/29 11:18:15.539399 [32406]: Thawing priority 1
>2014/10/29 11:18:15.539443 [32406]: Release freeze handler for prio 1
>2014/10/29 11:18:15.539480 [32406]: Thawing priority 2
>2014/10/29 11:18:15.539503 [32406]: Release freeze handler for prio 2
>2014/10/29 11:18:15.539537 [32406]: Thawing priority 3
>2014/10/29 11:18:15.539560 [32406]: Release freeze handler for prio 3
>2014/10/29 11:18:15.914249 [32406]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 11:18:16.181043 [recoverd:32690]: Resetting ban count to 0 for all nodes
>2014/10/29 11:18:22.591276 [32406]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 11:18:22.601991 [32406]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 11:21:14.096010 [ 1536]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/29 11:21:14.096118 [ 1536]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 11:21:14.096133 [ 1536]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 11:25:14.394731 [ 8359]: Starting CTDBD (Version 2.5.3) as PID: 8359
>2014/10/29 11:25:15.387147 [ 8359]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 11:25:15.415256 [ 8359]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 11:25:15.430873 [ 8359]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 11:25:15.445127 [ 8359]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 11:25:15.459038 [ 8359]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 11:25:15.472939 [ 8359]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 11:25:15.486751 [ 8359]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 11:25:15.486783 [ 8359]: Freeze priority 1
>2014/10/29 11:25:15.496220 [ 8359]: Freeze priority 2
>2014/10/29 11:25:15.496627 [ 8359]: Freeze priority 3
>2014/10/29 11:25:15.660399 [ 8359]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 11:25:15.663879 [ 8359]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 11:25:15.667414 [ 8359]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 11:25:15.788313 [ 8359]: Freeze priority 1
>2014/10/29 11:25:15.788388 [ 8359]: Freeze priority 2
>2014/10/29 11:25:15.788441 [ 8359]: Freeze priority 3
>2014/10/29 11:25:19.293664 [recoverd: 8591]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 11:25:19.293746 [ 8359]: Freeze priority 1
>2014/10/29 11:25:19.293809 [ 8359]: Freeze priority 2
>2014/10/29 11:25:19.293863 [ 8359]: Freeze priority 3
>2014/10/29 11:25:22.557346 [ 8359]: Freeze priority 1
>2014/10/29 11:25:22.557887 [ 8359]: Freeze priority 2
>2014/10/29 11:25:22.558372 [ 8359]: Freeze priority 3
>2014/10/29 11:25:25.616951 [ 8359]: Thawing priority 1
>2014/10/29 11:25:25.616987 [ 8359]: Release freeze handler for prio 1
>2014/10/29 11:25:25.617012 [ 8359]: Thawing priority 2
>2014/10/29 11:25:25.617030 [ 8359]: Release freeze handler for prio 2
>2014/10/29 11:25:25.617065 [ 8359]: Thawing priority 3
>2014/10/29 11:25:25.617082 [ 8359]: Release freeze handler for prio 3
>2014/10/29 11:25:29.680386 [ 8359]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 11:25:35.963365 [ 8359]: Freeze priority 1
>2014/10/29 11:25:35.964501 [ 8359]: Freeze priority 2
>2014/10/29 11:25:35.965473 [ 8359]: Freeze priority 3
>2014/10/29 11:25:38.577176 [ 8359]: Thawing priority 1
>2014/10/29 11:25:38.577245 [ 8359]: Release freeze handler for prio 1
>2014/10/29 11:25:38.577281 [ 8359]: Thawing priority 2
>2014/10/29 11:25:38.577302 [ 8359]: Release freeze handler for prio 2
>2014/10/29 11:25:38.577334 [ 8359]: Thawing priority 3
>2014/10/29 11:25:38.577354 [ 8359]: Release freeze handler for prio 3
>2014/10/29 11:25:53.601380 [recoverd: 8591]: Trigger takeoverrun
>2014/10/29 11:25:53.776285 [ 8359]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 11:25:54.058009 [ 8359]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 11:25:54.068818 [ 8359]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 11:25:54.090914 [ 8359]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 11:25:54.291962 [ 8359]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 11:25:56.742700 [ 8359]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/29 11:25:57.570321 [ 8359]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 11:26:12.624219 [recoverd: 8591]: Taking out recovery lock from recovery daemon
>2014/10/29 11:26:12.624286 [recoverd: 8591]: Take the recovery lock
>2014/10/29 11:26:12.633945 [ 8359]: Freeze priority 1
>2014/10/29 11:26:12.635864 [ 8359]: Freeze priority 2
>2014/10/29 11:26:12.636875 [ 8359]: Freeze priority 3
>2014/10/29 11:26:16.027676 [ 8359]: Thawing priority 1
>2014/10/29 11:26:16.027724 [ 8359]: Release freeze handler for prio 1
>2014/10/29 11:26:16.027758 [ 8359]: Thawing priority 2
>2014/10/29 11:26:16.027790 [ 8359]: Release freeze handler for prio 2
>2014/10/29 11:26:16.027819 [ 8359]: Thawing priority 3
>2014/10/29 11:26:16.027851 [ 8359]: Release freeze handler for prio 3
>2014/10/29 11:26:16.355892 [recoverd: 8591]: Resetting ban count to 0 for all nodes
>2014/10/29 14:06:49.479891 [ 1493]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/29 14:06:49.479994 [ 1493]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 14:06:49.480009 [ 1493]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 14:10:53.418766 [ 8284]: Starting CTDBD (Version 2.5.3) as PID: 8284
>2014/10/29 14:10:54.333021 [ 8284]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 14:10:54.356435 [ 8284]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 14:10:54.370534 [ 8284]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 14:10:54.384342 [ 8284]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 14:10:54.398206 [ 8284]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 14:10:54.412186 [ 8284]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 14:10:54.426139 [ 8284]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 14:10:54.426171 [ 8284]: Freeze priority 1
>2014/10/29 14:10:54.445413 [ 8284]: Freeze priority 2
>2014/10/29 14:10:54.445767 [ 8284]: Freeze priority 3
>2014/10/29 14:10:54.639168 [ 8284]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 14:10:54.643377 [ 8284]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 14:10:54.647529 [ 8284]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 14:10:54.770475 [ 8284]: Freeze priority 1
>2014/10/29 14:10:54.770548 [ 8284]: Freeze priority 2
>2014/10/29 14:10:54.770602 [ 8284]: Freeze priority 3
>2014/10/29 14:10:54.780011 [ 8284]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 14:10:55.133919 [ 8284]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 14:10:58.275532 [recoverd: 8515]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 14:10:58.275621 [ 8284]: Freeze priority 1
>2014/10/29 14:10:58.275684 [ 8284]: Freeze priority 2
>2014/10/29 14:10:58.275744 [ 8284]: Freeze priority 3
>2014/10/29 14:11:02.770926 [ 8284]: Freeze priority 1
>2014/10/29 14:11:02.780257 [ 8284]: Freeze priority 2
>2014/10/29 14:11:02.781256 [ 8284]: Freeze priority 3
>2014/10/29 14:11:02.918864 [ 8284]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 14:11:02.919638 [ 8284]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 14:11:04.581838 [ 8284]: Thawing priority 1
>2014/10/29 14:11:04.581886 [ 8284]: Release freeze handler for prio 1
>2014/10/29 14:11:04.581924 [ 8284]: Thawing priority 2
>2014/10/29 14:11:04.581943 [ 8284]: Release freeze handler for prio 2
>2014/10/29 14:11:04.581969 [ 8284]: Thawing priority 3
>2014/10/29 14:11:04.581986 [ 8284]: Release freeze handler for prio 3
>2014/10/29 14:11:18.806494 [recoverd: 8515]: Trigger takeoverrun
>2014/10/29 14:11:18.853554 [ 8284]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 14:11:19.219152 [ 8284]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 14:11:19.230098 [ 8284]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 14:11:19.253268 [ 8284]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 14:11:19.411442 [ 8284]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 14:11:21.843929 [ 8284]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/29 14:11:22.323842 [ 8284]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 14:13:44.788288 [recoverd: 8515]: ctdb_control error: 'node is disconnected'
>2014/10/29 14:13:44.788337 [recoverd: 8515]: ctdb_control error: 'node is disconnected'
>2014/10/29 14:13:44.788354 [recoverd: 8515]: Async operation failed with ret=-1 res=-1 opcode=80
>2014/10/29 14:13:44.788365 [recoverd: 8515]: Async wait failed - fail_count=1
>2014/10/29 14:13:44.788377 [recoverd: 8515]: server/ctdb_recoverd.c:345 Failed to read node capabilities.
>2014/10/29 14:13:44.788389 [recoverd: 8515]: server/ctdb_recoverd.c:3678 Unable to update node capabilities.
>2014/10/29 14:13:44.790037 [ 8284]: Freeze priority 1
>2014/10/29 14:13:44.791547 [ 8284]: Freeze priority 2
>2014/10/29 14:13:44.792587 [ 8284]: Freeze priority 3
>2014/10/29 14:14:09.335310 [ 8284]: Freeze priority 1
>2014/10/29 14:14:09.335469 [ 8284]: Freeze priority 2
>2014/10/29 14:14:09.335594 [ 8284]: Freeze priority 3
>2014/10/29 14:14:12.342634 [recoverd: 8515]: Taking out recovery lock from recovery daemon
>2014/10/29 14:14:12.342696 [recoverd: 8515]: Take the recovery lock
>2014/10/29 14:14:24.099076 [ 8284]: Event script '00.ctdb monitor ' timed out after 60.0s, count: 0, pid: 19353
>2014/10/29 14:14:33.373727 [recoverd: 8515]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile'
>2014/10/29 14:14:33.373764 [recoverd: 8515]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/29 14:14:33.373827 [ 8284]: Banning this node for 30 seconds
>2014/10/29 14:14:33.374961 [ 8284]: Hung-script: ===== Start of hung script debug for PID="19353", event="monitor" =====
>2014/10/29 14:14:33.375006 [ 8284]: Hung-script: pstree -p -a 19353:
>2014/10/29 14:14:33.415218 [ 8284]: Hung-script: 
>2014/10/29 14:14:33.416861 [ 8284]: 10.interface: Killing TCP connection 10.10.10.208:57370 10.10.10.184:445
>2014/10/29 14:14:33.417298 [ 8284]: Hung-script: ---- ctdb scriptstatus monitor: ----
>2014/10/29 14:14:33.418463 [ 8284]: Hung-script: 1 scripts were executed last monitor cycle
>2014/10/29 14:14:33.418498 [ 8284]: Hung-script: 00.ctdb              Status:TIMEDOUT    Wed Oct 29 14:13:24 2014
>2014/10/29 14:14:33.418522 [ 8284]: Hung-script:    OUTPUT:
>2014/10/29 14:14:33.418657 [ 8284]: Hung-script: ===== End of hung script debug for PID="19353", event="monitor" =====
>2014/10/29 14:14:33.419037 [ 8284]: ctdb_kill: trying to kill(19353, 9) a process that does not exist
>2014/10/29 14:14:33.422912 [ 8284]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.184
>2014/10/29 14:15:03.373964 [ 8284]: Banning timedout
>2014/10/29 14:15:03.597247 [ 8284]: Freeze priority 1
>2014/10/29 14:15:03.597326 [ 8284]: Freeze priority 2
>2014/10/29 14:15:03.597370 [ 8284]: Freeze priority 3
>2014/10/29 14:15:06.602285 [recoverd: 8515]: Public IP '10.10.10.184' is not assigned and we could serve it
>2014/10/29 14:15:06.602419 [recoverd: 8515]: Trigger takeoverrun
>2014/10/29 14:15:06.602923 [recoverd: 8515]: Taking out recovery lock from recovery daemon
>2014/10/29 14:15:06.602943 [recoverd: 8515]: Take the recovery lock
>2014/10/29 14:15:06.607383 [recoverd: 8515]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile'
>2014/10/29 14:15:06.607417 [recoverd: 8515]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/29 14:15:06.607472 [ 8284]: Banning this node for 30 seconds
>2014/10/29 14:15:36.608086 [ 8284]: Banning timedout
>2014/10/29 14:15:36.649226 [ 8284]: Freeze priority 1
>2014/10/29 14:15:36.649407 [ 8284]: Freeze priority 2
>2014/10/29 14:15:36.649571 [ 8284]: Freeze priority 3
>2014/10/29 14:15:39.654977 [recoverd: 8515]: Public IP '10.10.10.184' is not assigned and we could serve it
>2014/10/29 14:15:39.655117 [recoverd: 8515]: Trigger takeoverrun
>2014/10/29 14:15:39.655707 [recoverd: 8515]: Taking out recovery lock from recovery daemon
>2014/10/29 14:15:39.655721 [recoverd: 8515]: Take the recovery lock
>2014/10/29 14:15:39.661064 [ 8284]: Freeze priority 1
>2014/10/29 14:15:39.661129 [ 8284]: Freeze priority 2
>2014/10/29 14:15:39.661175 [ 8284]: Freeze priority 3
>2014/10/29 14:15:41.613703 [ 8284]: Thawing priority 1
>2014/10/29 14:15:41.613742 [ 8284]: Release freeze handler for prio 1
>2014/10/29 14:15:41.613778 [ 8284]: Thawing priority 2
>2014/10/29 14:15:41.613810 [ 8284]: Release freeze handler for prio 2
>2014/10/29 14:15:41.613849 [ 8284]: Thawing priority 3
>2014/10/29 14:15:41.613869 [ 8284]: Release freeze handler for prio 3
>2014/10/29 14:15:41.615178 [ 8284]: server/ctdb_call.c:1005 reqid 4760 not found
>2014/10/29 14:15:41.615222 [ 8284]: server/ctdb_call.c:1005 reqid 4761 not found
>2014/10/29 14:15:41.915021 [ 8284]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 14:15:42.073080 [recoverd: 8515]: Resetting ban count to 0 for all nodes
>2014/10/29 14:15:52.077427 [recoverd: 8515]: Taking out recovery lock from recovery daemon
>2014/10/29 14:15:52.077481 [recoverd: 8515]: Take the recovery lock
>2014/10/29 14:15:52.090130 [ 8284]: Freeze priority 1
>2014/10/29 14:15:52.091017 [ 8284]: Freeze priority 2
>2014/10/29 14:15:52.091672 [ 8284]: Freeze priority 3
>2014/10/29 14:15:54.653999 [ 8284]: Thawing priority 1
>2014/10/29 14:15:54.654052 [ 8284]: Release freeze handler for prio 1
>2014/10/29 14:15:54.654095 [ 8284]: Thawing priority 2
>2014/10/29 14:15:54.654120 [ 8284]: Release freeze handler for prio 2
>2014/10/29 14:15:54.654154 [ 8284]: Thawing priority 3
>2014/10/29 14:15:54.654177 [ 8284]: Release freeze handler for prio 3
>2014/10/29 14:15:55.102501 [ 8284]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 14:15:55.265147 [recoverd: 8515]: Resetting ban count to 0 for all nodes
>2014/10/29 14:21:51.579877 [ 8284]: Event script '00.ctdb monitor ' timed out after 60.0s, count: 0, pid: 8945
>2014/10/29 14:22:06.874627 [ 8284]: Hung-script: ===== Start of hung script debug for PID="8945", event="monitor" =====
>2014/10/29 14:22:06.874685 [ 8284]: Hung-script: pstree -p -a 8945:
>2014/10/29 14:22:06.913145 [ 8284]: Hung-script: 
>2014/10/29 14:22:06.914938 [ 8284]: Hung-script: ---- ctdb scriptstatus monitor: ----
>2014/10/29 14:22:06.923328 [ 8284]: Hung-script: 1 scripts were executed last monitor cycle
>2014/10/29 14:22:06.923414 [ 8284]: Hung-script: 00.ctdb              Status:TIMEDOUT    Wed Oct 29 14:20:51 2014
>2014/10/29 14:22:06.923440 [ 8284]: Hung-script:    OUTPUT:
>2014/10/29 14:22:06.923574 [ 8284]: Hung-script: ===== End of hung script debug for PID="8945", event="monitor" =====
>2014/10/29 14:22:06.924125 [ 8284]: ctdb_kill: trying to kill(8945, 9) a process that does not exist
>2014/10/29 14:24:28.029239 [recoverd: 8515]: Taking out recovery lock from recovery daemon
>2014/10/29 14:24:28.029289 [recoverd: 8515]: Take the recovery lock
>2014/10/29 14:24:28.071502 [ 8284]: Freeze priority 1
>2014/10/29 14:24:28.082101 [ 8284]: Freeze priority 2
>2014/10/29 14:24:28.082974 [ 8284]: Freeze priority 3
>2014/10/29 14:24:30.171332 [ 8284]: Thawing priority 1
>2014/10/29 14:24:30.171390 [ 8284]: Release freeze handler for prio 1
>2014/10/29 14:24:30.171431 [ 8284]: Thawing priority 2
>2014/10/29 14:24:30.171453 [ 8284]: Release freeze handler for prio 2
>2014/10/29 14:24:30.171485 [ 8284]: Thawing priority 3
>2014/10/29 14:24:30.171505 [ 8284]: Release freeze handler for prio 3
>2014/10/29 14:24:30.502276 [recoverd: 8515]: Resetting ban count to 0 for all nodes
>2014/10/29 14:24:40.686428 [recoverd: 8515]: server/ctdb_recoverd.c:3933 Remote node:0 has different flags for node 1. It has 0x02 vs our 0x00
>2014/10/29 14:24:40.686473 [recoverd: 8515]: Use flags 0x00 from local recmaster node for cluster update of node 1 flags
>2014/10/29 14:24:40.687211 [recoverd: 8515]: Taking out recovery lock from recovery daemon
>2014/10/29 14:24:40.687241 [recoverd: 8515]: Take the recovery lock
>2014/10/29 14:24:40.698006 [ 8284]: Freeze priority 1
>2014/10/29 14:24:40.699032 [ 8284]: Freeze priority 2
>2014/10/29 14:24:40.699853 [ 8284]: Freeze priority 3
>2014/10/29 14:24:43.231676 [ 8284]: Thawing priority 1
>2014/10/29 14:24:43.231733 [ 8284]: Release freeze handler for prio 1
>2014/10/29 14:24:43.231794 [ 8284]: Thawing priority 2
>2014/10/29 14:24:43.231841 [ 8284]: Release freeze handler for prio 2
>2014/10/29 14:24:43.231871 [ 8284]: Thawing priority 3
>2014/10/29 14:24:43.231889 [ 8284]: Release freeze handler for prio 3
>2014/10/29 14:24:43.560511 [recoverd: 8515]: Resetting ban count to 0 for all nodes
>2014/10/29 14:25:00.585508 [ 8284]: Monitoring event was cancelled
>2014/10/29 14:25:00.585556 [ 8284]: server/eventscript.c:569 Sending SIGTERM to child pid:21463
>2014/10/29 14:25:00.654842 [ 8284]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1
>2014/10/29 14:25:01.013086 [ 8284]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 15:32:08.634266 [recoverd: 8515]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 3 vs 2
>2014/10/29 15:32:08.634312 [recoverd: 8515]: Taking out recovery lock from recovery daemon
>2014/10/29 15:32:08.634326 [recoverd: 8515]: Take the recovery lock
>2014/10/29 15:32:08.644807 [ 8284]: Freeze priority 1
>2014/10/29 15:32:08.645896 [ 8284]: Freeze priority 2
>2014/10/29 15:32:08.646766 [ 8284]: Freeze priority 3
>2014/10/29 15:32:10.157111 [ 8284]: Thawing priority 1
>2014/10/29 15:32:10.157165 [ 8284]: Release freeze handler for prio 1
>2014/10/29 15:32:10.157205 [ 8284]: Thawing priority 2
>2014/10/29 15:32:10.157221 [ 8284]: Release freeze handler for prio 2
>2014/10/29 15:32:10.157256 [ 8284]: Thawing priority 3
>2014/10/29 15:32:10.157271 [ 8284]: Release freeze handler for prio 3
>2014/10/29 15:32:10.163833 [set_recmode:  676]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Stale file handle)
>2014/10/29 15:32:10.730682 [recoverd: 8515]: Resetting ban count to 0 for all nodes
>2014/10/29 15:33:47.569913 [ 8284]: 50.samba: Redirecting to /bin/systemctl stop  smb.service
>2014/10/29 15:33:47.612808 [ 8284]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 15:33:47.623810 [ 8284]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 15:47:03.213887 [ 1543]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/29 15:47:03.213984 [ 1543]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 15:47:03.213998 [ 1543]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 16:04:51.484599 [31052]: Starting CTDBD (Version 2.5.3) as PID: 31052
>2014/10/29 16:04:52.482214 [31052]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 16:04:52.506103 [31052]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 16:04:52.520207 [31052]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 16:04:52.534119 [31052]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 16:04:52.547942 [31052]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 16:04:52.561833 [31052]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 16:04:52.575691 [31052]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 16:04:52.575731 [31052]: Freeze priority 1
>2014/10/29 16:04:52.594418 [31052]: Freeze priority 2
>2014/10/29 16:04:52.594771 [31052]: Freeze priority 3
>2014/10/29 16:04:52.759012 [31052]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 16:04:52.762740 [31052]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 16:04:52.766730 [31052]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 16:04:52.887677 [31052]: Freeze priority 1
>2014/10/29 16:04:52.887750 [31052]: Freeze priority 2
>2014/10/29 16:04:52.887802 [31052]: Freeze priority 3
>2014/10/29 16:04:53.188060 [31052]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 16:04:53.299282 [31052]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 16:04:56.894524 [recoverd:31372]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 16:04:56.894600 [31052]: Freeze priority 1
>2014/10/29 16:04:56.894662 [31052]: Freeze priority 2
>2014/10/29 16:04:56.894715 [31052]: Freeze priority 3
>2014/10/29 16:05:00.622479 [31052]: Freeze priority 1
>2014/10/29 16:05:00.623559 [31052]: Freeze priority 2
>2014/10/29 16:05:00.624175 [31052]: Freeze priority 3
>2014/10/29 16:05:00.785776 [31052]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 16:05:00.786481 [31052]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 16:05:02.339456 [31052]: Thawing priority 1
>2014/10/29 16:05:02.339514 [31052]: Release freeze handler for prio 1
>2014/10/29 16:05:02.339548 [31052]: Thawing priority 2
>2014/10/29 16:05:02.339569 [31052]: Release freeze handler for prio 2
>2014/10/29 16:05:02.339598 [31052]: Thawing priority 3
>2014/10/29 16:05:02.339618 [31052]: Release freeze handler for prio 3
>2014/10/29 16:05:16.619296 [31052]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 16:05:16.885470 [31052]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 16:05:16.896194 [31052]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 16:05:16.918829 [31052]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 16:05:16.924715 [recoverd:31372]: Trigger takeoverrun
>2014/10/29 16:05:17.128274 [31052]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 16:05:19.551456 [31052]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/29 16:05:20.116198 [31052]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 16:21:43.220828 [31052]: Freeze priority 1
>2014/10/29 16:21:43.258151 [31052]: Freeze priority 2
>2014/10/29 16:21:43.259643 [31052]: Freeze priority 3
>2014/10/29 16:21:44.840755 [31052]: Thawing priority 1
>2014/10/29 16:21:44.840806 [31052]: Release freeze handler for prio 1
>2014/10/29 16:21:44.840862 [31052]: Thawing priority 2
>2014/10/29 16:21:44.840886 [31052]: Release freeze handler for prio 2
>2014/10/29 16:21:44.840916 [31052]: Thawing priority 3
>2014/10/29 16:21:44.840934 [31052]: Release freeze handler for prio 3
>2014/10/29 16:21:45.224989 [31052]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 16:21:55.421110 [31052]: Freeze priority 1
>2014/10/29 16:21:55.422390 [31052]: Freeze priority 2
>2014/10/29 16:21:55.423374 [31052]: Freeze priority 3
>2014/10/29 16:21:56.902704 [recoverd:31372]: server/ctdb_recoverd.c:2343 Reload nodes file from recovery daemon
>2014/10/29 16:21:56.929327 [31052]: Thawing priority 1
>2014/10/29 16:21:56.929361 [31052]: Release freeze handler for prio 1
>2014/10/29 16:21:56.929397 [31052]: Thawing priority 2
>2014/10/29 16:21:56.929419 [31052]: Release freeze handler for prio 2
>2014/10/29 16:21:56.929449 [31052]: Thawing priority 3
>2014/10/29 16:21:56.929469 [31052]: Release freeze handler for prio 3
>2014/10/29 16:32:18.159385 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:32:18.159984 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:33:53.716041 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:33:53.725644 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:35:41.704939 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:35:41.705245 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:35:42.653587 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:35:42.653921 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:42:44.379423 [31052]: Freeze priority 1
>2014/10/29 16:42:44.389703 [31052]: Freeze priority 2
>2014/10/29 16:42:44.390459 [31052]: Freeze priority 3
>2014/10/29 16:42:47.394200 [recoverd:31372]: Taking out recovery lock from recovery daemon
>2014/10/29 16:42:47.394233 [recoverd:31372]: Take the recovery lock
>2014/10/29 16:42:47.396711 [31052]: Freeze priority 1
>2014/10/29 16:42:47.396770 [31052]: Freeze priority 2
>2014/10/29 16:42:47.396811 [31052]: Freeze priority 3
>2014/10/29 16:42:50.378398 [31052]: Thawing priority 1
>2014/10/29 16:42:50.378444 [31052]: Release freeze handler for prio 1
>2014/10/29 16:42:50.378477 [31052]: Thawing priority 2
>2014/10/29 16:42:50.378499 [31052]: Release freeze handler for prio 2
>2014/10/29 16:42:50.378530 [31052]: Thawing priority 3
>2014/10/29 16:42:50.378551 [31052]: Release freeze handler for prio 3
>2014/10/29 16:42:50.789394 [31052]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 16:42:50.951608 [recoverd:31372]: Resetting ban count to 0 for all nodes
>2014/10/29 16:45:18.758701 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:45:18.758763 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:55:04.122418 [recoverd:31372]: Taking out recovery lock from recovery daemon
>2014/10/29 16:55:04.122464 [recoverd:31372]: Take the recovery lock
>2014/10/29 16:55:04.166436 [31052]: Freeze priority 1
>2014/10/29 16:55:04.167362 [31052]: Freeze priority 2
>2014/10/29 16:55:04.168040 [31052]: Freeze priority 3
>2014/10/29 16:55:05.734658 [31052]: Thawing priority 1
>2014/10/29 16:55:05.734703 [31052]: Release freeze handler for prio 1
>2014/10/29 16:55:05.734758 [31052]: Thawing priority 2
>2014/10/29 16:55:05.734779 [31052]: Release freeze handler for prio 2
>2014/10/29 16:55:05.734808 [31052]: Thawing priority 3
>2014/10/29 16:55:05.734827 [31052]: Release freeze handler for prio 3
>2014/10/29 16:55:06.062160 [recoverd:31372]: Resetting ban count to 0 for all nodes
>2014/10/29 16:55:23.112168 [31052]: 10.interface: Killing TCP connection 10.10.10.206:49648 10.10.10.183:445
>2014/10/29 16:55:23.135261 [31052]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.183
>2014/10/29 16:55:23.483946 [31052]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 16:58:19.321135 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 16:58:19.322090 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1
>2014/10/29 17:09:31.669668 [31052]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1
>2014/10/29 17:09:32.020211 [31052]: 50.samba: Redirecting to /bin/systemctl stop  smb.service
>2014/10/29 17:09:32.063776 [31052]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 17:09:32.074771 [31052]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 17:34:39.850701 [ 9566]: Starting CTDBD (Version 2.5.3) as PID: 9566
>2014/10/29 17:34:40.265163 [ 9566]: Ignoring persistent database 'registry.tdb.2'
>2014/10/29 17:34:40.265186 [ 9566]: Ignoring persistent database 'passdb.tdb.2'
>2014/10/29 17:34:40.265196 [ 9566]: Ignoring persistent database 'secrets.tdb.2'
>2014/10/29 17:34:40.265205 [ 9566]: Ignoring persistent database 'share_info.tdb.2'
>2014/10/29 17:34:40.265214 [ 9566]: Ignoring persistent database 'ctdb.tdb.2'
>2014/10/29 17:34:40.265224 [ 9566]: Ignoring persistent database 'account_policy.tdb.2'
>2014/10/29 17:34:40.265233 [ 9566]: Ignoring persistent database 'group_mapping.tdb.2'
>2014/10/29 17:34:40.265255 [ 9566]: Freeze priority 1
>2014/10/29 17:34:40.265609 [ 9566]: Freeze priority 2
>2014/10/29 17:34:40.265939 [ 9566]: Freeze priority 3
>2014/10/29 17:34:40.280864 [ 9566]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 17:34:40.284109 [ 9566]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 17:34:40.287224 [ 9566]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 17:34:40.409963 [ 9566]: Freeze priority 1
>2014/10/29 17:34:40.410056 [ 9566]: Freeze priority 2
>2014/10/29 17:34:40.410109 [ 9566]: Freeze priority 3
>2014/10/29 17:34:40.658460 [ 9566]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 17:34:40.658487 [ 9566]: Unknown db_id 0x6cf2837d in ctdb_ltdb_update_seqnum
>2014/10/29 17:34:43.915647 [recoverd: 9796]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 1) have - force an election
>2014/10/29 17:34:43.915766 [ 9566]: Freeze priority 1
>2014/10/29 17:34:43.915861 [ 9566]: Freeze priority 2
>2014/10/29 17:34:43.915942 [ 9566]: Freeze priority 3
>2014/10/29 17:34:49.008816 [ 9566]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 17:34:49.027289 [ 9566]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 17:34:49.047947 [ 9566]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 17:34:49.066597 [ 9566]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 17:34:49.085280 [ 9566]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 17:34:49.104282 [ 9566]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 17:34:49.123159 [ 9566]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 17:34:49.130130 [ 9566]: Freeze priority 1
>2014/10/29 17:34:49.130873 [ 9566]: Freeze priority 2
>2014/10/29 17:34:49.131412 [ 9566]: Freeze priority 3
>2014/10/29 17:34:49.264825 [ 9566]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 17:34:50.635880 [ 9566]: Thawing priority 1
>2014/10/29 17:34:50.635920 [ 9566]: Release freeze handler for prio 1
>2014/10/29 17:34:50.635950 [ 9566]: Thawing priority 2
>2014/10/29 17:34:50.635968 [ 9566]: Release freeze handler for prio 2
>2014/10/29 17:34:50.635992 [ 9566]: Thawing priority 3
>2014/10/29 17:34:50.636009 [ 9566]: Release freeze handler for prio 3
>2014/10/29 17:35:04.951870 [ 9566]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 17:35:05.140320 [ 9566]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 17:35:05.153538 [ 9566]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 17:35:05.184185 [ 9566]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 17:35:05.388373 [ 9566]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 17:35:05.448499 [recoverd: 9796]: Trigger takeoverrun
>2014/10/29 17:35:07.656092 [ 9566]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/29 17:35:08.389715 [ 9566]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 18:37:49.090781 [ 9566]: Freeze priority 1
>2014/10/29 18:37:49.121643 [ 9566]: Freeze priority 2
>2014/10/29 18:37:49.122481 [ 9566]: Freeze priority 3
>2014/10/29 18:37:52.128366 [recoverd: 9796]: Taking out recovery lock from recovery daemon
>2014/10/29 18:37:52.128408 [recoverd: 9796]: Take the recovery lock
>2014/10/29 18:37:52.131497 [ 9566]: Freeze priority 1
>2014/10/29 18:37:52.131565 [ 9566]: Freeze priority 2
>2014/10/29 18:37:52.131616 [ 9566]: Freeze priority 3
>2014/10/29 18:37:54.590220 [ 9566]: Thawing priority 1
>2014/10/29 18:37:54.590270 [ 9566]: Release freeze handler for prio 1
>2014/10/29 18:37:54.590304 [ 9566]: Thawing priority 2
>2014/10/29 18:37:54.590326 [ 9566]: Release freeze handler for prio 2
>2014/10/29 18:37:54.590357 [ 9566]: Thawing priority 3
>2014/10/29 18:37:54.590376 [ 9566]: Release freeze handler for prio 3
>2014/10/29 18:37:55.028359 [ 9566]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 18:37:55.208120 [recoverd: 9796]: Resetting ban count to 0 for all nodes
>2014/10/29 19:20:53.298177 [recoverd: 9796]: Taking out recovery lock from recovery daemon
>2014/10/29 19:20:53.298245 [recoverd: 9796]: Take the recovery lock
>2014/10/29 19:20:53.461598 [ 9566]: Freeze priority 1
>2014/10/29 19:20:53.462346 [ 9566]: Freeze priority 2
>2014/10/29 19:20:53.462908 [ 9566]: Freeze priority 3
>2014/10/29 19:20:55.096956 [ 9566]: Thawing priority 1
>2014/10/29 19:20:55.097078 [ 9566]: Release freeze handler for prio 1
>2014/10/29 19:20:55.097139 [ 9566]: Thawing priority 2
>2014/10/29 19:20:55.097161 [ 9566]: Release freeze handler for prio 2
>2014/10/29 19:20:55.097202 [ 9566]: Thawing priority 3
>2014/10/29 19:20:55.097216 [ 9566]: Release freeze handler for prio 3
>2014/10/29 19:20:55.424596 [recoverd: 9796]: Resetting ban count to 0 for all nodes
>2014/10/29 19:21:12.521040 [ 9566]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1
>2014/10/29 19:21:12.937205 [ 9566]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 19:30:42.335813 [recoverd: 9796]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 2 vs 1
>2014/10/29 19:30:42.335874 [recoverd: 9796]: Taking out recovery lock from recovery daemon
>2014/10/29 19:30:42.335889 [recoverd: 9796]: Take the recovery lock
>2014/10/29 19:30:42.341301 [ 9566]: Freeze priority 1
>2014/10/29 19:30:42.342120 [ 9566]: Freeze priority 2
>2014/10/29 19:30:42.342707 [ 9566]: Freeze priority 3
>2014/10/29 19:30:43.702157 [ 9566]: Thawing priority 1
>2014/10/29 19:30:43.702199 [ 9566]: Release freeze handler for prio 1
>2014/10/29 19:30:43.702229 [ 9566]: Thawing priority 2
>2014/10/29 19:30:43.702246 [ 9566]: Release freeze handler for prio 2
>2014/10/29 19:30:43.702272 [ 9566]: Thawing priority 3
>2014/10/29 19:30:43.702288 [ 9566]: Release freeze handler for prio 3
>2014/10/29 19:30:43.956338 [ 9566]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 19:30:44.114648 [recoverd: 9796]: Resetting ban count to 0 for all nodes
>2014/10/29 19:30:50.588104 [ 9566]: 50.samba: Redirecting to /bin/systemctl stop  smb.service
>2014/10/29 19:30:50.631129 [ 9566]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 19:30:50.641948 [ 9566]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 19:36:42.884469 [19587]: Starting CTDBD (Version 2.5.3) as PID: 19587
>2014/10/29 19:36:44.482193 [19587]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 19:36:44.508792 [19587]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 19:36:44.523263 [19587]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 19:36:44.537928 [19587]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 19:36:44.537981 [19587]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/29 19:36:44.537991 [19587]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/29 19:36:44.538000 [19587]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/29 19:36:44.538009 [19587]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/29 19:36:44.538018 [19587]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/29 19:36:44.553007 [19587]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 19:36:44.567757 [19587]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 19:36:44.567805 [19587]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/29 19:36:44.567815 [19587]: Ignoring persistent database 'registry.tdb.1'
>2014/10/29 19:36:44.582528 [19587]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 19:36:44.582598 [19587]: Freeze priority 1
>2014/10/29 19:36:44.587185 [19587]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 19:36:44.593743 [19587]: Freeze priority 2
>2014/10/29 19:36:44.594143 [19587]: Freeze priority 3
>2014/10/29 19:36:44.758454 [19587]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 19:36:44.762491 [19587]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 19:36:44.766251 [19587]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 19:36:44.892084 [19587]: Freeze priority 1
>2014/10/29 19:36:44.892171 [19587]: Freeze priority 2
>2014/10/29 19:36:44.892238 [19587]: Freeze priority 3
>2014/10/29 19:36:48.397613 [recoverd:19916]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 19:36:48.397708 [19587]: Freeze priority 1
>2014/10/29 19:36:48.397768 [19587]: Freeze priority 2
>2014/10/29 19:36:48.397831 [19587]: Freeze priority 3
>2014/10/29 19:36:52.028422 [19587]: Freeze priority 1
>2014/10/29 19:36:52.029314 [19587]: Freeze priority 2
>2014/10/29 19:36:52.030042 [19587]: Freeze priority 3
>2014/10/29 19:36:52.190865 [19587]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 19:36:52.191667 [19587]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 19:36:53.532585 [19587]: Thawing priority 1
>2014/10/29 19:36:53.532640 [19587]: Release freeze handler for prio 1
>2014/10/29 19:36:53.532675 [19587]: Thawing priority 2
>2014/10/29 19:36:53.532708 [19587]: Release freeze handler for prio 2
>2014/10/29 19:36:53.532736 [19587]: Thawing priority 3
>2014/10/29 19:36:53.532766 [19587]: Release freeze handler for prio 3
>2014/10/29 19:37:07.852270 [19587]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 19:37:08.138667 [19587]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 19:37:08.149968 [19587]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 19:37:08.179908 [19587]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 19:37:08.428002 [recoverd:19916]: Trigger takeoverrun
>2014/10/29 19:37:08.824127 [19587]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 19:37:11.423222 [19587]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/29 19:37:12.375109 [19587]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 19:38:46.055746 [19587]: Freeze priority 1
>2014/10/29 19:38:46.057420 [19587]: Freeze priority 2
>2014/10/29 19:38:46.058510 [19587]: Freeze priority 3
>2014/10/29 19:38:46.364846 [recoverd:19916]: server/ctdb_recoverd.c:2343 Reload nodes file from recovery daemon
>2014/10/29 19:38:47.588413 [19587]: Thawing priority 1
>2014/10/29 19:38:47.588455 [19587]: Release freeze handler for prio 1
>2014/10/29 19:38:47.588493 [19587]: Thawing priority 2
>2014/10/29 19:38:47.588516 [19587]: Release freeze handler for prio 2
>2014/10/29 19:38:47.588558 [19587]: Thawing priority 3
>2014/10/29 19:38:47.588578 [19587]: Release freeze handler for prio 3
>2014/10/29 19:38:47.605766 [19587]: Freeze priority 1
>2014/10/29 19:38:47.606796 [19587]: Freeze priority 2
>2014/10/29 19:38:47.607622 [19587]: Freeze priority 3
>2014/10/29 19:38:49.126353 [19587]: Thawing priority 1
>2014/10/29 19:38:49.126407 [19587]: Release freeze handler for prio 1
>2014/10/29 19:38:49.126439 [19587]: Thawing priority 2
>2014/10/29 19:38:49.126471 [19587]: Release freeze handler for prio 2
>2014/10/29 19:38:49.126501 [19587]: Thawing priority 3
>2014/10/29 19:38:49.126520 [19587]: Release freeze handler for prio 3
>2014/10/29 19:38:59.631216 [19587]: Freeze priority 1
>2014/10/29 19:38:59.632494 [19587]: Freeze priority 2
>2014/10/29 19:38:59.633461 [19587]: Freeze priority 3
>2014/10/29 19:39:01.163936 [19587]: Thawing priority 1
>2014/10/29 19:39:01.164025 [19587]: Release freeze handler for prio 1
>2014/10/29 19:39:01.164086 [19587]: Thawing priority 2
>2014/10/29 19:39:01.164108 [19587]: Release freeze handler for prio 2
>2014/10/29 19:39:01.164146 [19587]: Thawing priority 3
>2014/10/29 19:39:01.164164 [19587]: Release freeze handler for prio 3
>2014/10/29 19:51:22.676194 [19587]: Freeze priority 1
>2014/10/29 19:51:22.701308 [19587]: Freeze priority 2
>2014/10/29 19:51:22.704192 [19587]: Freeze priority 3
>2014/10/29 19:51:24.633925 [19587]: Thawing priority 1
>2014/10/29 19:51:24.633971 [19587]: Release freeze handler for prio 1
>2014/10/29 19:51:24.634018 [19587]: Thawing priority 2
>2014/10/29 19:51:24.634036 [19587]: Release freeze handler for prio 2
>2014/10/29 19:51:24.634062 [19587]: Thawing priority 3
>2014/10/29 19:51:24.634078 [19587]: Release freeze handler for prio 3
>2014/10/29 19:57:52.819632 [19587]: Freeze priority 1
>2014/10/29 19:57:52.835994 [19587]: Freeze priority 2
>2014/10/29 19:57:52.837507 [19587]: Freeze priority 3
>2014/10/29 19:57:56.080585 [19587]: Thawing priority 1
>2014/10/29 19:57:56.080620 [19587]: Release freeze handler for prio 1
>2014/10/29 19:57:56.080657 [19587]: Thawing priority 2
>2014/10/29 19:57:56.080676 [19587]: Release freeze handler for prio 2
>2014/10/29 19:57:56.080706 [19587]: Thawing priority 3
>2014/10/29 19:57:56.080720 [19587]: Release freeze handler for prio 3
>2014/10/29 19:58:06.470038 [19587]: Freeze priority 1
>2014/10/29 19:58:06.472536 [19587]: Freeze priority 2
>2014/10/29 19:58:06.476259 [19587]: Freeze priority 3
>2014/10/29 19:58:09.343004 [19587]: Thawing priority 1
>2014/10/29 19:58:09.343059 [19587]: Release freeze handler for prio 1
>2014/10/29 19:58:09.343088 [19587]: Thawing priority 2
>2014/10/29 19:58:09.343106 [19587]: Release freeze handler for prio 2
>2014/10/29 19:58:09.343131 [19587]: Thawing priority 3
>2014/10/29 19:58:09.343147 [19587]: Release freeze handler for prio 3
>2014/10/29 20:07:56.594391 [19587]: Freeze priority 1
>2014/10/29 20:07:56.607629 [19587]: Freeze priority 1
>2014/10/29 20:07:56.609710 [19587]: Freeze priority 1
>2014/10/29 20:07:56.610131 [19587]: Freeze priority 2
>2014/10/29 20:07:56.610454 [19587]: Freeze priority 2
>2014/10/29 20:07:56.610973 [19587]: Freeze priority 2
>2014/10/29 20:07:56.611333 [19587]: Freeze priority 3
>2014/10/29 20:07:56.611578 [19587]: Freeze priority 3
>2014/10/29 20:07:56.611633 [19587]: Freeze priority 3
>2014/10/29 20:07:59.638236 [19587]: Freeze priority 1
>2014/10/29 20:07:59.638876 [19587]: Freeze priority 2
>2014/10/29 20:07:59.639232 [19587]: Freeze priority 3
>2014/10/29 20:08:01.721927 [19587]: Thawing priority 1
>2014/10/29 20:08:01.721980 [19587]: Release freeze handler for prio 1
>2014/10/29 20:08:01.722028 [19587]: Thawing priority 2
>2014/10/29 20:08:01.722050 [19587]: Release freeze handler for prio 2
>2014/10/29 20:08:01.722090 [19587]: Thawing priority 3
>2014/10/29 20:08:01.722113 [19587]: Release freeze handler for prio 3
>2014/10/29 20:08:01.723472 [19587]: server/ctdb_call.c:1005 reqid 50163 not found
>2014/10/29 20:08:01.723518 [19587]: server/ctdb_call.c:1005 reqid 50164 not found
>2014/10/29 20:13:17.913201 [19587]: Freeze priority 1
>2014/10/29 20:13:17.930023 [19587]: Freeze priority 2
>2014/10/29 20:13:17.931784 [19587]: Freeze priority 3
>2014/10/29 20:13:20.606663 [19587]: Thawing priority 1
>2014/10/29 20:13:20.606716 [19587]: Release freeze handler for prio 1
>2014/10/29 20:13:20.606752 [19587]: Thawing priority 2
>2014/10/29 20:13:20.606772 [19587]: Release freeze handler for prio 2
>2014/10/29 20:13:20.606799 [19587]: Thawing priority 3
>2014/10/29 20:13:20.606815 [19587]: Release freeze handler for prio 3
>2014/10/29 20:18:01.185968 [19587]: Freeze priority 1
>2014/10/29 20:18:01.200509 [19587]: Freeze priority 2
>2014/10/29 20:18:01.201704 [19587]: Freeze priority 3
>2014/10/29 20:18:01.958173 [19587]: Freeze priority 1
>2014/10/29 20:18:01.958561 [19587]: Freeze priority 2
>2014/10/29 20:18:01.962885 [19587]: Freeze priority 3
>2014/10/29 20:18:01.964849 [19587]: Freeze priority 1
>2014/10/29 20:18:01.965385 [19587]: Freeze priority 2
>2014/10/29 20:18:01.965786 [19587]: Freeze priority 3
>2014/10/29 20:18:05.979314 [recoverd:19916]: Taking out recovery lock from recovery daemon
>2014/10/29 20:18:05.979384 [recoverd:19916]: Take the recovery lock
>2014/10/29 20:18:05.990859 [19587]: Freeze priority 1
>2014/10/29 20:18:05.991291 [19587]: Freeze priority 2
>2014/10/29 20:18:05.991732 [19587]: Freeze priority 3
>2014/10/29 20:18:07.677249 [19587]: Thawing priority 1
>2014/10/29 20:18:07.677316 [19587]: Release freeze handler for prio 1
>2014/10/29 20:18:07.677356 [19587]: Thawing priority 2
>2014/10/29 20:18:07.677378 [19587]: Release freeze handler for prio 2
>2014/10/29 20:18:07.677454 [19587]: Thawing priority 3
>2014/10/29 20:18:07.677471 [19587]: Release freeze handler for prio 3
>2014/10/29 20:18:08.265992 [recoverd:19916]: Resetting ban count to 0 for all nodes
>2014/10/29 20:24:35.795558 [recoverd:19916]: Taking out recovery lock from recovery daemon
>2014/10/29 20:24:35.803344 [recoverd:19916]: Take the recovery lock
>2014/10/29 20:24:35.955913 [19587]: Freeze priority 1
>2014/10/29 20:24:35.960140 [19587]: Freeze priority 2
>2014/10/29 20:24:35.964264 [19587]: Freeze priority 3
>2014/10/29 20:24:38.101830 [19587]: Thawing priority 1
>2014/10/29 20:24:38.101882 [19587]: Release freeze handler for prio 1
>2014/10/29 20:24:38.101917 [19587]: Thawing priority 2
>2014/10/29 20:24:38.101939 [19587]: Release freeze handler for prio 2
>2014/10/29 20:24:38.101969 [19587]: Thawing priority 3
>2014/10/29 20:24:38.101989 [19587]: Release freeze handler for prio 3
>2014/10/29 20:24:38.453923 [recoverd:19916]: Resetting ban count to 0 for all nodes
>2014/10/29 20:24:48.663352 [recoverd:19916]: server/ctdb_recoverd.c:3933 Remote node:1 has different flags for node 0. It has 0x02 vs our 0x00
>2014/10/29 20:24:48.663405 [recoverd:19916]: Use flags 0x00 from local recmaster node for cluster update of node 0 flags
>2014/10/29 20:24:48.665287 [recoverd:19916]: Taking out recovery lock from recovery daemon
>2014/10/29 20:24:48.665304 [recoverd:19916]: Take the recovery lock
>2014/10/29 20:24:48.717549 [19587]: Freeze priority 1
>2014/10/29 20:24:48.721595 [19587]: Freeze priority 2
>2014/10/29 20:24:48.723699 [19587]: Freeze priority 3
>2014/10/29 20:24:50.768575 [19587]: Thawing priority 1
>2014/10/29 20:24:50.768613 [19587]: Release freeze handler for prio 1
>2014/10/29 20:24:50.768654 [19587]: Thawing priority 2
>2014/10/29 20:24:50.768672 [19587]: Release freeze handler for prio 2
>2014/10/29 20:24:50.768702 [19587]: Thawing priority 3
>2014/10/29 20:24:50.768718 [19587]: Release freeze handler for prio 3
>2014/10/29 20:24:51.119071 [recoverd:19916]: Resetting ban count to 0 for all nodes
>2014/10/29 20:28:05.383085 [recovery-lock:28201]: failed read from recovery_lock_fd - Transport endpoint is not connected
>2014/10/29 20:28:05.383160 [recoverd:19916]: server/ctdb_recoverd.c:3349 reclock child process returned error 2
>2014/10/29 20:28:05.383189 [recoverd:19916]: server/ctdb_recoverd.c:3449 reclock child failed when checking file
>2014/10/29 20:28:05.383272 [recoverd:19916]: Failed check_recovery_lock. Force a recovery
>2014/10/29 20:28:05.383291 [recoverd:19916]: Taking out recovery lock from recovery daemon
>2014/10/29 20:28:05.383301 [recoverd:19916]: Take the recovery lock
>2014/10/29 20:28:05.421263 [19587]: Freeze priority 1
>2014/10/29 20:28:05.424264 [19587]: Freeze priority 2
>2014/10/29 20:28:05.426321 [19587]: Freeze priority 3
>2014/10/29 20:28:05.562588 [19587]: pnn 2 Invalid reqid 201907 in ctdb_reply_control
>2014/10/29 20:28:05.567449 [19587]: pnn 2 Invalid reqid 201906 in ctdb_reply_control
>2014/10/29 20:28:05.608227 [19587]: pnn 2 Invalid reqid 201908 in ctdb_reply_control
>2014/10/29 20:28:05.622262 [19587]: pnn 2 Invalid reqid 201905 in ctdb_reply_control
>2014/10/29 20:28:06.548117 [19587]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 20:28:06.559023 [19587]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 20:29:33.146117 [ 1500]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/29 20:29:33.146219 [ 1500]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 20:29:33.146233 [ 1500]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 20:33:33.828865 [ 7703]: Starting CTDBD (Version 2.5.3) as PID: 7703
>2014/10/29 20:33:35.385619 [ 7703]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 20:33:35.409261 [ 7703]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 20:33:35.423401 [ 7703]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 20:33:35.437382 [ 7703]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 20:33:35.437400 [ 7703]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/29 20:33:35.437409 [ 7703]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/29 20:33:35.437418 [ 7703]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/29 20:33:35.437437 [ 7703]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/29 20:33:35.437447 [ 7703]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/29 20:33:35.451438 [ 7703]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 20:33:35.465479 [ 7703]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 20:33:35.465498 [ 7703]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/29 20:33:35.465507 [ 7703]: Ignoring persistent database 'registry.tdb.1'
>2014/10/29 20:33:35.479491 [ 7703]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 20:33:35.479524 [ 7703]: Freeze priority 1
>2014/10/29 20:33:35.491053 [ 7703]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 20:33:35.497643 [ 7703]: Freeze priority 2
>2014/10/29 20:33:35.498034 [ 7703]: Freeze priority 3
>2014/10/29 20:33:35.661354 [ 7703]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 20:33:35.662102 [ 7703]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 20:33:35.665682 [ 7703]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 20:33:35.669034 [ 7703]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 20:33:35.732269 [ 7703]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 20:33:35.789485 [ 7703]: Freeze priority 1
>2014/10/29 20:33:35.789557 [ 7703]: Freeze priority 2
>2014/10/29 20:33:35.789611 [ 7703]: Freeze priority 3
>2014/10/29 20:33:39.797430 [recoverd: 7987]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 20:33:39.797509 [ 7703]: Freeze priority 1
>2014/10/29 20:33:39.797572 [ 7703]: Freeze priority 2
>2014/10/29 20:33:39.797626 [ 7703]: Freeze priority 3
>2014/10/29 20:33:44.593367 [ 7703]: Freeze priority 1
>2014/10/29 20:33:44.628185 [ 7703]: Freeze priority 2
>2014/10/29 20:33:44.629196 [ 7703]: Freeze priority 3
>2014/10/29 20:33:44.790635 [ 7703]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 20:33:44.791300 [ 7703]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 20:33:44.792238 [ 7703]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/29 20:33:46.662165 [ 7703]: Thawing priority 1
>2014/10/29 20:33:46.662216 [ 7703]: Release freeze handler for prio 1
>2014/10/29 20:33:46.662257 [ 7703]: Thawing priority 2
>2014/10/29 20:33:46.662277 [ 7703]: Release freeze handler for prio 2
>2014/10/29 20:33:46.662304 [ 7703]: Thawing priority 3
>2014/10/29 20:33:46.662322 [ 7703]: Release freeze handler for prio 3
>2014/10/29 20:34:00.838744 [recoverd: 7987]: Trigger takeoverrun
>2014/10/29 20:34:01.547095 [ 7703]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 20:34:01.869872 [ 7703]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 20:34:01.880712 [ 7703]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 20:34:01.927504 [ 7703]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 20:34:02.102264 [ 7703]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 20:34:04.552433 [ 7703]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/29 20:34:05.654833 [ 7703]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 20:38:07.987015 [ 7703]: Freeze priority 1
>2014/10/29 20:38:07.988688 [ 7703]: Freeze priority 1
>2014/10/29 20:38:08.133702 [ 7703]: Freeze priority 1
>2014/10/29 20:38:09.870904 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:38:24.871170 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:38:39.871412 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:38:54.871914 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:39:07.990231 [ 7703]: Freeze priority 1
>2014/10/29 20:39:07.991166 [ 7703]: Freeze priority 1
>2014/10/29 20:39:08.132278 [ 7703]: Recovery daemon ping timeout. Count : 0
>2014/10/29 20:39:08.134454 [recoverd: 7987]: ctdb_control error: 'ctdb_control timed out'
>2014/10/29 20:39:08.134515 [recoverd: 7987]: ctdb_control error: 'ctdb_control timed out'
>2014/10/29 20:39:08.134539 [recoverd: 7987]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/29 20:39:08.134555 [recoverd: 7987]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits
>2014/10/29 20:39:08.134573 [recoverd: 7987]: Async wait failed - fail_count=1
>2014/10/29 20:39:08.134587 [recoverd: 7987]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/29 20:39:08.134602 [recoverd: 7987]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/29 20:39:08.136121 [ 7703]: Freeze priority 1
>2014/10/29 20:39:09.872470 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:39:24.873425 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:39:39.873551 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:39:54.873814 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:40:08.134995 [ 7703]: Recovery daemon ping timeout. Count : 0
>2014/10/29 20:40:08.136375 [recoverd: 7987]: ctdb_control error: 'ctdb_control timed out'
>2014/10/29 20:40:08.136410 [recoverd: 7987]: ctdb_control error: 'ctdb_control timed out'
>2014/10/29 20:40:08.136426 [recoverd: 7987]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/29 20:40:08.136437 [recoverd: 7987]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits
>2014/10/29 20:40:08.136450 [recoverd: 7987]: Async wait failed - fail_count=1
>2014/10/29 20:40:08.136461 [recoverd: 7987]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/29 20:40:08.136472 [recoverd: 7987]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/29 20:40:08.136836 [ 7703]: Freeze priority 1
>2014/10/29 20:40:08.138207 [ 7703]: Freeze priority 1
>2014/10/29 20:40:08.145633 [ 7703]: pnn 2 Invalid reqid 12817 in ctdb_reply_control
>2014/10/29 20:40:08.145673 [ 7703]: pnn 2 Invalid reqid 12618 in ctdb_reply_control
>2014/10/29 20:40:08.145731 [ 7703]: Freeze priority 2
>2014/10/29 20:40:08.145881 [ 7703]: Freeze priority 2
>2014/10/29 20:40:08.146649 [ 7703]: Freeze priority 3
>2014/10/29 20:40:08.146787 [ 7703]: Freeze priority 3
>2014/10/29 20:40:11.166825 [ 7703]: Freeze priority 1
>2014/10/29 20:40:11.167189 [ 7703]: Freeze priority 2
>2014/10/29 20:40:11.167450 [ 7703]: Freeze priority 3
>2014/10/29 20:40:12.931813 [ 7703]: Thawing priority 1
>2014/10/29 20:40:12.931857 [ 7703]: Release freeze handler for prio 1
>2014/10/29 20:40:12.931891 [ 7703]: Thawing priority 2
>2014/10/29 20:40:12.931910 [ 7703]: Release freeze handler for prio 2
>2014/10/29 20:40:12.931939 [ 7703]: Thawing priority 3
>2014/10/29 20:40:12.931956 [ 7703]: Release freeze handler for prio 3
>2014/10/29 20:40:13.383420 [ 7703]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 20:40:38.747328 [ 7703]: Freeze priority 1
>2014/10/29 20:40:38.762646 [ 7703]: Freeze priority 2
>2014/10/29 20:40:38.769499 [ 7703]: Freeze priority 3
>2014/10/29 20:40:41.763907 [ 7703]: Thawing priority 1
>2014/10/29 20:40:41.763959 [ 7703]: Release freeze handler for prio 1
>2014/10/29 20:40:41.764002 [ 7703]: Thawing priority 2
>2014/10/29 20:40:41.764026 [ 7703]: Release freeze handler for prio 2
>2014/10/29 20:40:41.764071 [ 7703]: Thawing priority 3
>2014/10/29 20:40:41.764090 [ 7703]: Release freeze handler for prio 3
>2014/10/29 20:40:41.877602 [ 7703]: 10.interface: Killing TCP connection 10.10.10.205:54678 10.10.10.184:445
>2014/10/29 20:40:41.884118 [ 7703]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.184
>2014/10/29 20:40:41.893310 [ 7703]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1
>2014/10/29 20:40:42.347878 [ 7703]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 20:43:15.815605 [ 7703]: Freeze priority 1
>2014/10/29 20:43:15.830273 [ 7703]: Freeze priority 2
>2014/10/29 20:43:15.831558 [ 7703]: Freeze priority 3
>2014/10/29 20:43:18.927342 [ 7703]: Thawing priority 1
>2014/10/29 20:43:18.927384 [ 7703]: Release freeze handler for prio 1
>2014/10/29 20:43:18.927427 [ 7703]: Thawing priority 2
>2014/10/29 20:43:18.927449 [ 7703]: Release freeze handler for prio 2
>2014/10/29 20:43:18.927478 [ 7703]: Thawing priority 3
>2014/10/29 20:43:18.927497 [ 7703]: Release freeze handler for prio 3
>2014/10/29 20:43:36.724040 [ 7703]: Monitoring event was cancelled
>2014/10/29 20:53:15.123874 [ 7703]: Freeze priority 1
>2014/10/29 20:53:27.745637 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:53:42.746568 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:53:57.747386 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:54:12.748302 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:54:15.142595 [ 7703]: Freeze priority 1
>2014/10/29 20:54:27.748974 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:54:42.750051 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:54:57.750707 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:55:12.751741 [ 7703]: Skip monitoring since databases are frozen
>2014/10/29 20:55:15.323543 [ 7703]: Freeze priority 1
>2014/10/29 20:55:15.323863 [ 7703]: Freeze priority 2
>2014/10/29 20:55:15.336130 [ 7703]: Freeze priority 3
>2014/10/29 20:55:17.469643 [ 7703]: Thawing priority 1
>2014/10/29 20:55:17.469685 [ 7703]: Release freeze handler for prio 1
>2014/10/29 20:55:17.469717 [ 7703]: Thawing priority 2
>2014/10/29 20:55:17.469738 [ 7703]: Release freeze handler for prio 2
>2014/10/29 20:55:17.469770 [ 7703]: Thawing priority 3
>2014/10/29 20:55:17.469799 [ 7703]: Release freeze handler for prio 3
>2014/10/29 20:55:17.955612 [ 7703]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 20:55:45.609546 [ 7703]: Freeze priority 1
>2014/10/29 20:55:45.619347 [ 7703]: Freeze priority 2
>2014/10/29 20:55:45.622235 [ 7703]: Freeze priority 3
>2014/10/29 20:55:47.896212 [ 7703]: Thawing priority 1
>2014/10/29 20:55:47.896270 [ 7703]: Release freeze handler for prio 1
>2014/10/29 20:55:47.896301 [ 7703]: Thawing priority 2
>2014/10/29 20:55:47.896319 [ 7703]: Release freeze handler for prio 2
>2014/10/29 20:55:47.896358 [ 7703]: Thawing priority 3
>2014/10/29 20:55:47.896375 [ 7703]: Release freeze handler for prio 3
>2014/10/29 20:55:47.980705 [ 7703]: 10.interface: Killing TCP connection 10.10.10.206:49659 10.10.10.183:445
>2014/10/29 20:55:47.980836 [ 7703]: 10.interface: Killing TCP connection 10.10.10.205:54682 10.10.10.183:445
>2014/10/29 20:55:47.980937 [ 7703]: 10.interface: Killing TCP connection 10.10.10.208:49559 10.10.10.183:445
>2014/10/29 20:55:47.997639 [ 7703]: 10.interface: Killed 3 TCP connections to released IP 10.10.10.183
>2014/10/29 20:55:48.412545 [ 7703]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 20:58:35.926501 [ 7703]: Freeze priority 1
>2014/10/29 20:58:35.962450 [ 7703]: Freeze priority 2
>2014/10/29 20:58:35.964598 [ 7703]: Freeze priority 3
>2014/10/29 20:58:38.293222 [ 7703]: Thawing priority 1
>2014/10/29 20:58:38.293261 [ 7703]: Release freeze handler for prio 1
>2014/10/29 20:58:38.293291 [ 7703]: Thawing priority 2
>2014/10/29 20:58:38.293320 [ 7703]: Release freeze handler for prio 2
>2014/10/29 20:58:38.293350 [ 7703]: Thawing priority 3
>2014/10/29 20:58:38.293366 [ 7703]: Release freeze handler for prio 3
>2014/10/29 20:58:52.711848 [ 7703]: Monitoring event was cancelled
>2014/10/29 20:58:52.711911 [ 7703]: server/eventscript.c:569 Sending SIGTERM to child pid:22264
>2014/10/29 21:03:19.688685 [ 7703]: Freeze priority 1
>2014/10/29 21:03:19.701810 [ 7703]: Freeze priority 1
>2014/10/29 21:03:19.705917 [ 7703]: Freeze priority 2
>2014/10/29 21:03:19.709275 [ 7703]: Freeze priority 2
>2014/10/29 21:03:19.710564 [ 7703]: Freeze priority 3
>2014/10/29 21:03:19.715646 [ 7703]: Freeze priority 3
>2014/10/29 21:03:22.728977 [recoverd: 7987]: Taking out recovery lock from recovery daemon
>2014/10/29 21:03:22.729042 [recoverd: 7987]: Take the recovery lock
>2014/10/29 21:03:22.742293 [ 7703]: Freeze priority 1
>2014/10/29 21:03:22.742744 [ 7703]: Freeze priority 2
>2014/10/29 21:03:22.743127 [ 7703]: Freeze priority 3
>2014/10/29 21:03:24.968853 [ 7703]: Thawing priority 1
>2014/10/29 21:03:24.968895 [ 7703]: Release freeze handler for prio 1
>2014/10/29 21:03:24.968925 [ 7703]: Thawing priority 2
>2014/10/29 21:03:24.968941 [ 7703]: Release freeze handler for prio 2
>2014/10/29 21:03:24.968975 [ 7703]: Thawing priority 3
>2014/10/29 21:03:24.968990 [ 7703]: Release freeze handler for prio 3
>2014/10/29 21:03:25.536232 [recoverd: 7987]: Resetting ban count to 0 for all nodes
>2014/10/29 21:09:52.683379 [ 7703]: Freeze priority 1
>2014/10/29 21:09:52.758389 [ 7703]: Freeze priority 2
>2014/10/29 21:09:52.762013 [ 7703]: Freeze priority 3
>2014/10/29 21:09:55.000036 [ 7703]: Thawing priority 1
>2014/10/29 21:09:55.000091 [ 7703]: Release freeze handler for prio 1
>2014/10/29 21:09:55.000145 [ 7703]: Thawing priority 2
>2014/10/29 21:09:55.000173 [ 7703]: Release freeze handler for prio 2
>2014/10/29 21:09:55.000219 [ 7703]: Thawing priority 3
>2014/10/29 21:09:55.000239 [ 7703]: Release freeze handler for prio 3
>2014/10/29 21:13:20.539806 [ 7703]: 50.samba: ERROR: samba tcp port 445 is not responding
>2014/10/29 21:13:20.551343 [ 7703]: 50.samba: Redirecting to /bin/systemctl restart  smb.service
>2014/10/29 21:13:26.475886 [ 7703]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 21:13:26.489390 [ 7703]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 21:14:55.894453 [ 1545]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/29 21:14:55.894586 [ 1545]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 21:14:55.894607 [ 1545]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 21:18:57.121914 [ 7733]: Starting CTDBD (Version 2.5.3) as PID: 7733
>2014/10/29 21:18:58.522610 [ 7733]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 21:18:58.547699 [ 7733]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 21:18:58.562058 [ 7733]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 21:18:58.576048 [ 7733]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 21:18:58.576066 [ 7733]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/29 21:18:58.576076 [ 7733]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/29 21:18:58.576085 [ 7733]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/29 21:18:58.576094 [ 7733]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/29 21:18:58.576103 [ 7733]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/29 21:18:58.590021 [ 7733]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 21:18:58.604179 [ 7733]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 21:18:58.604198 [ 7733]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/29 21:18:58.604208 [ 7733]: Ignoring persistent database 'registry.tdb.1'
>2014/10/29 21:18:58.618380 [ 7733]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 21:18:58.618433 [ 7733]: Freeze priority 1
>2014/10/29 21:18:58.632379 [ 7733]: Freeze priority 2
>2014/10/29 21:18:58.632742 [ 7733]: Freeze priority 3
>2014/10/29 21:18:58.796205 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 21:18:58.796293 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 21:18:58.796315 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 21:18:58.797398 [ 7733]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 21:18:58.801127 [ 7733]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 21:18:58.804912 [ 7733]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 21:18:58.931054 [ 7733]: Freeze priority 1
>2014/10/29 21:18:58.931132 [ 7733]: Freeze priority 2
>2014/10/29 21:18:58.931186 [ 7733]: Freeze priority 3
>2014/10/29 21:18:59.074947 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 21:18:59.074996 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 21:18:59.075019 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 21:18:59.075056 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 21:18:59.075205 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node
>2014/10/29 21:18:59.115692 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 21:18:59.115805 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node
>2014/10/29 21:18:59.115832 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x6afb8c09 on local node
>2014/10/29 21:18:59.115845 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x4e66c2b2 on local node
>2014/10/29 21:18:59.115856 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x6afb8c09 on local node
>2014/10/29 21:18:59.115869 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node
>2014/10/29 21:19:03.438871 [recoverd: 8019]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 21:19:03.438927 [ 7733]: Freeze priority 1
>2014/10/29 21:19:03.438986 [ 7733]: Freeze priority 2
>2014/10/29 21:19:03.439038 [ 7733]: Freeze priority 3
>2014/10/29 21:19:07.568045 [ 7733]: Freeze priority 1
>2014/10/29 21:19:07.584929 [ 7733]: Freeze priority 2
>2014/10/29 21:19:07.586052 [ 7733]: Freeze priority 3
>2014/10/29 21:19:07.927762 [ 7733]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 21:19:07.928596 [ 7733]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 21:19:07.929713 [ 7733]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/29 21:19:13.202457 [ 7733]: Handling event took 4 seconds!
>2014/10/29 21:19:13.214740 [ 7733]: Thawing priority 1
>2014/10/29 21:19:13.214790 [ 7733]: Release freeze handler for prio 1
>2014/10/29 21:19:13.214825 [ 7733]: Thawing priority 2
>2014/10/29 21:19:13.214844 [ 7733]: Release freeze handler for prio 2
>2014/10/29 21:19:13.214871 [ 7733]: Thawing priority 3
>2014/10/29 21:19:13.214890 [ 7733]: Release freeze handler for prio 3
>2014/10/29 21:19:28.229571 [recoverd: 8019]: Trigger takeoverrun
>2014/10/29 21:19:28.654289 [ 7733]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 21:19:28.937295 [ 7733]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 21:19:28.951032 [ 7733]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 21:19:29.077939 [ 7733]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 21:19:29.282157 [ 7733]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 21:19:32.658785 [ 7733]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/29 21:19:33.785605 [ 7733]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 21:23:30.683205 [ 7733]: Freeze priority 1
>2014/10/29 21:23:30.691674 [ 7733]: Freeze priority 2
>2014/10/29 21:23:30.692923 [ 7733]: Freeze priority 3
>2014/10/29 21:23:32.390311 [ 7733]: Thawing priority 1
>2014/10/29 21:23:32.390360 [ 7733]: Release freeze handler for prio 1
>2014/10/29 21:23:32.390391 [ 7733]: Thawing priority 2
>2014/10/29 21:23:32.390420 [ 7733]: Release freeze handler for prio 2
>2014/10/29 21:23:32.390455 [ 7733]: Thawing priority 3
>2014/10/29 21:23:32.390474 [ 7733]: Release freeze handler for prio 3
>2014/10/29 21:28:42.503439 [ 7733]: Freeze priority 1
>2014/10/29 21:28:42.550319 [ 7733]: Freeze priority 2
>2014/10/29 21:28:42.551750 [ 7733]: Freeze priority 3
>2014/10/29 21:28:48.214657 [ 7733]: Thawing priority 1
>2014/10/29 21:28:48.214712 [ 7733]: Release freeze handler for prio 1
>2014/10/29 21:28:48.214755 [ 7733]: Thawing priority 2
>2014/10/29 21:28:48.214774 [ 7733]: Release freeze handler for prio 2
>2014/10/29 21:28:48.214800 [ 7733]: Thawing priority 3
>2014/10/29 21:28:48.214827 [ 7733]: Release freeze handler for prio 3
>2014/10/29 21:38:44.119638 [recoverd: 8019]: ctdb_control error: 'node is disconnected'
>2014/10/29 21:38:44.184309 [recoverd: 8019]: client/ctdb_client.c:1535 ctdb_control for getnodes failed ret:-1 res:-1
>2014/10/29 21:38:44.184348 [recoverd: 8019]: server/ctdb_recoverd.c:3739 Unable to get nodemap from recovery master 0
>2014/10/29 21:38:44.188220 [ 7733]: Freeze priority 1
>2014/10/29 21:38:44.306435 [ 7733]: Freeze priority 2
>2014/10/29 21:38:44.318573 [ 7733]: Freeze priority 3
>2014/10/29 21:38:48.102867 [ 7733]: Freeze priority 1
>2014/10/29 21:38:48.103309 [ 7733]: Freeze priority 2
>2014/10/29 21:38:48.103699 [ 7733]: Freeze priority 3
>2014/10/29 21:38:50.545125 [ 7733]: Thawing priority 1
>2014/10/29 21:38:50.545182 [ 7733]: Release freeze handler for prio 1
>2014/10/29 21:38:50.545229 [ 7733]: Thawing priority 2
>2014/10/29 21:38:50.545250 [ 7733]: Release freeze handler for prio 2
>2014/10/29 21:38:50.545293 [ 7733]: Thawing priority 3
>2014/10/29 21:38:50.545315 [ 7733]: Release freeze handler for prio 3
>2014/10/29 21:44:21.726587 [ 7733]: Freeze priority 1
>2014/10/29 21:44:21.833622 [ 7733]: Freeze priority 2
>2014/10/29 21:44:21.835851 [ 7733]: Freeze priority 3
>2014/10/29 21:44:27.149173 [ 7733]: Thawing priority 1
>2014/10/29 21:44:27.149206 [ 7733]: Release freeze handler for prio 1
>2014/10/29 21:44:27.149249 [ 7733]: Thawing priority 2
>2014/10/29 21:44:27.149272 [ 7733]: Release freeze handler for prio 2
>2014/10/29 21:44:27.149312 [ 7733]: Thawing priority 3
>2014/10/29 21:44:27.149332 [ 7733]: Release freeze handler for prio 3
>2014/10/29 21:48:49.509367 [ 7733]: Freeze priority 1
>2014/10/29 21:48:49.546322 [ 7733]: Freeze priority 1
>2014/10/29 21:48:49.613279 [ 7733]: Freeze priority 2
>2014/10/29 21:48:49.616277 [ 7733]: Freeze priority 2
>2014/10/29 21:48:49.618482 [ 7733]: Freeze priority 3
>2014/10/29 21:48:49.621113 [ 7733]: Freeze priority 3
>2014/10/29 21:48:53.135398 [recoverd: 8019]: Taking out recovery lock from recovery daemon
>2014/10/29 21:48:53.135459 [recoverd: 8019]: Take the recovery lock
>2014/10/29 21:48:53.162481 [ 7733]: Freeze priority 1
>2014/10/29 21:48:53.162905 [ 7733]: Freeze priority 2
>2014/10/29 21:48:53.163261 [ 7733]: Freeze priority 3
>2014/10/29 21:48:55.633723 [ 7733]: Thawing priority 1
>2014/10/29 21:48:55.633763 [ 7733]: Release freeze handler for prio 1
>2014/10/29 21:48:55.633793 [ 7733]: Thawing priority 2
>2014/10/29 21:48:55.633823 [ 7733]: Release freeze handler for prio 2
>2014/10/29 21:48:55.633851 [ 7733]: Thawing priority 3
>2014/10/29 21:48:55.633867 [ 7733]: Release freeze handler for prio 3
>2014/10/29 21:48:56.125072 [recoverd: 8019]: Resetting ban count to 0 for all nodes
>2014/10/29 21:55:25.664194 [recoverd: 8019]: Taking out recovery lock from recovery daemon
>2014/10/29 21:55:25.680005 [recoverd: 8019]: Take the recovery lock
>2014/10/29 21:55:25.894437 [ 7733]: Freeze priority 1
>2014/10/29 21:55:25.971850 [ 7733]: Freeze priority 2
>2014/10/29 21:55:25.974183 [ 7733]: Freeze priority 3
>2014/10/29 21:55:28.587627 [ 7733]: Thawing priority 1
>2014/10/29 21:55:28.587672 [ 7733]: Release freeze handler for prio 1
>2014/10/29 21:55:28.587708 [ 7733]: Thawing priority 2
>2014/10/29 21:55:28.587727 [ 7733]: Release freeze handler for prio 2
>2014/10/29 21:55:28.587754 [ 7733]: Thawing priority 3
>2014/10/29 21:55:28.587770 [ 7733]: Release freeze handler for prio 3
>2014/10/29 21:55:29.050719 [recoverd: 8019]: Resetting ban count to 0 for all nodes
>2014/10/29 21:58:57.499627 [ 7733]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 21:58:57.562483 [ 7733]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/29 21:58:57.562514 [ 7733]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 22:00:22.787574 [ 1544]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/29 22:00:22.787679 [ 1544]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 22:00:22.787693 [ 1544]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 22:04:24.642995 [ 7695]: Starting CTDBD (Version 2.5.3) as PID: 7695
>2014/10/29 22:04:26.406387 [ 7695]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 22:04:26.429877 [ 7695]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 22:04:26.444024 [ 7695]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 22:04:26.457952 [ 7695]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 22:04:26.457970 [ 7695]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/29 22:04:26.457978 [ 7695]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/29 22:04:26.457987 [ 7695]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/29 22:04:26.457995 [ 7695]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/29 22:04:26.458003 [ 7695]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/29 22:04:26.471922 [ 7695]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 22:04:26.485876 [ 7695]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 22:04:26.485895 [ 7695]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/29 22:04:26.485903 [ 7695]: Ignoring persistent database 'registry.tdb.1'
>2014/10/29 22:04:26.499876 [ 7695]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 22:04:26.499909 [ 7695]: Freeze priority 1
>2014/10/29 22:04:26.518342 [ 7695]: Freeze priority 2
>2014/10/29 22:04:26.518693 [ 7695]: Freeze priority 3
>2014/10/29 22:04:26.682010 [ 7695]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 22:04:26.682978 [ 7695]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 22:04:26.686767 [ 7695]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 22:04:26.690108 [ 7695]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 22:04:26.809850 [ 7695]: Freeze priority 1
>2014/10/29 22:04:26.809925 [ 7695]: Freeze priority 2
>2014/10/29 22:04:26.809981 [ 7695]: Freeze priority 3
>2014/10/29 22:04:30.817245 [recoverd: 8069]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 22:04:30.872895 [ 7695]: Freeze priority 1
>2014/10/29 22:04:30.873007 [ 7695]: Freeze priority 2
>2014/10/29 22:04:30.873082 [ 7695]: Freeze priority 3
>2014/10/29 22:04:34.989486 [ 7695]: Freeze priority 1
>2014/10/29 22:04:35.013016 [ 7695]: Freeze priority 2
>2014/10/29 22:04:35.015139 [ 7695]: Freeze priority 3
>2014/10/29 22:04:35.183761 [ 7695]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 22:04:35.184917 [ 7695]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 22:04:35.186958 [ 7695]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/29 22:04:39.731992 [ 7695]: Handling event took 4 seconds!
>2014/10/29 22:04:39.737870 [ 7695]: Thawing priority 1
>2014/10/29 22:04:39.737924 [ 7695]: Release freeze handler for prio 1
>2014/10/29 22:04:39.737971 [ 7695]: Thawing priority 2
>2014/10/29 22:04:39.738001 [ 7695]: Release freeze handler for prio 2
>2014/10/29 22:04:39.738044 [ 7695]: Thawing priority 3
>2014/10/29 22:04:39.738062 [ 7695]: Release freeze handler for prio 3
>2014/10/29 22:04:54.759553 [recoverd: 8069]: Trigger takeoverrun
>2014/10/29 22:04:54.891173 [ 7695]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 22:04:55.332796 [ 7695]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 22:04:55.346536 [ 7695]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 22:04:55.381160 [ 7695]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 22:04:55.560516 [ 7695]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 22:04:57.993373 [ 7695]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/29 22:04:58.607435 [ 7695]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 22:09:02.089179 [ 7695]: Freeze priority 1
>2014/10/29 22:09:02.095662 [ 7695]: Freeze priority 1
>2014/10/29 22:09:02.105818 [ 7695]: Freeze priority 1
>2014/10/29 22:09:04.215481 [ 7695]: Skip monitoring since databases are frozen
>2014/10/29 22:09:19.216271 [ 7695]: Skip monitoring since databases are frozen
>2014/10/29 22:09:34.216980 [ 7695]: Skip monitoring since databases are frozen
>2014/10/29 22:09:49.217970 [ 7695]: Skip monitoring since databases are frozen
>2014/10/29 22:10:02.092533 [ 7695]: Freeze priority 1
>2014/10/29 22:10:02.095650 [ 7695]: Recovery daemon ping timeout. Count : 0
>2014/10/29 22:10:02.095760 [recoverd: 8069]: ctdb_control error: 'ctdb_control timed out'
>2014/10/29 22:10:02.095794 [recoverd: 8069]: ctdb_control error: 'ctdb_control timed out'
>2014/10/29 22:10:02.095811 [recoverd: 8069]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/29 22:10:02.095823 [recoverd: 8069]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits
>2014/10/29 22:10:02.095837 [recoverd: 8069]: Async wait failed - fail_count=1
>2014/10/29 22:10:02.095860 [recoverd: 8069]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/29 22:10:02.095875 [recoverd: 8069]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/29 22:10:02.097114 [ 7695]: Freeze priority 1
>2014/10/29 22:10:02.107117 [ 7695]: Freeze priority 1
>2014/10/29 22:10:04.218509 [ 7695]: Skip monitoring since databases are frozen
>2014/10/29 22:10:19.218773 [ 7695]: Skip monitoring since databases are frozen
>2014/10/29 22:10:34.219457 [ 7695]: Skip monitoring since databases are frozen
>2014/10/29 22:10:49.220503 [ 7695]: Skip monitoring since databases are frozen
>2014/10/29 22:11:02.096430 [ 7695]: Recovery daemon ping timeout. Count : 0
>2014/10/29 22:11:02.097581 [recoverd: 8069]: ctdb_control error: 'ctdb_control timed out'
>2014/10/29 22:11:02.097618 [recoverd: 8069]: ctdb_control error: 'ctdb_control timed out'
>2014/10/29 22:11:02.097634 [recoverd: 8069]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/29 22:11:02.097645 [recoverd: 8069]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits
>2014/10/29 22:11:02.097657 [recoverd: 8069]: Async wait failed - fail_count=1
>2014/10/29 22:11:02.097668 [recoverd: 8069]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/29 22:11:02.097678 [recoverd: 8069]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/29 22:11:02.171284 [ 7695]: pnn 2 Invalid reqid 7683 in ctdb_reply_control
>2014/10/29 22:11:02.171321 [ 7695]: pnn 2 Invalid reqid 7474 in ctdb_reply_control
>2014/10/29 22:11:02.174281 [ 7695]: Freeze priority 1
>2014/10/29 22:11:02.174889 [ 7695]: Freeze priority 2
>2014/10/29 22:11:02.176002 [ 7695]: Freeze priority 3
>2014/10/29 22:11:02.282396 [ 7695]: Freeze priority 1
>2014/10/29 22:11:02.282894 [ 7695]: Freeze priority 2
>2014/10/29 22:11:02.283393 [ 7695]: Freeze priority 3
>2014/10/29 22:11:05.300510 [ 7695]: Freeze priority 1
>2014/10/29 22:11:05.300694 [ 7695]: Freeze priority 2
>2014/10/29 22:11:05.300903 [ 7695]: Freeze priority 3
>2014/10/29 22:11:08.089106 [ 7695]: Thawing priority 1
>2014/10/29 22:11:08.089157 [ 7695]: Release freeze handler for prio 1
>2014/10/29 22:11:08.089204 [ 7695]: Thawing priority 2
>2014/10/29 22:11:08.089238 [ 7695]: Release freeze handler for prio 2
>2014/10/29 22:11:08.089278 [ 7695]: Thawing priority 3
>2014/10/29 22:11:08.089299 [ 7695]: Release freeze handler for prio 3
>2014/10/29 22:11:08.091596 [ 7695]: server/ctdb_call.c:1005 reqid 7956 not found
>2014/10/29 22:11:08.091645 [ 7695]: server/ctdb_call.c:1005 reqid 7957 not found
>2014/10/29 22:11:08.091668 [ 7695]: server/ctdb_call.c:1005 reqid 7958 not found
>2014/10/29 22:11:08.091689 [ 7695]: server/ctdb_call.c:1005 reqid 7959 not found
>2014/10/29 22:11:08.501152 [ 7695]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 22:11:32.728170 [ 7695]: Freeze priority 1
>2014/10/29 22:11:32.735922 [ 7695]: Freeze priority 2
>2014/10/29 22:11:32.737233 [ 7695]: Freeze priority 3
>2014/10/29 22:11:37.307586 [ 7695]: Thawing priority 1
>2014/10/29 22:11:37.307639 [ 7695]: Release freeze handler for prio 1
>2014/10/29 22:11:37.307670 [ 7695]: Thawing priority 2
>2014/10/29 22:11:37.307687 [ 7695]: Release freeze handler for prio 2
>2014/10/29 22:11:37.307716 [ 7695]: Thawing priority 3
>2014/10/29 22:11:37.307732 [ 7695]: Release freeze handler for prio 3
>2014/10/29 22:11:37.964911 [ 7695]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 22:14:07.433182 [vacuum-locking.tdb: 4972]: Error storing record copies on node 3: ret[0] res[-1]
>2014/10/29 22:14:11.442373 [ 7695]: Freeze priority 1
>2014/10/29 22:14:11.495362 [ 7695]: Freeze priority 2
>2014/10/29 22:14:11.497606 [ 7695]: Freeze priority 3
>2014/10/29 22:14:15.511627 [ 7695]: Thawing priority 1
>2014/10/29 22:14:15.511691 [ 7695]: Release freeze handler for prio 1
>2014/10/29 22:14:15.511729 [ 7695]: Thawing priority 2
>2014/10/29 22:14:15.511747 [ 7695]: Release freeze handler for prio 2
>2014/10/29 22:14:15.511774 [ 7695]: Thawing priority 3
>2014/10/29 22:14:15.511791 [ 7695]: Release freeze handler for prio 3
>2014/10/29 22:24:11.727635 [ 7695]: Freeze priority 1
>2014/10/29 22:24:11.835898 [ 7695]: Freeze priority 2
>2014/10/29 22:24:11.837055 [ 7695]: Freeze priority 3
>2014/10/29 22:24:14.602534 [ 7695]: Thawing priority 1
>2014/10/29 22:24:14.602598 [ 7695]: Release freeze handler for prio 1
>2014/10/29 22:24:14.602642 [ 7695]: Thawing priority 2
>2014/10/29 22:24:14.602660 [ 7695]: Release freeze handler for prio 2
>2014/10/29 22:24:14.602686 [ 7695]: Thawing priority 3
>2014/10/29 22:24:14.602701 [ 7695]: Release freeze handler for prio 3
>2014/10/29 22:29:28.932187 [ 7695]: Freeze priority 1
>2014/10/29 22:29:29.052498 [ 7695]: Freeze priority 2
>2014/10/29 22:29:29.054779 [ 7695]: Freeze priority 3
>2014/10/29 22:29:34.468102 [ 7695]: Thawing priority 1
>2014/10/29 22:29:34.468149 [ 7695]: Release freeze handler for prio 1
>2014/10/29 22:29:34.468183 [ 7695]: Thawing priority 2
>2014/10/29 22:29:34.468200 [ 7695]: Release freeze handler for prio 2
>2014/10/29 22:29:34.468227 [ 7695]: Thawing priority 3
>2014/10/29 22:29:34.468243 [ 7695]: Release freeze handler for prio 3
>2014/10/29 22:34:20.295762 [ 7695]: Freeze priority 1
>2014/10/29 22:34:20.597091 [ 7695]: Freeze priority 1
>2014/10/29 22:34:20.597650 [ 7695]: Freeze priority 2
>2014/10/29 22:34:20.600329 [ 7695]: Freeze priority 3
>2014/10/29 22:34:23.612575 [recoverd: 8069]: Taking out recovery lock from recovery daemon
>2014/10/29 22:34:23.612621 [recoverd: 8069]: Take the recovery lock
>2014/10/29 22:34:23.653004 [ 7695]: Freeze priority 1
>2014/10/29 22:34:23.653841 [ 7695]: Freeze priority 2
>2014/10/29 22:34:23.655403 [ 7695]: Freeze priority 3
>2014/10/29 22:34:32.997051 [ 7695]: Handling event took 6 seconds!
>2014/10/29 22:34:33.002155 [ 7695]: Thawing priority 1
>2014/10/29 22:34:33.002183 [ 7695]: Release freeze handler for prio 1
>2014/10/29 22:34:33.002213 [ 7695]: Thawing priority 2
>2014/10/29 22:34:33.002235 [ 7695]: Release freeze handler for prio 2
>2014/10/29 22:34:33.002265 [ 7695]: Thawing priority 3
>2014/10/29 22:34:33.002282 [ 7695]: Release freeze handler for prio 3
>2014/10/29 22:34:33.004932 [ 7695]: pnn 2 Invalid reqid 64206 in ctdb_become_dmaster from node 0
>2014/10/29 22:34:33.005017 [ 7695]: server/ctdb_call.c:1005 reqid 64207 not found
>2014/10/29 22:34:33.597318 [recoverd: 8069]: Resetting ban count to 0 for all nodes
>2014/10/29 22:41:01.148617 [recoverd: 8069]: Taking out recovery lock from recovery daemon
>2014/10/29 22:41:01.168964 [recoverd: 8069]: Take the recovery lock
>2014/10/29 22:41:01.386753 [ 7695]: Freeze priority 1
>2014/10/29 22:41:01.435934 [ 7695]: Freeze priority 2
>2014/10/29 22:41:01.440804 [ 7695]: Freeze priority 3
>2014/10/29 22:41:05.960833 [ 7695]: Thawing priority 1
>2014/10/29 22:41:05.960897 [ 7695]: Release freeze handler for prio 1
>2014/10/29 22:41:05.960936 [ 7695]: Thawing priority 2
>2014/10/29 22:41:05.960955 [ 7695]: Release freeze handler for prio 2
>2014/10/29 22:41:05.960983 [ 7695]: Thawing priority 3
>2014/10/29 22:41:05.961000 [ 7695]: Release freeze handler for prio 3
>2014/10/29 22:41:06.336975 [recoverd: 8069]: Resetting ban count to 0 for all nodes
>2014/10/29 22:41:23.735790 [ 7695]: Monitoring event was cancelled
>2014/10/29 22:41:23.735866 [ 7695]: server/eventscript.c:569 Sending SIGTERM to child pid:22549
>2014/10/29 22:44:30.967991 [ 7695]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 22:44:31.028272 [ 7695]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 22:45:56.442406 [ 1547]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/29 22:45:56.442515 [ 1547]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 22:45:56.522882 [ 1547]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 22:49:57.568477 [ 7729]: Starting CTDBD (Version 2.5.3) as PID: 7729
>2014/10/29 22:49:59.050435 [ 7729]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 22:49:59.073952 [ 7729]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 22:49:59.088288 [ 7729]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 22:49:59.102299 [ 7729]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 22:49:59.102318 [ 7729]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/29 22:49:59.102327 [ 7729]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/29 22:49:59.102336 [ 7729]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/29 22:49:59.102344 [ 7729]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/29 22:49:59.102353 [ 7729]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/29 22:49:59.116256 [ 7729]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 22:49:59.130246 [ 7729]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 22:49:59.130264 [ 7729]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/29 22:49:59.130273 [ 7729]: Ignoring persistent database 'registry.tdb.1'
>2014/10/29 22:49:59.144231 [ 7729]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 22:49:59.144261 [ 7729]: Freeze priority 1
>2014/10/29 22:49:59.162305 [ 7729]: Freeze priority 2
>2014/10/29 22:49:59.162649 [ 7729]: Freeze priority 3
>2014/10/29 22:49:59.326011 [ 7729]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 22:49:59.327161 [ 7729]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 22:49:59.330731 [ 7729]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 22:49:59.334264 [ 7729]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 22:49:59.454141 [ 7729]: Freeze priority 1
>2014/10/29 22:49:59.454220 [ 7729]: Freeze priority 2
>2014/10/29 22:49:59.454274 [ 7729]: Freeze priority 3
>2014/10/29 22:49:59.875144 [ 7729]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 22:49:59.889926 [ 7729]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 22:50:03.461069 [recoverd: 8044]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 22:50:03.461151 [ 7729]: Freeze priority 1
>2014/10/29 22:50:03.461224 [ 7729]: Freeze priority 2
>2014/10/29 22:50:03.461285 [ 7729]: Freeze priority 3
>2014/10/29 22:50:07.979011 [ 7729]: Freeze priority 1
>2014/10/29 22:50:08.005448 [ 7729]: Freeze priority 2
>2014/10/29 22:50:08.011733 [ 7729]: Freeze priority 3
>2014/10/29 22:50:08.190335 [ 7729]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 22:50:08.195204 [ 7729]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 22:50:08.204382 [ 7729]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/29 22:50:10.797286 [ 7729]: Thawing priority 1
>2014/10/29 22:50:10.797325 [ 7729]: Release freeze handler for prio 1
>2014/10/29 22:50:10.797358 [ 7729]: Thawing priority 2
>2014/10/29 22:50:10.797379 [ 7729]: Release freeze handler for prio 2
>2014/10/29 22:50:10.797405 [ 7729]: Thawing priority 3
>2014/10/29 22:50:10.797424 [ 7729]: Release freeze handler for prio 3
>2014/10/29 22:50:24.816503 [recoverd: 8044]: Trigger takeoverrun
>2014/10/29 22:50:25.402247 [ 7729]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 22:50:25.924084 [ 7729]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 22:50:25.924127 [ 7729]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 22:50:25.924143 [ 7729]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 22:50:25.936175 [ 7729]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 22:50:28.408966 [ 7729]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/29 22:50:29.641802 [ 7729]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 22:54:35.142201 [ 7729]: Freeze priority 1
>2014/10/29 22:54:35.152126 [ 7729]: Freeze priority 1
>2014/10/29 22:54:35.169292 [ 7729]: Freeze priority 1
>2014/10/29 22:54:35.175011 [ 7729]: Freeze priority 2
>2014/10/29 22:54:35.175283 [ 7729]: Freeze priority 2
>2014/10/29 22:54:35.175457 [ 7729]: Freeze priority 2
>2014/10/29 22:54:35.176254 [ 7729]: Freeze priority 3
>2014/10/29 22:54:35.178201 [ 7729]: Freeze priority 3
>2014/10/29 22:54:35.178352 [ 7729]: Freeze priority 3
>2014/10/29 22:54:38.207528 [ 7729]: Freeze priority 1
>2014/10/29 22:54:38.207817 [ 7729]: Freeze priority 2
>2014/10/29 22:54:38.208081 [ 7729]: Freeze priority 3
>2014/10/29 22:54:41.337852 [ 7729]: Thawing priority 1
>2014/10/29 22:54:41.337907 [ 7729]: Release freeze handler for prio 1
>2014/10/29 22:54:41.337939 [ 7729]: Thawing priority 2
>2014/10/29 22:54:41.337960 [ 7729]: Release freeze handler for prio 2
>2014/10/29 22:54:41.337991 [ 7729]: Thawing priority 3
>2014/10/29 22:54:41.338011 [ 7729]: Release freeze handler for prio 3
>2014/10/29 22:59:38.444753 [ 7729]: Freeze priority 1
>2014/10/29 22:59:38.455755 [ 7729]: Freeze priority 2
>2014/10/29 22:59:38.460534 [ 7729]: Freeze priority 3
>2014/10/29 22:59:41.557976 [ 7729]: Thawing priority 1
>2014/10/29 22:59:41.558028 [ 7729]: Release freeze handler for prio 1
>2014/10/29 22:59:41.558064 [ 7729]: Thawing priority 2
>2014/10/29 22:59:41.558084 [ 7729]: Release freeze handler for prio 2
>2014/10/29 22:59:41.558113 [ 7729]: Thawing priority 3
>2014/10/29 22:59:41.558131 [ 7729]: Release freeze handler for prio 3
>2014/10/29 23:09:35.333728 [ 7729]: Freeze priority 1
>2014/10/29 23:09:35.336905 [ 7729]: Freeze priority 1
>2014/10/29 23:09:35.343959 [ 7729]: Freeze priority 2
>2014/10/29 23:09:35.346200 [ 7729]: Freeze priority 2
>2014/10/29 23:09:35.347036 [ 7729]: Freeze priority 3
>2014/10/29 23:09:35.350977 [ 7729]: Freeze priority 3
>2014/10/29 23:09:38.372768 [ 7729]: Freeze priority 1
>2014/10/29 23:09:38.373171 [ 7729]: Freeze priority 2
>2014/10/29 23:09:38.373455 [ 7729]: Freeze priority 3
>2014/10/29 23:09:41.167856 [ 7729]: Thawing priority 1
>2014/10/29 23:09:41.167907 [ 7729]: Release freeze handler for prio 1
>2014/10/29 23:09:41.167944 [ 7729]: Thawing priority 2
>2014/10/29 23:09:41.167964 [ 7729]: Release freeze handler for prio 2
>2014/10/29 23:09:41.167992 [ 7729]: Thawing priority 3
>2014/10/29 23:09:41.168011 [ 7729]: Release freeze handler for prio 3
>2014/10/29 23:14:50.614114 [recoverd: 8044]: Taking out recovery lock from recovery daemon
>2014/10/29 23:14:50.614200 [recoverd: 8044]: Take the recovery lock
>2014/10/29 23:14:50.786906 [ 7729]: Freeze priority 1
>2014/10/29 23:14:50.803207 [ 7729]: Freeze priority 2
>2014/10/29 23:14:50.807766 [ 7729]: Freeze priority 3
>2014/10/29 23:14:53.623758 [ 7729]: Thawing priority 1
>2014/10/29 23:14:53.623821 [ 7729]: Release freeze handler for prio 1
>2014/10/29 23:14:53.623869 [ 7729]: Thawing priority 2
>2014/10/29 23:14:53.623889 [ 7729]: Release freeze handler for prio 2
>2014/10/29 23:14:53.623915 [ 7729]: Thawing priority 3
>2014/10/29 23:14:53.623930 [ 7729]: Release freeze handler for prio 3
>2014/10/29 23:14:53.631706 [recoverd: 8044]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:14:53.631746 [recoverd: 8044]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:14:53.631765 [recoverd: 8044]: Async operation failed with ret=-1 res=-1 opcode=16
>2014/10/29 23:14:53.634033 [set_recmode:20412]: ERROR: recovery lock file /mnt/lock/lockfile not locked when recovering!
>2014/10/29 23:14:53.634232 [recoverd: 8044]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:14:53.634266 [recoverd: 8044]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:14:53.634290 [recoverd: 8044]: Async operation failed with ret=-1 res=-1 opcode=16
>2014/10/29 23:14:53.634334 [recoverd: 8044]: Async wait failed - fail_count=2
>2014/10/29 23:14:53.634360 [recoverd: 8044]: server/ctdb_recoverd.c:412 Unable to set recovery mode. Recovery failed.
>2014/10/29 23:14:53.634380 [recoverd: 8044]: server/ctdb_recoverd.c:1996 Unable to set recovery mode to normal on cluster
>2014/10/29 23:14:56.659434 [recoverd: 8044]: Taking out recovery lock from recovery daemon
>2014/10/29 23:14:56.659501 [recoverd: 8044]: Take the recovery lock
>2014/10/29 23:14:56.708790 [ 7729]: Freeze priority 1
>2014/10/29 23:14:56.710013 [ 7729]: Freeze priority 2
>2014/10/29 23:14:56.710958 [ 7729]: Freeze priority 3
>2014/10/29 23:14:59.321285 [ 7729]: Thawing priority 1
>2014/10/29 23:14:59.321341 [ 7729]: Release freeze handler for prio 1
>2014/10/29 23:14:59.321377 [ 7729]: Thawing priority 2
>2014/10/29 23:14:59.321395 [ 7729]: Release freeze handler for prio 2
>2014/10/29 23:14:59.321425 [ 7729]: Thawing priority 3
>2014/10/29 23:14:59.321441 [ 7729]: Release freeze handler for prio 3
>2014/10/29 23:14:59.735550 [recoverd: 8044]: Resetting ban count to 0 for all nodes
>2014/10/29 23:19:41.107313 [recoverd: 8044]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 4 vs 3
>2014/10/29 23:19:41.121047 [recoverd: 8044]: Taking out recovery lock from recovery daemon
>2014/10/29 23:19:41.121072 [recoverd: 8044]: Take the recovery lock
>2014/10/29 23:19:41.176354 [ 7729]: Freeze priority 1
>2014/10/29 23:19:41.236485 [ 7729]: Freeze priority 2
>2014/10/29 23:19:41.239826 [ 7729]: Freeze priority 3
>2014/10/29 23:19:44.020922 [ 7729]: Thawing priority 1
>2014/10/29 23:19:44.020972 [ 7729]: Release freeze handler for prio 1
>2014/10/29 23:19:44.021023 [ 7729]: Thawing priority 2
>2014/10/29 23:19:44.021043 [ 7729]: Release freeze handler for prio 2
>2014/10/29 23:19:44.021072 [ 7729]: Thawing priority 3
>2014/10/29 23:19:44.021089 [ 7729]: Release freeze handler for prio 3
>2014/10/29 23:19:44.025304 [ 7729]: pnn 2 Invalid reqid 57283 in ctdb_become_dmaster from node 3
>2014/10/29 23:19:44.025987 [ 7729]: server/ctdb_call.c:1005 reqid 57284 not found
>2014/10/29 23:19:44.566011 [recoverd: 8044]: Resetting ban count to 0 for all nodes
>2014/10/29 23:26:22.113291 [recoverd: 8044]: Taking out recovery lock from recovery daemon
>2014/10/29 23:26:22.130858 [recoverd: 8044]: Take the recovery lock
>2014/10/29 23:26:22.243073 [ 7729]: Freeze priority 1
>2014/10/29 23:26:22.261883 [ 7729]: Freeze priority 2
>2014/10/29 23:26:22.266598 [ 7729]: Freeze priority 3
>2014/10/29 23:26:25.799245 [ 7729]: Thawing priority 1
>2014/10/29 23:26:25.799287 [ 7729]: Release freeze handler for prio 1
>2014/10/29 23:26:25.799322 [ 7729]: Thawing priority 2
>2014/10/29 23:26:25.799343 [ 7729]: Release freeze handler for prio 2
>2014/10/29 23:26:25.799372 [ 7729]: Thawing priority 3
>2014/10/29 23:26:25.799390 [ 7729]: Release freeze handler for prio 3
>2014/10/29 23:26:26.153355 [recoverd: 8044]: Resetting ban count to 0 for all nodes
>2014/10/29 23:29:48.437778 [recovery-lock: 2433]: failed read from recovery_lock_fd - Transport endpoint is not connected
>2014/10/29 23:29:48.801422 [recoverd: 8044]: server/ctdb_recoverd.c:3349 reclock child process returned error 2
>2014/10/29 23:29:48.801458 [recoverd: 8044]: server/ctdb_recoverd.c:3449 reclock child failed when checking file
>2014/10/29 23:29:48.801548 [recoverd: 8044]: Failed check_recovery_lock. Force a recovery
>2014/10/29 23:29:48.801565 [recoverd: 8044]: Taking out recovery lock from recovery daemon
>2014/10/29 23:29:48.801575 [recoverd: 8044]: Take the recovery lock
>2014/10/29 23:29:48.903663 [ 7729]: Freeze priority 1
>2014/10/29 23:29:48.958435 [ 7729]: Freeze priority 2
>2014/10/29 23:29:48.963127 [ 7729]: Freeze priority 3
>2014/10/29 23:29:49.379307 [ 7729]: pnn 2 Invalid reqid 86537 in ctdb_reply_control
>2014/10/29 23:29:49.393106 [ 7729]: pnn 2 Invalid reqid 86539 in ctdb_reply_control
>2014/10/29 23:29:49.423906 [ 7729]: pnn 2 Invalid reqid 86536 in ctdb_reply_control
>2014/10/29 23:29:49.550269 [ 7729]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 23:29:49.560911 [ 7729]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 23:31:20.462344 [ 1523]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/29 23:31:20.462448 [ 1523]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 23:31:20.462466 [ 1523]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/29 23:35:20.437306 [ 7517]: Starting CTDBD (Version 2.5.3) as PID: 7517
>2014/10/29 23:35:21.852604 [ 7517]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/29 23:35:21.876129 [ 7517]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/29 23:35:21.890263 [ 7517]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/29 23:35:21.904314 [ 7517]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/29 23:35:21.904332 [ 7517]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/29 23:35:21.904341 [ 7517]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/29 23:35:21.904350 [ 7517]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/29 23:35:21.904359 [ 7517]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/29 23:35:21.904367 [ 7517]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/29 23:35:21.918360 [ 7517]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/29 23:35:21.932297 [ 7517]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/29 23:35:21.932316 [ 7517]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/29 23:35:21.932325 [ 7517]: Ignoring persistent database 'registry.tdb.1'
>2014/10/29 23:35:21.946216 [ 7517]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/29 23:35:21.946246 [ 7517]: Freeze priority 1
>2014/10/29 23:35:21.964554 [ 7517]: Freeze priority 2
>2014/10/29 23:35:21.964913 [ 7517]: Freeze priority 3
>2014/10/29 23:35:22.129126 [ 7517]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/29 23:35:22.133292 [ 7517]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/29 23:35:22.136881 [ 7517]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/29 23:35:22.259327 [ 7517]: Freeze priority 1
>2014/10/29 23:35:22.259402 [ 7517]: Freeze priority 2
>2014/10/29 23:35:22.259456 [ 7517]: Freeze priority 3
>2014/10/29 23:35:22.599896 [ 7517]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/29 23:35:26.267285 [recoverd: 7799]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/29 23:35:26.267390 [ 7517]: Freeze priority 1
>2014/10/29 23:35:26.267468 [ 7517]: Freeze priority 2
>2014/10/29 23:35:26.267542 [ 7517]: Freeze priority 3
>2014/10/29 23:35:30.631487 [ 7517]: Freeze priority 1
>2014/10/29 23:35:30.645045 [ 7517]: Freeze priority 2
>2014/10/29 23:35:30.651808 [ 7517]: Freeze priority 3
>2014/10/29 23:35:30.837055 [ 7517]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/29 23:35:30.838974 [ 7517]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/29 23:35:30.845611 [ 7517]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/29 23:35:34.653197 [ 7517]: Thawing priority 1
>2014/10/29 23:35:34.653241 [ 7517]: Release freeze handler for prio 1
>2014/10/29 23:35:34.653275 [ 7517]: Thawing priority 2
>2014/10/29 23:35:34.653296 [ 7517]: Release freeze handler for prio 2
>2014/10/29 23:35:34.653323 [ 7517]: Thawing priority 3
>2014/10/29 23:35:34.653343 [ 7517]: Release freeze handler for prio 3
>2014/10/29 23:35:49.665525 [ 7517]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/29 23:35:49.672285 [recoverd: 7799]: Trigger takeoverrun
>2014/10/29 23:35:50.030247 [ 7517]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/29 23:35:50.043913 [ 7517]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/29 23:35:50.081005 [ 7517]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/29 23:35:50.393738 [ 7517]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/29 23:35:52.857040 [ 7517]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/29 23:35:53.689664 [ 7517]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/29 23:39:52.002593 [ 7517]: Freeze priority 1
>2014/10/29 23:39:52.002843 [ 7517]: Freeze priority 1
>2014/10/29 23:39:52.016368 [ 7517]: Freeze priority 2
>2014/10/29 23:39:52.016508 [ 7517]: Freeze priority 2
>2014/10/29 23:39:52.017521 [ 7517]: Freeze priority 3
>2014/10/29 23:39:52.017659 [ 7517]: Freeze priority 3
>2014/10/29 23:39:55.036875 [ 7517]: Freeze priority 1
>2014/10/29 23:39:55.037190 [ 7517]: Freeze priority 2
>2014/10/29 23:39:55.037453 [ 7517]: Freeze priority 3
>2014/10/29 23:39:57.560875 [ 7517]: Thawing priority 1
>2014/10/29 23:39:57.560925 [ 7517]: Release freeze handler for prio 1
>2014/10/29 23:39:57.560960 [ 7517]: Thawing priority 2
>2014/10/29 23:39:57.560982 [ 7517]: Release freeze handler for prio 2
>2014/10/29 23:39:57.561012 [ 7517]: Thawing priority 3
>2014/10/29 23:39:57.561031 [ 7517]: Release freeze handler for prio 3
>2014/10/29 23:44:55.998836 [recoverd: 7799]: Taking out recovery lock from recovery daemon
>2014/10/29 23:44:55.998906 [recoverd: 7799]: Take the recovery lock
>2014/10/29 23:44:56.081334 [ 7517]: Freeze priority 1
>2014/10/29 23:44:56.084543 [ 7517]: Freeze priority 2
>2014/10/29 23:44:56.089685 [ 7517]: Freeze priority 3
>2014/10/29 23:44:58.921741 [ 7517]: Thawing priority 1
>2014/10/29 23:44:58.921805 [ 7517]: Release freeze handler for prio 1
>2014/10/29 23:44:58.921857 [ 7517]: Thawing priority 2
>2014/10/29 23:44:58.921881 [ 7517]: Release freeze handler for prio 2
>2014/10/29 23:44:58.921913 [ 7517]: Thawing priority 3
>2014/10/29 23:44:58.921932 [ 7517]: Release freeze handler for prio 3
>2014/10/29 23:44:58.928547 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:44:58.928628 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:44:58.928647 [recoverd: 7799]: Async operation failed with ret=-1 res=-1 opcode=16
>2014/10/29 23:44:58.929613 [set_recmode: 4994]: ERROR: recovery lock file /mnt/lock/lockfile not locked when recovering!
>2014/10/29 23:44:58.929825 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:44:58.929859 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:44:58.929875 [recoverd: 7799]: Async operation failed with ret=-1 res=-1 opcode=16
>2014/10/29 23:44:58.944487 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:44:58.944522 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon'
>2014/10/29 23:44:58.944540 [recoverd: 7799]: Async operation failed with ret=-1 res=-1 opcode=16
>2014/10/29 23:44:58.944552 [recoverd: 7799]: Async wait failed - fail_count=3
>2014/10/29 23:44:58.944565 [recoverd: 7799]: server/ctdb_recoverd.c:412 Unable to set recovery mode. Recovery failed.
>2014/10/29 23:44:58.944577 [recoverd: 7799]: server/ctdb_recoverd.c:1996 Unable to set recovery mode to normal on cluster
>2014/10/29 23:45:02.023026 [ 7517]: Freeze priority 1
>2014/10/29 23:45:02.024833 [ 7517]: Freeze priority 2
>2014/10/29 23:45:02.027860 [ 7517]: Freeze priority 3
>2014/10/29 23:45:06.757661 [ 7517]: Thawing priority 1
>2014/10/29 23:45:06.757709 [ 7517]: Release freeze handler for prio 1
>2014/10/29 23:45:06.757743 [ 7517]: Thawing priority 2
>2014/10/29 23:45:06.757764 [ 7517]: Release freeze handler for prio 2
>2014/10/29 23:45:06.757793 [ 7517]: Thawing priority 3
>2014/10/29 23:45:06.757812 [ 7517]: Release freeze handler for prio 3
>2014/10/29 23:54:58.562006 [ 7517]: Freeze priority 1
>2014/10/29 23:54:58.566363 [ 7517]: Freeze priority 2
>2014/10/29 23:54:58.570732 [ 7517]: Freeze priority 3
>2014/10/29 23:55:01.591803 [ 7517]: Freeze priority 1
>2014/10/29 23:55:01.592138 [ 7517]: Freeze priority 2
>2014/10/29 23:55:01.592520 [ 7517]: Freeze priority 3
>2014/10/29 23:55:05.236799 [ 7517]: Thawing priority 1
>2014/10/29 23:55:05.236891 [ 7517]: Release freeze handler for prio 1
>2014/10/29 23:55:05.236939 [ 7517]: Thawing priority 2
>2014/10/29 23:55:05.236959 [ 7517]: Release freeze handler for prio 2
>2014/10/29 23:55:05.236990 [ 7517]: Thawing priority 3
>2014/10/29 23:55:05.237008 [ 7517]: Release freeze handler for prio 3
>2014/10/30 00:00:19.275139 [ 7517]: Freeze priority 1
>2014/10/30 00:00:19.381570 [ 7517]: Freeze priority 2
>2014/10/30 00:00:19.382780 [ 7517]: Freeze priority 3
>2014/10/30 00:00:24.319174 [ 7517]: Thawing priority 1
>2014/10/30 00:00:24.319222 [ 7517]: Release freeze handler for prio 1
>2014/10/30 00:00:24.319256 [ 7517]: Thawing priority 2
>2014/10/30 00:00:24.319277 [ 7517]: Release freeze handler for prio 2
>2014/10/30 00:00:24.319309 [ 7517]: Thawing priority 3
>2014/10/30 00:00:24.319328 [ 7517]: Release freeze handler for prio 3
>2014/10/30 00:05:05.687461 [ 7517]: Freeze priority 1
>2014/10/30 00:05:05.693033 [ 7517]: Freeze priority 1
>2014/10/30 00:05:05.695439 [ 7517]: Freeze priority 1
>2014/10/30 00:05:05.702088 [ 7517]: Freeze priority 2
>2014/10/30 00:05:05.704678 [ 7517]: Freeze priority 2
>2014/10/30 00:05:05.706283 [ 7517]: Freeze priority 2
>2014/10/30 00:05:05.706377 [ 7517]: Freeze priority 3
>2014/10/30 00:05:05.706836 [ 7517]: Freeze priority 3
>2014/10/30 00:05:05.710962 [ 7517]: Freeze priority 3
>2014/10/30 00:05:09.222426 [recoverd: 7799]: Taking out recovery lock from recovery daemon
>2014/10/30 00:05:09.222482 [recoverd: 7799]: Take the recovery lock
>2014/10/30 00:05:09.236333 [ 7517]: Freeze priority 1
>2014/10/30 00:05:09.236689 [ 7517]: Freeze priority 2
>2014/10/30 00:05:09.237025 [ 7517]: Freeze priority 3
>2014/10/30 00:05:11.688669 [ 7517]: Thawing priority 1
>2014/10/30 00:05:11.688709 [ 7517]: Release freeze handler for prio 1
>2014/10/30 00:05:11.688739 [ 7517]: Thawing priority 2
>2014/10/30 00:05:11.688766 [ 7517]: Release freeze handler for prio 2
>2014/10/30 00:05:11.688791 [ 7517]: Thawing priority 3
>2014/10/30 00:05:11.688806 [ 7517]: Release freeze handler for prio 3
>2014/10/30 00:05:12.278504 [recoverd: 7799]: Resetting ban count to 0 for all nodes
>2014/10/30 00:11:46.429803 [ 7517]: Freeze priority 1
>2014/10/30 00:11:46.493960 [ 7517]: Freeze priority 2
>2014/10/30 00:11:46.499003 [ 7517]: Freeze priority 3
>2014/10/30 00:11:51.797756 [ 7517]: Thawing priority 1
>2014/10/30 00:11:51.797867 [ 7517]: Release freeze handler for prio 1
>2014/10/30 00:11:51.797926 [ 7517]: Thawing priority 2
>2014/10/30 00:11:51.797947 [ 7517]: Release freeze handler for prio 2
>2014/10/30 00:11:51.797980 [ 7517]: Thawing priority 3
>2014/10/30 00:11:51.797995 [ 7517]: Release freeze handler for prio 3
>2014/10/30 00:15:14.434917 [ 7517]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 00:15:14.555376 [ 7517]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/30 00:15:14.555410 [ 7517]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 00:16:40.983759 [ 1538]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 00:16:40.983858 [ 1538]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 00:16:40.983872 [ 1538]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 00:20:40.944984 [ 7818]: Starting CTDBD (Version 2.5.3) as PID: 7818
>2014/10/30 00:20:42.884811 [ 7818]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 00:20:42.908316 [ 7818]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 00:20:42.922385 [ 7818]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 00:20:42.936300 [ 7818]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 00:20:42.936318 [ 7818]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 00:20:42.936327 [ 7818]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 00:20:42.936336 [ 7818]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 00:20:42.936345 [ 7818]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 00:20:42.936353 [ 7818]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 00:20:42.950277 [ 7818]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 00:20:42.964190 [ 7818]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 00:20:42.964209 [ 7818]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 00:20:42.964218 [ 7818]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 00:20:42.978215 [ 7818]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 00:20:42.978247 [ 7818]: Freeze priority 1
>2014/10/30 00:20:42.985056 [ 7818]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 00:20:42.988498 [ 7818]: Freeze priority 2
>2014/10/30 00:20:42.988845 [ 7818]: Freeze priority 3
>2014/10/30 00:20:43.152836 [ 7818]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 00:20:43.156710 [ 7818]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 00:20:43.160453 [ 7818]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 00:20:43.282466 [ 7818]: Freeze priority 1
>2014/10/30 00:20:43.282540 [ 7818]: Freeze priority 2
>2014/10/30 00:20:43.282593 [ 7818]: Freeze priority 3
>2014/10/30 00:20:43.302134 [ 7818]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 00:20:43.484755 [ 7818]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 00:20:47.290378 [recoverd: 8097]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 00:20:47.290494 [ 7818]: Freeze priority 1
>2014/10/30 00:20:47.290571 [ 7818]: Freeze priority 2
>2014/10/30 00:20:47.290624 [ 7818]: Freeze priority 3
>2014/10/30 00:20:51.691775 [ 7818]: Freeze priority 1
>2014/10/30 00:20:51.693474 [ 7818]: Freeze priority 2
>2014/10/30 00:20:51.695242 [ 7818]: Freeze priority 3
>2014/10/30 00:20:51.867529 [ 7818]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 00:20:51.869836 [ 7818]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 00:20:51.872102 [ 7818]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 00:20:54.036238 [ 7818]: Thawing priority 1
>2014/10/30 00:20:54.036287 [ 7818]: Release freeze handler for prio 1
>2014/10/30 00:20:54.036321 [ 7818]: Thawing priority 2
>2014/10/30 00:20:54.036341 [ 7818]: Release freeze handler for prio 2
>2014/10/30 00:20:54.036367 [ 7818]: Thawing priority 3
>2014/10/30 00:20:54.036384 [ 7818]: Release freeze handler for prio 3
>2014/10/30 00:21:08.329645 [recoverd: 8097]: Trigger takeoverrun
>2014/10/30 00:21:08.406254 [ 7818]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 00:21:08.720224 [ 7818]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 00:21:08.731642 [ 7818]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 00:21:08.756017 [ 7818]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 00:21:08.934080 [ 7818]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 00:21:11.408393 [ 7818]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 00:21:12.003982 [ 7818]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 00:25:17.387048 [ 7818]: Freeze priority 1
>2014/10/30 00:25:17.398957 [ 7818]: Freeze priority 2
>2014/10/30 00:25:17.400094 [ 7818]: Freeze priority 3
>2014/10/30 00:25:20.486223 [ 7818]: Freeze priority 1
>2014/10/30 00:25:20.486637 [ 7818]: Freeze priority 2
>2014/10/30 00:25:20.486984 [ 7818]: Freeze priority 3
>2014/10/30 00:25:23.306957 [ 7818]: Thawing priority 1
>2014/10/30 00:25:23.307015 [ 7818]: Release freeze handler for prio 1
>2014/10/30 00:25:23.307053 [ 7818]: Thawing priority 2
>2014/10/30 00:25:23.307074 [ 7818]: Release freeze handler for prio 2
>2014/10/30 00:25:23.307098 [ 7818]: Thawing priority 3
>2014/10/30 00:25:23.307109 [ 7818]: Release freeze handler for prio 3
>2014/10/30 00:25:23.309681 [ 7818]: pnn 2 Invalid reqid 8045 in ctdb_become_dmaster from node 0
>2014/10/30 00:25:23.309742 [ 7818]: server/ctdb_call.c:1005 reqid 8046 not found
>2014/10/30 00:30:23.316826 [ 7818]: Freeze priority 1
>2014/10/30 00:30:23.410573 [ 7818]: Freeze priority 2
>2014/10/30 00:30:23.411530 [ 7818]: Freeze priority 3
>2014/10/30 00:30:26.712131 [ 7818]: Thawing priority 1
>2014/10/30 00:30:26.712181 [ 7818]: Release freeze handler for prio 1
>2014/10/30 00:30:26.712213 [ 7818]: Thawing priority 2
>2014/10/30 00:30:26.712229 [ 7818]: Release freeze handler for prio 2
>2014/10/30 00:30:26.712255 [ 7818]: Thawing priority 3
>2014/10/30 00:30:26.712270 [ 7818]: Release freeze handler for prio 3
>2014/10/30 00:40:26.491813 [ 7818]: Freeze priority 1
>2014/10/30 00:40:26.530234 [ 7818]: Freeze priority 1
>2014/10/30 00:40:26.530297 [ 7818]: Freeze priority 1
>2014/10/30 00:40:33.912559 [ 7818]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=5236 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=5236 =====
>===== Start of debug locks PID=5748 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=5748 =====
>2014/10/30 00:40:48.912906 [ 7818]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=6443 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=6443 =====
>2014/10/30 00:41:03.913859 [ 7818]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=6839 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=6839 =====
>===== Start of debug locks PID=7275 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=7275 =====
>2014/10/30 00:41:18.914467 [ 7818]: Skip monitoring since databases are frozen
>2014/10/30 00:41:26.495095 [ 7818]: Freeze priority 1
>2014/10/30 00:41:26.530205 [ 7818]: Recovery daemon ping timeout. Count : 0
>2014/10/30 00:41:26.530332 [recoverd: 8097]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 00:41:26.530379 [recoverd: 8097]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 00:41:26.530403 [recoverd: 8097]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/30 00:41:26.530421 [recoverd: 8097]: Failed to freeze node 2 during recovery. Set it as ban culprit for 4 credits
>2014/10/30 00:41:26.530440 [recoverd: 8097]: Async wait failed - fail_count=1
>2014/10/30 00:41:26.530455 [recoverd: 8097]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/30 00:41:26.530472 [recoverd: 8097]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/30 00:41:26.532117 [ 7818]: Freeze priority 1
>2014/10/30 00:41:26.532157 [ 7818]: Freeze priority 1
>===== Start of debug locks PID=7670 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=7670 =====
>2014/10/30 00:41:33.914908 [ 7818]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=8027 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=8027 =====
>===== Start of debug locks PID=8466 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=8466 =====
>2014/10/30 00:41:48.915071 [ 7818]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=8990 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=8990 =====
>2014/10/30 00:42:03.915852 [ 7818]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=9397 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=9397 =====
>===== Start of debug locks PID=9836 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>10172 /usr/sbin/smbd printer_list.tdb.2 2984 2984 W
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>----- Stack trace for PID=23687 -----
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0
>#28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#31 0x00007f95c57ad1b4 in smbd_accept_connection ()
>#32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0
>#33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#35 0x00007f95c57a9d01 in main ()
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a430094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537
>#5  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541
>#7  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541
>#8  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541
>#10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537
>#13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541
>#14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541
>#18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537
>#19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537
>#20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129
>===== End of debug locks PID=9836 =====
>2014/10/30 00:42:18.916751 [ 7818]: Skip monitoring since databases are frozen
>2014/10/30 00:42:26.497770 [ 7818]: Banning this node for 30 seconds
>2014/10/30 00:42:26.497824 [ 7818]: Freeze priority 1
>2014/10/30 00:42:26.497838 [ 7818]: Freeze priority 2
>2014/10/30 00:42:26.498006 [ 7818]: Freeze priority 3
>2014/10/30 00:42:26.531568 [ 7818]: Recovery daemon ping timeout. Count : 0
>2014/10/30 00:42:26.532695 [recoverd: 8097]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 00:42:26.532731 [recoverd: 8097]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 00:42:26.532746 [recoverd: 8097]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/30 00:42:26.532757 [recoverd: 8097]: Failed to freeze node 2 during recovery. Set it as ban culprit for 4 credits
>2014/10/30 00:42:26.532769 [recoverd: 8097]: Async wait failed - fail_count=1
>2014/10/30 00:42:26.532779 [recoverd: 8097]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/30 00:42:26.532789 [recoverd: 8097]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/30 00:42:26.533422 [ 7818]: Banning this node for 30 seconds
>2014/10/30 00:42:26.533457 [ 7818]: Freeze priority 1
>2014/10/30 00:42:26.533760 [ 7818]: Banning this node for 30 seconds
>2014/10/30 00:42:26.533791 [ 7818]: Freeze priority 1
>===== Start of debug locks PID=10316 =====
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454
>4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W
>4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>10172 /usr/sbin/smbd printer_list.tdb.2 2984 2984 W
>10277 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>10277 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>23687 /usr/sbin/smbd locking.tdb.2 27456 27456
>4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W
>10276 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>10276 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>10276 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>----- Stack trace for PID=23687 -----
>2014/10/30 00:42:26.625068 [ 7818]: 10.interface: Killing TCP connection 10.10.10.210:53439 10.10.10.183:445
>2014/10/30 00:42:26.625244 [ 7818]: 10.interface: Killing TCP connection 10.10.10.205:54702 10.10.10.183:445
>2014/10/30 00:42:26.625343 [ 7818]: 10.interface: Killing TCP connection 10.10.10.208:54232 10.10.10.183:445
>2014/10/30 00:42:26.625439 [ 7818]: 10.interface: Killing TCP connection 10.10.10.206:49680 10.10.10.183:445
>2014/10/30 00:42:26.626103 [ 7818]: 10.interface: Killing TCP connection 10.10.10.210:53439 10.10.10.183:445
>2014/10/30 00:42:26.626217 [ 7818]: 10.interface: Killing TCP connection 10.10.10.205:54702 10.10.10.183:445
>2014/10/30 00:42:26.626325 [ 7818]: 10.interface: Killing TCP connection 10.10.10.208:54232 10.10.10.183:445
>2014/10/30 00:42:26.626409 [ 7818]: 10.interface: Killing TCP connection 10.10.10.206:49680 10.10.10.183:445
>2014/10/30 00:42:26.626836 [ 7818]: 10.interface: Killing TCP connection 10.10.10.210:53439 10.10.10.183:445
>2014/10/30 00:42:26.626966 [ 7818]: 10.interface: Killing TCP connection 10.10.10.205:54702 10.10.10.183:445
>2014/10/30 00:42:26.627070 [ 7818]: 10.interface: Killing TCP connection 10.10.10.208:54232 10.10.10.183:445
>2014/10/30 00:42:26.627158 [ 7818]: 10.interface: Killing TCP connection 10.10.10.206:49680 10.10.10.183:445
>2014/10/30 00:42:26.655282 [ 7818]: 10.interface: Killed 4 TCP connections to released IP 10.10.10.183
>2014/10/30 00:42:26.655536 [ 7818]: 10.interface: Killed 4 TCP connections to released IP 10.10.10.183
>2014/10/30 00:42:26.655699 [ 7818]: 10.interface: Killed 4 TCP connections to released IP 10.10.10.183
>2014/10/30 00:42:26.669693 [ 7818]: 10.interface: RTNETLINK answers: Cannot assign requested address
>2014/10/30 00:42:26.669840 [ 7818]: 10.interface: Failed to del 10.10.10.183 on dev bond1
>2014/10/30 00:42:26.672502 [ 7818]: 10.interface: RTNETLINK answers: Cannot assign requested address
>2014/10/30 00:42:26.672650 [ 7818]: 10.interface: Failed to del 10.10.10.183 on dev bond1
>2014/10/30 00:42:26.789022 [ 7818]: Freeze priority 1
>2014/10/30 00:42:26.789126 [ 7818]: Freeze priority 1
>2014/10/30 00:42:26.837155 [ 7818]: pnn 2 Invalid reqid 40862 in ctdb_reply_control
>2014/10/30 00:42:26.837188 [ 7818]: pnn 2 Invalid reqid 40656 in ctdb_reply_control
>2014/10/30 00:42:26.837545 [ 7818]: Freeze priority 2
>2014/10/30 00:42:26.837596 [ 7818]: Freeze priority 2
>2014/10/30 00:42:26.838430 [ 7818]: Freeze priority 3
>2014/10/30 00:42:26.838522 [ 7818]: Freeze priority 3
>#0  0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6
>----- Stack trace for PID=4899 -----
>#0  0x00007f319a405890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f319a405744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=10316 =====
>2014/10/30 00:42:33.547346 [ 7818]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 00:42:36.389861 [ 7818]: DB Attach to database ctdb.tdb refused since node is inactive (flags=0x8)
>2014/10/30 00:42:56.534749 [ 7818]: Banning timedout
>2014/10/30 00:42:56.886614 [ 7818]: Freeze priority 1
>2014/10/30 00:42:56.891571 [ 7818]: Freeze priority 2
>2014/10/30 00:42:56.896050 [ 7818]: Freeze priority 3
>2014/10/30 00:42:59.646260 [ 7818]: Thawing priority 1
>2014/10/30 00:42:59.646336 [ 7818]: Release freeze handler for prio 1
>2014/10/30 00:42:59.646377 [ 7818]: Thawing priority 2
>2014/10/30 00:42:59.646392 [ 7818]: Release freeze handler for prio 2
>2014/10/30 00:42:59.646418 [ 7818]: Thawing priority 3
>2014/10/30 00:42:59.646438 [ 7818]: Release freeze handler for prio 3
>2014/10/30 00:42:59.647993 [ 7818]: server/ctdb_call.c:1005 reqid 41465 not found
>2014/10/30 00:42:59.648069 [ 7818]: server/ctdb_call.c:1005 reqid 41466 not found
>2014/10/30 00:43:00.261087 [ 7818]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 00:45:41.909761 [ 7818]: Freeze priority 1
>2014/10/30 00:45:42.054977 [ 7818]: Freeze priority 2
>2014/10/30 00:45:42.082373 [ 7818]: Freeze priority 3
>2014/10/30 00:45:45.405501 [ 7818]: Thawing priority 1
>2014/10/30 00:45:45.405545 [ 7818]: Release freeze handler for prio 1
>2014/10/30 00:45:45.405577 [ 7818]: Thawing priority 2
>2014/10/30 00:45:45.405593 [ 7818]: Release freeze handler for prio 2
>2014/10/30 00:45:45.405619 [ 7818]: Thawing priority 3
>2014/10/30 00:45:45.405633 [ 7818]: Release freeze handler for prio 3
>2014/10/30 00:50:30.261912 [ 7818]: Freeze priority 1
>2014/10/30 00:50:30.403324 [ 7818]: Freeze priority 2
>2014/10/30 00:50:30.415014 [ 7818]: Freeze priority 3
>2014/10/30 00:50:33.225966 [ 7818]: Thawing priority 1
>2014/10/30 00:50:33.226003 [ 7818]: Release freeze handler for prio 1
>2014/10/30 00:50:33.226031 [ 7818]: Thawing priority 2
>2014/10/30 00:50:33.226049 [ 7818]: Release freeze handler for prio 2
>2014/10/30 00:50:33.226078 [ 7818]: Thawing priority 3
>2014/10/30 00:50:33.226094 [ 7818]: Release freeze handler for prio 3
>2014/10/30 00:57:16.326220 [ 7818]: Freeze priority 1
>2014/10/30 00:57:16.468323 [ 7818]: Freeze priority 2
>2014/10/30 00:57:16.484816 [ 7818]: Freeze priority 3
>2014/10/30 00:57:19.140647 [ 7818]: Thawing priority 1
>2014/10/30 00:57:19.140700 [ 7818]: Release freeze handler for prio 1
>2014/10/30 00:57:19.140732 [ 7818]: Thawing priority 2
>2014/10/30 00:57:19.140749 [ 7818]: Release freeze handler for prio 2
>2014/10/30 00:57:19.140776 [ 7818]: Thawing priority 3
>2014/10/30 00:57:19.140792 [ 7818]: Release freeze handler for prio 3
>2014/10/30 01:00:40.696663 [ 7818]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 01:00:40.708314 [ 7818]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 01:02:06.927678 [ 1542]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 01:02:06.927777 [ 1542]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 01:02:06.927791 [ 1542]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 01:06:06.551288 [ 7671]: Starting CTDBD (Version 2.5.3) as PID: 7671
>2014/10/30 01:06:08.031432 [ 7671]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 01:06:08.055104 [ 7671]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 01:06:08.069248 [ 7671]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 01:06:08.083141 [ 7671]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 01:06:08.083159 [ 7671]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 01:06:08.083168 [ 7671]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 01:06:08.083177 [ 7671]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 01:06:08.083186 [ 7671]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 01:06:08.083194 [ 7671]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 01:06:08.097072 [ 7671]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 01:06:08.110955 [ 7671]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 01:06:08.110973 [ 7671]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 01:06:08.110983 [ 7671]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 01:06:08.124888 [ 7671]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 01:06:08.124923 [ 7671]: Freeze priority 1
>2014/10/30 01:06:08.143125 [ 7671]: Freeze priority 2
>2014/10/30 01:06:08.143540 [ 7671]: Freeze priority 3
>2014/10/30 01:06:08.307477 [ 7671]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 01:06:08.311274 [ 7671]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 01:06:08.314694 [ 7671]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 01:06:08.433478 [ 7671]: Freeze priority 1
>2014/10/30 01:06:08.433551 [ 7671]: Freeze priority 2
>2014/10/30 01:06:08.433604 [ 7671]: Freeze priority 3
>2014/10/30 01:06:08.878277 [ 7671]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 01:06:13.615452 [ 7671]: Freeze priority 1
>2014/10/30 01:06:13.632204 [ 7671]: Freeze priority 2
>2014/10/30 01:06:13.636861 [ 7671]: Freeze priority 3
>2014/10/30 01:06:13.835966 [ 7671]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 01:06:13.836480 [ 7671]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 01:06:13.837485 [ 7671]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 01:06:16.379645 [ 7671]: Thawing priority 1
>2014/10/30 01:06:16.379682 [ 7671]: Release freeze handler for prio 1
>2014/10/30 01:06:16.379712 [ 7671]: Thawing priority 2
>2014/10/30 01:06:16.379729 [ 7671]: Release freeze handler for prio 2
>2014/10/30 01:06:16.379762 [ 7671]: Thawing priority 3
>2014/10/30 01:06:16.379777 [ 7671]: Release freeze handler for prio 3
>2014/10/30 01:06:30.403218 [recoverd: 7950]: Trigger takeoverrun
>2014/10/30 01:06:30.836087 [ 7671]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 01:06:31.135355 [ 7671]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 01:06:31.146472 [ 7671]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 01:06:31.168316 [ 7671]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 01:06:31.347091 [ 7671]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 01:06:33.765203 [ 7671]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 01:06:34.265995 [ 7671]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 01:10:42.744205 [ 7671]: Freeze priority 1
>2014/10/30 01:10:42.754744 [ 7671]: Freeze priority 1
>2014/10/30 01:10:42.765990 [ 7671]: Freeze priority 2
>2014/10/30 01:10:42.767464 [ 7671]: Freeze priority 2
>2014/10/30 01:10:42.768599 [ 7671]: Freeze priority 3
>2014/10/30 01:10:42.768956 [ 7671]: Freeze priority 3
>2014/10/30 01:10:45.789669 [ 7671]: Freeze priority 1
>2014/10/30 01:10:45.790081 [ 7671]: Freeze priority 2
>2014/10/30 01:10:45.790776 [ 7671]: Freeze priority 3
>2014/10/30 01:10:48.754812 [ 7671]: Thawing priority 1
>2014/10/30 01:10:48.754854 [ 7671]: Release freeze handler for prio 1
>2014/10/30 01:10:48.754910 [ 7671]: Thawing priority 2
>2014/10/30 01:10:48.754954 [ 7671]: Release freeze handler for prio 2
>2014/10/30 01:10:48.754984 [ 7671]: Thawing priority 3
>2014/10/30 01:10:48.755016 [ 7671]: Release freeze handler for prio 3
>2014/10/30 01:15:47.008563 [ 7671]: Freeze priority 1
>2014/10/30 01:15:47.072168 [ 7671]: Freeze priority 2
>2014/10/30 01:15:47.075867 [ 7671]: Freeze priority 3
>2014/10/30 01:15:50.440067 [ 7671]: Thawing priority 1
>2014/10/30 01:15:50.440153 [ 7671]: Release freeze handler for prio 1
>2014/10/30 01:15:50.440198 [ 7671]: Thawing priority 2
>2014/10/30 01:15:50.440218 [ 7671]: Release freeze handler for prio 2
>2014/10/30 01:15:50.440251 [ 7671]: Thawing priority 3
>2014/10/30 01:15:50.440269 [ 7671]: Release freeze handler for prio 3
>2014/10/30 01:16:00.943534 [ 7671]: Monitoring event was cancelled
>2014/10/30 01:16:01.114035 [ 7671]: Freeze priority 1
>2014/10/30 01:16:01.116321 [ 7671]: Freeze priority 2
>2014/10/30 01:16:01.117798 [ 7671]: Freeze priority 3
>2014/10/30 01:16:04.153080 [ 7671]: Thawing priority 1
>2014/10/30 01:16:04.153123 [ 7671]: Release freeze handler for prio 1
>2014/10/30 01:16:04.153158 [ 7671]: Thawing priority 2
>2014/10/30 01:16:04.153178 [ 7671]: Release freeze handler for prio 2
>2014/10/30 01:16:04.153210 [ 7671]: Thawing priority 3
>2014/10/30 01:16:04.153229 [ 7671]: Release freeze handler for prio 3
>2014/10/30 01:25:53.930571 [ 7671]: Freeze priority 1
>2014/10/30 01:25:53.946193 [ 7671]: Freeze priority 1
>2014/10/30 01:25:53.979062 [ 7671]: Freeze priority 1
>2014/10/30 01:25:53.984722 [ 7671]: Freeze priority 2
>2014/10/30 01:25:53.985562 [ 7671]: Freeze priority 2
>2014/10/30 01:25:53.986101 [ 7671]: Freeze priority 2
>2014/10/30 01:25:53.986540 [ 7671]: Freeze priority 3
>2014/10/30 01:25:53.987071 [ 7671]: Freeze priority 3
>2014/10/30 01:25:53.987316 [ 7671]: Freeze priority 3
>2014/10/30 01:25:57.506025 [ 7671]: Freeze priority 1
>2014/10/30 01:25:57.506350 [ 7671]: Freeze priority 2
>2014/10/30 01:25:57.506666 [ 7671]: Freeze priority 3
>2014/10/30 01:26:01.992757 [ 7671]: Thawing priority 1
>2014/10/30 01:26:01.992809 [ 7671]: Release freeze handler for prio 1
>2014/10/30 01:26:01.992840 [ 7671]: Thawing priority 2
>2014/10/30 01:26:01.992859 [ 7671]: Release freeze handler for prio 2
>2014/10/30 01:26:01.992894 [ 7671]: Thawing priority 3
>2014/10/30 01:26:01.992916 [ 7671]: Release freeze handler for prio 3
>2014/10/30 01:31:15.018448 [ 7671]: Freeze priority 1
>2014/10/30 01:31:15.243361 [ 7671]: Freeze priority 2
>2014/10/30 01:31:15.244585 [ 7671]: Freeze priority 3
>2014/10/30 01:31:18.685801 [ 7671]: Thawing priority 1
>2014/10/30 01:31:18.685856 [ 7671]: Release freeze handler for prio 1
>2014/10/30 01:31:18.685899 [ 7671]: Thawing priority 2
>2014/10/30 01:31:18.685920 [ 7671]: Release freeze handler for prio 2
>2014/10/30 01:31:18.685953 [ 7671]: Thawing priority 3
>2014/10/30 01:31:18.685970 [ 7671]: Release freeze handler for prio 3
>2014/10/30 01:31:18.687402 [ 7671]: server/ctdb_call.c:1005 reqid 46805 not found
>2014/10/30 01:31:18.687445 [ 7671]: server/ctdb_call.c:1005 reqid 46806 not found
>2014/10/30 01:36:01.071753 [ 7671]: Freeze priority 1
>2014/10/30 01:36:01.087906 [ 7671]: Freeze priority 1
>2014/10/30 01:36:01.090507 [ 7671]: Freeze priority 1
>2014/10/30 01:36:01.149678 [ 7671]: Freeze priority 2
>2014/10/30 01:36:01.151637 [ 7671]: Freeze priority 2
>2014/10/30 01:36:01.152381 [ 7671]: Freeze priority 2
>2014/10/30 01:36:01.152909 [ 7671]: Freeze priority 3
>2014/10/30 01:36:01.153502 [ 7671]: Freeze priority 3
>2014/10/30 01:36:01.155226 [ 7671]: Freeze priority 3
>2014/10/30 01:36:04.167545 [recoverd: 7950]: Taking out recovery lock from recovery daemon
>2014/10/30 01:36:04.167579 [recoverd: 7950]: Take the recovery lock
>2014/10/30 01:36:04.196520 [ 7671]: Freeze priority 1
>2014/10/30 01:36:04.196867 [ 7671]: Freeze priority 2
>2014/10/30 01:36:04.197152 [ 7671]: Freeze priority 3
>2014/10/30 01:36:06.982332 [ 7671]: Thawing priority 1
>2014/10/30 01:36:06.982379 [ 7671]: Release freeze handler for prio 1
>2014/10/30 01:36:06.982431 [ 7671]: Thawing priority 2
>2014/10/30 01:36:06.982465 [ 7671]: Release freeze handler for prio 2
>2014/10/30 01:36:06.982513 [ 7671]: Thawing priority 3
>2014/10/30 01:36:06.982536 [ 7671]: Release freeze handler for prio 3
>2014/10/30 01:36:06.984715 [ 7671]: server/ctdb_call.c:1005 reqid 70174 not found
>2014/10/30 01:36:06.984772 [ 7671]: server/ctdb_call.c:1005 reqid 70175 not found
>2014/10/30 01:36:06.984801 [ 7671]: server/ctdb_call.c:1005 reqid 70176 not found
>2014/10/30 01:36:06.984823 [ 7671]: server/ctdb_call.c:1005 reqid 70177 not found
>2014/10/30 01:36:08.030817 [recoverd: 7950]: Resetting ban count to 0 for all nodes
>2014/10/30 01:42:43.600216 [recoverd: 7950]: Taking out recovery lock from recovery daemon
>2014/10/30 01:42:43.610535 [recoverd: 7950]: Take the recovery lock
>2014/10/30 01:42:44.654301 [ 7671]: High RECLOCK latency 1.043671s for operation recd reclock
>2014/10/30 01:42:44.697237 [ 7671]: Freeze priority 1
>2014/10/30 01:42:44.724353 [ 7671]: Freeze priority 2
>2014/10/30 01:42:44.725658 [ 7671]: Freeze priority 3
>2014/10/30 01:42:48.589487 [ 7671]: Thawing priority 1
>2014/10/30 01:42:48.589527 [ 7671]: Release freeze handler for prio 1
>2014/10/30 01:42:48.589564 [ 7671]: Thawing priority 2
>2014/10/30 01:42:48.589599 [ 7671]: Release freeze handler for prio 2
>2014/10/30 01:42:48.589632 [ 7671]: Thawing priority 3
>2014/10/30 01:42:48.589651 [ 7671]: Release freeze handler for prio 3
>2014/10/30 01:42:48.919489 [recoverd: 7950]: Resetting ban count to 0 for all nodes
>2014/10/30 01:46:10.513121 [ 7671]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 01:46:10.539069 [ 7671]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/30 01:46:10.539096 [ 7671]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 01:47:40.448907 [ 1531]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 01:47:40.449009 [ 1531]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 01:47:40.449024 [ 1531]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 01:51:40.820220 [ 7638]: Starting CTDBD (Version 2.5.3) as PID: 7638
>2014/10/30 01:51:42.285542 [ 7638]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 01:51:42.309438 [ 7638]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 01:51:42.323793 [ 7638]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 01:51:42.337895 [ 7638]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 01:51:42.337920 [ 7638]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 01:51:42.337935 [ 7638]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 01:51:42.337956 [ 7638]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 01:51:42.337966 [ 7638]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 01:51:42.337975 [ 7638]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 01:51:42.352162 [ 7638]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 01:51:42.366408 [ 7638]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 01:51:42.366427 [ 7638]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 01:51:42.366451 [ 7638]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 01:51:42.381460 [ 7638]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 01:51:42.381497 [ 7638]: Freeze priority 1
>2014/10/30 01:51:42.405497 [ 7638]: Freeze priority 2
>2014/10/30 01:51:42.405854 [ 7638]: Freeze priority 3
>2014/10/30 01:51:42.569517 [ 7638]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 01:51:42.570246 [ 7638]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 01:51:42.574766 [ 7638]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 01:51:42.578884 [ 7638]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 01:51:42.701508 [ 7638]: Freeze priority 1
>2014/10/30 01:51:42.701586 [ 7638]: Freeze priority 2
>2014/10/30 01:51:42.701639 [ 7638]: Freeze priority 3
>2014/10/30 01:51:43.094042 [ 7638]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 01:51:43.166105 [ 7638]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 01:51:46.505061 [ 7638]: Freeze priority 1
>2014/10/30 01:51:46.515471 [ 7638]: Freeze priority 2
>2014/10/30 01:51:46.520928 [ 7638]: Freeze priority 3
>2014/10/30 01:51:46.707693 [recoverd: 7923]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 01:51:46.707781 [ 7638]: Freeze priority 1
>2014/10/30 01:51:46.707867 [ 7638]: Freeze priority 2
>2014/10/30 01:51:46.707947 [ 7638]: Freeze priority 3
>2014/10/30 01:51:50.069689 [ 7638]: Thawing priority 1
>2014/10/30 01:51:50.069731 [ 7638]: Release freeze handler for prio 1
>2014/10/30 01:51:50.069763 [ 7638]: Thawing priority 2
>2014/10/30 01:51:50.069784 [ 7638]: Release freeze handler for prio 2
>2014/10/30 01:51:50.069810 [ 7638]: Thawing priority 3
>2014/10/30 01:51:50.069828 [ 7638]: Release freeze handler for prio 3
>2014/10/30 01:51:52.094002 [ 7638]: Freeze priority 1
>2014/10/30 01:51:52.095453 [ 7638]: Freeze priority 2
>2014/10/30 01:51:52.096809 [ 7638]: Freeze priority 3
>2014/10/30 01:51:52.263251 [ 7638]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 01:51:52.263850 [ 7638]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 01:51:52.265529 [ 7638]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 01:51:54.975981 [ 7638]: Thawing priority 1
>2014/10/30 01:51:54.976054 [ 7638]: Release freeze handler for prio 1
>2014/10/30 01:51:54.976117 [ 7638]: Thawing priority 2
>2014/10/30 01:51:54.976142 [ 7638]: Release freeze handler for prio 2
>2014/10/30 01:51:54.976186 [ 7638]: Thawing priority 3
>2014/10/30 01:51:54.976205 [ 7638]: Release freeze handler for prio 3
>2014/10/30 01:52:09.896451 [ 7638]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 01:52:10.002851 [recoverd: 7923]: Trigger takeoverrun
>2014/10/30 01:52:10.720329 [ 7638]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 01:52:10.732338 [ 7638]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 01:52:10.760227 [ 7638]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 01:52:10.959626 [ 7638]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 01:52:13.398476 [ 7638]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/30 01:56:15.831445 [ 7638]: Freeze priority 1
>2014/10/30 01:56:17.896360 [ 7638]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=24063 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=24063 =====
>2014/10/30 01:56:32.896794 [ 7638]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=24422 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=24422 =====
>===== Start of debug locks PID=24884 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=24884 =====
>2014/10/30 01:56:47.897908 [ 7638]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=25373 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=25373 =====
>2014/10/30 01:57:02.899012 [ 7638]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=25704 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=25704 =====
>2014/10/30 01:57:15.849073 [ 7638]: Freeze priority 1
>===== Start of debug locks PID=26108 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=26108 =====
>2014/10/30 01:57:17.899503 [ 7638]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=26586 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=26586 =====
>2014/10/30 01:57:32.900138 [ 7638]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=26928 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=26928 =====
>===== Start of debug locks PID=27349 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=27349 =====
>2014/10/30 01:57:47.900344 [ 7638]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=27950 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=27950 =====
>2014/10/30 01:58:02.900696 [ 7638]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=28857 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=28857 =====
>===== Start of debug locks PID=29261 =====
>23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W
>23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=23499 -----
>#0  0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f97e1e49744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=29261 =====
>2014/10/30 01:58:16.504479 [ 7638]: Freeze priority 1
>2014/10/30 01:58:16.505094 [ 7638]: Freeze priority 2
>2014/10/30 01:58:16.506219 [ 7638]: Freeze priority 3
>2014/10/30 01:58:19.525097 [ 7638]: Freeze priority 1
>2014/10/30 01:58:19.525487 [ 7638]: Freeze priority 2
>2014/10/30 01:58:19.525796 [ 7638]: Freeze priority 3
>2014/10/30 01:58:22.934716 [ 7638]: Thawing priority 1
>2014/10/30 01:58:22.934760 [ 7638]: Release freeze handler for prio 1
>2014/10/30 01:58:22.934804 [ 7638]: Thawing priority 2
>2014/10/30 01:58:22.934825 [ 7638]: Release freeze handler for prio 2
>2014/10/30 01:58:22.934858 [ 7638]: Thawing priority 3
>2014/10/30 01:58:22.934888 [ 7638]: Release freeze handler for prio 3
>2014/10/30 01:58:23.290622 [ 7638]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 01:58:46.592550 [ 7638]: Freeze priority 1
>2014/10/30 01:58:46.594777 [ 7638]: Freeze priority 2
>2014/10/30 01:58:46.597023 [ 7638]: Freeze priority 3
>2014/10/30 01:58:49.988048 [ 7638]: Thawing priority 1
>2014/10/30 01:58:49.988102 [ 7638]: Release freeze handler for prio 1
>2014/10/30 01:58:49.988141 [ 7638]: Thawing priority 2
>2014/10/30 01:58:49.988165 [ 7638]: Release freeze handler for prio 2
>2014/10/30 01:58:49.988216 [ 7638]: Thawing priority 3
>2014/10/30 01:58:49.988232 [ 7638]: Release freeze handler for prio 3
>2014/10/30 01:58:50.114793 [ 7638]: 10.interface: Killing TCP connection 10.10.10.205:54715 10.10.10.183:445
>2014/10/30 01:58:50.114966 [ 7638]: 10.interface: Killing TCP connection 10.10.10.206:49692 10.10.10.183:445
>2014/10/30 01:58:50.115095 [ 7638]: 10.interface: Killing TCP connection 10.10.10.208:55833 10.10.10.183:445
>2014/10/30 01:58:50.138472 [ 7638]: 10.interface: Waiting for 2 connections to be killed for IP 10.10.10.183
>2014/10/30 01:58:51.142827 [ 7638]: 10.interface: Killed 3 TCP connections to released IP 10.10.10.183
>2014/10/30 01:58:51.151174 [ 7638]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1
>2014/10/30 01:58:51.545742 [ 7638]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 02:01:19.115676 [ 7638]: Freeze priority 1
>2014/10/30 02:01:19.121665 [ 7638]: Freeze priority 2
>2014/10/30 02:01:19.127993 [ 7638]: Freeze priority 3
>2014/10/30 02:01:22.536733 [ 7638]: Thawing priority 1
>2014/10/30 02:01:22.536825 [ 7638]: Release freeze handler for prio 1
>2014/10/30 02:01:22.536866 [ 7638]: Thawing priority 2
>2014/10/30 02:01:22.536899 [ 7638]: Release freeze handler for prio 2
>2014/10/30 02:01:22.536933 [ 7638]: Thawing priority 3
>2014/10/30 02:01:22.536953 [ 7638]: Release freeze handler for prio 3
>2014/10/30 02:01:33.811722 [ 7638]: Freeze priority 1
>2014/10/30 02:01:33.816157 [ 7638]: Freeze priority 2
>2014/10/30 02:01:33.820637 [ 7638]: Freeze priority 3
>2014/10/30 02:01:36.742206 [ 7638]: Thawing priority 1
>2014/10/30 02:01:36.742264 [ 7638]: Release freeze handler for prio 1
>2014/10/30 02:01:36.742305 [ 7638]: Thawing priority 2
>2014/10/30 02:01:36.742329 [ 7638]: Release freeze handler for prio 2
>2014/10/30 02:01:36.742361 [ 7638]: Thawing priority 3
>2014/10/30 02:01:36.742382 [ 7638]: Release freeze handler for prio 3
>2014/10/30 02:11:22.053583 [ 7638]: Freeze priority 1
>2014/10/30 02:11:22.059544 [ 7638]: Freeze priority 2
>2014/10/30 02:11:22.060649 [ 7638]: Freeze priority 3
>2014/10/30 02:11:24.939058 [ 7638]: Thawing priority 1
>2014/10/30 02:11:24.939132 [ 7638]: Release freeze handler for prio 1
>2014/10/30 02:11:24.939169 [ 7638]: Thawing priority 2
>2014/10/30 02:11:24.939187 [ 7638]: Release freeze handler for prio 2
>2014/10/30 02:11:24.939225 [ 7638]: Thawing priority 3
>2014/10/30 02:11:24.939241 [ 7638]: Release freeze handler for prio 3
>2014/10/30 02:16:41.029396 [ 7638]: Freeze priority 1
>2014/10/30 02:16:41.036342 [ 7638]: Freeze priority 2
>2014/10/30 02:16:41.038555 [ 7638]: Freeze priority 3
>2014/10/30 02:16:44.497775 [ 7638]: Thawing priority 1
>2014/10/30 02:16:44.497810 [ 7638]: Release freeze handler for prio 1
>2014/10/30 02:16:44.497852 [ 7638]: Thawing priority 2
>2014/10/30 02:16:44.497870 [ 7638]: Release freeze handler for prio 2
>2014/10/30 02:16:44.497923 [ 7638]: Thawing priority 3
>2014/10/30 02:16:44.497946 [ 7638]: Release freeze handler for prio 3
>2014/10/30 02:21:26.568922 [ 7638]: Freeze priority 1
>2014/10/30 02:21:26.575640 [ 7638]: Freeze priority 1
>2014/10/30 02:21:26.578669 [ 7638]: Freeze priority 2
>2014/10/30 02:21:26.582962 [ 7638]: Freeze priority 2
>2014/10/30 02:21:26.583195 [ 7638]: Freeze priority 3
>2014/10/30 02:21:26.587978 [ 7638]: Freeze priority 3
>2014/10/30 02:21:30.110244 [recoverd: 7923]: Taking out recovery lock from recovery daemon
>2014/10/30 02:21:30.110329 [recoverd: 7923]: Take the recovery lock
>2014/10/30 02:21:30.123476 [ 7638]: Freeze priority 1
>2014/10/30 02:21:30.123922 [ 7638]: Freeze priority 2
>2014/10/30 02:21:30.124297 [ 7638]: Freeze priority 3
>2014/10/30 02:21:31.964760 [ 7638]: Thawing priority 1
>2014/10/30 02:21:31.964812 [ 7638]: Release freeze handler for prio 1
>2014/10/30 02:21:31.964849 [ 7638]: Thawing priority 2
>2014/10/30 02:21:31.964874 [ 7638]: Release freeze handler for prio 2
>2014/10/30 02:21:31.964928 [ 7638]: Thawing priority 3
>2014/10/30 02:21:31.964952 [ 7638]: Release freeze handler for prio 3
>2014/10/30 02:21:32.566902 [recoverd: 7923]: Resetting ban count to 0 for all nodes
>2014/10/30 02:28:12.098849 [recoverd: 7923]: Taking out recovery lock from recovery daemon
>2014/10/30 02:28:12.148209 [recoverd: 7923]: Take the recovery lock
>2014/10/30 02:28:12.256201 [ 7638]: Freeze priority 1
>2014/10/30 02:28:12.321698 [ 7638]: Freeze priority 2
>2014/10/30 02:28:12.322949 [ 7638]: Freeze priority 3
>2014/10/30 02:28:17.158071 [ 7638]: Thawing priority 1
>2014/10/30 02:28:17.158138 [ 7638]: Release freeze handler for prio 1
>2014/10/30 02:28:17.158182 [ 7638]: Thawing priority 2
>2014/10/30 02:28:17.158203 [ 7638]: Release freeze handler for prio 2
>2014/10/30 02:28:17.158231 [ 7638]: Thawing priority 3
>2014/10/30 02:28:17.158248 [ 7638]: Release freeze handler for prio 3
>2014/10/30 02:28:17.505566 [recoverd: 7923]: Resetting ban count to 0 for all nodes
>2014/10/30 02:31:34.235331 [ 7638]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 02:31:34.247756 [ 7638]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 02:33:01.297452 [ 1515]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 02:33:01.297551 [ 1515]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 02:33:01.297565 [ 1515]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 02:37:00.913407 [ 7503]: Starting CTDBD (Version 2.5.3) as PID: 7503
>2014/10/30 02:37:02.645312 [ 7503]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 02:37:02.669055 [ 7503]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 02:37:02.684931 [ 7503]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 02:37:02.699111 [ 7503]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 02:37:02.699137 [ 7503]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 02:37:02.699148 [ 7503]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 02:37:02.699157 [ 7503]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 02:37:02.699167 [ 7503]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 02:37:02.699176 [ 7503]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 02:37:02.713943 [ 7503]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 02:37:02.728574 [ 7503]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 02:37:02.728618 [ 7503]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 02:37:02.728628 [ 7503]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 02:37:02.743180 [ 7503]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 02:37:02.743280 [ 7503]: Freeze priority 1
>2014/10/30 02:37:02.765381 [ 7503]: Freeze priority 2
>2014/10/30 02:37:02.773712 [ 7503]: Freeze priority 3
>2014/10/30 02:37:02.954602 [ 7503]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 02:37:02.962452 [ 7503]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 02:37:02.965885 [ 7503]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 02:37:03.075670 [ 7503]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 02:37:03.085898 [ 7503]: Freeze priority 1
>2014/10/30 02:37:03.085980 [ 7503]: Freeze priority 2
>2014/10/30 02:37:03.086033 [ 7503]: Freeze priority 3
>2014/10/30 02:37:07.093960 [recoverd: 7786]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 02:37:07.094040 [ 7503]: Freeze priority 1
>2014/10/30 02:37:07.094101 [ 7503]: Freeze priority 2
>2014/10/30 02:37:07.094155 [ 7503]: Freeze priority 3
>2014/10/30 02:37:11.264358 [ 7503]: Freeze priority 1
>2014/10/30 02:37:11.273756 [ 7503]: Freeze priority 2
>2014/10/30 02:37:11.274653 [ 7503]: Freeze priority 3
>2014/10/30 02:37:11.412515 [ 7503]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 02:37:11.413027 [ 7503]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 02:37:11.414361 [ 7503]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 02:37:13.793817 [ 7503]: Thawing priority 1
>2014/10/30 02:37:13.793886 [ 7503]: Release freeze handler for prio 1
>2014/10/30 02:37:13.793927 [ 7503]: Thawing priority 2
>2014/10/30 02:37:13.793947 [ 7503]: Release freeze handler for prio 2
>2014/10/30 02:37:13.793977 [ 7503]: Thawing priority 3
>2014/10/30 02:37:13.793994 [ 7503]: Release freeze handler for prio 3
>2014/10/30 02:37:15.795207 [recoverd: 7786]: Trigger takeoverrun
>2014/10/30 02:37:15.926775 [ 7503]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 02:37:16.215936 [ 7503]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 02:37:16.226772 [ 7503]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 02:37:16.248890 [ 7503]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 02:37:16.421381 [ 7503]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 02:37:18.811484 [ 7503]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 02:37:24.772302 [ 7503]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 02:41:38.129001 [ 7503]: Freeze priority 1
>2014/10/30 02:41:38.130501 [ 7503]: Freeze priority 1
>2014/10/30 02:41:38.130909 [ 7503]: Freeze priority 2
>2014/10/30 02:41:38.131379 [ 7503]: Freeze priority 2
>2014/10/30 02:41:38.132702 [ 7503]: Freeze priority 3
>2014/10/30 02:41:38.133255 [ 7503]: Freeze priority 3
>2014/10/30 02:41:41.153191 [ 7503]: Freeze priority 1
>2014/10/30 02:41:41.153555 [ 7503]: Freeze priority 2
>2014/10/30 02:41:41.153934 [ 7503]: Freeze priority 3
>2014/10/30 02:41:43.693639 [ 7503]: Thawing priority 1
>2014/10/30 02:41:43.693693 [ 7503]: Release freeze handler for prio 1
>2014/10/30 02:41:43.693727 [ 7503]: Thawing priority 2
>2014/10/30 02:41:43.693749 [ 7503]: Release freeze handler for prio 2
>2014/10/30 02:41:43.693779 [ 7503]: Thawing priority 3
>2014/10/30 02:41:43.693798 [ 7503]: Release freeze handler for prio 3
>2014/10/30 02:46:42.735558 [ 7503]: Freeze priority 1
>2014/10/30 02:46:42.749472 [ 7503]: Freeze priority 2
>2014/10/30 02:46:42.751270 [ 7503]: Freeze priority 3
>2014/10/30 02:46:46.714760 [ 7503]: Thawing priority 1
>2014/10/30 02:46:46.714841 [ 7503]: Release freeze handler for prio 1
>2014/10/30 02:46:46.714880 [ 7503]: Thawing priority 2
>2014/10/30 02:46:46.714898 [ 7503]: Release freeze handler for prio 2
>2014/10/30 02:46:46.714926 [ 7503]: Thawing priority 3
>2014/10/30 02:46:46.714941 [ 7503]: Release freeze handler for prio 3
>2014/10/30 02:47:04.546243 [ 7503]: Monitoring event was cancelled
>2014/10/30 02:47:04.546286 [ 7503]: server/eventscript.c:569 Sending SIGTERM to child pid:6864
>2014/10/30 02:56:45.497013 [ 7503]: Freeze priority 1
>2014/10/30 02:56:45.526888 [ 7503]: Freeze priority 1
>2014/10/30 02:56:45.841148 [ 7503]: Freeze priority 1
>2014/10/30 02:56:46.273086 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 02:57:01.273590 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 02:57:16.273885 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 02:57:31.274837 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 02:57:45.500574 [ 7503]: Freeze priority 1
>2014/10/30 02:57:45.528732 [ 7503]: Freeze priority 1
>2014/10/30 02:57:45.839845 [ 7503]: Recovery daemon ping timeout. Count : 0
>2014/10/30 02:57:45.842052 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 02:57:45.842105 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 02:57:45.842131 [recoverd: 7786]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/30 02:57:45.842150 [recoverd: 7786]: Failed to freeze node 1 during recovery. Set it as ban culprit for 4 credits
>2014/10/30 02:57:45.842171 [recoverd: 7786]: Async wait failed - fail_count=1
>2014/10/30 02:57:45.842187 [recoverd: 7786]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/30 02:57:45.842205 [recoverd: 7786]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/30 02:57:45.844141 [ 7503]: Freeze priority 1
>2014/10/30 02:57:46.275940 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 02:58:01.277022 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 02:58:16.278077 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 02:58:31.278506 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 02:58:45.609319 [ 7503]: Freeze priority 1
>2014/10/30 02:58:45.611461 [ 7503]: pnn 2 Invalid reqid 52689 in ctdb_reply_control
>2014/10/30 02:58:45.611592 [ 7503]: Freeze priority 2
>2014/10/30 02:58:45.611773 [ 7503]: Freeze priority 2
>2014/10/30 02:58:45.612647 [ 7503]: Freeze priority 3
>2014/10/30 02:58:45.612839 [ 7503]: Freeze priority 3
>2014/10/30 02:58:45.842603 [ 7503]: Recovery daemon ping timeout. Count : 0
>2014/10/30 02:58:48.621329 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 02:58:48.621392 [recoverd: 7786]: Take the recovery lock
>2014/10/30 02:58:48.633333 [ 7503]: Freeze priority 1
>2014/10/30 02:58:48.633616 [ 7503]: Freeze priority 2
>2014/10/30 02:58:48.633941 [ 7503]: Freeze priority 3
>2014/10/30 02:58:50.360145 [ 7503]: Thawing priority 1
>2014/10/30 02:58:50.360185 [ 7503]: Release freeze handler for prio 1
>2014/10/30 02:58:50.360230 [ 7503]: Thawing priority 2
>2014/10/30 02:58:50.360264 [ 7503]: Release freeze handler for prio 2
>2014/10/30 02:58:50.360299 [ 7503]: Thawing priority 3
>2014/10/30 02:58:50.360320 [ 7503]: Release freeze handler for prio 3
>2014/10/30 02:58:50.362363 [ 7503]: server/ctdb_call.c:1005 reqid 53362 not found
>2014/10/30 02:58:50.362409 [ 7503]: server/ctdb_call.c:1005 reqid 53363 not found
>2014/10/30 02:58:50.747988 [ 7503]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 02:58:50.947847 [recoverd: 7786]: Resetting ban count to 0 for all nodes
>2014/10/30 02:59:15.981154 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 02:59:15.981213 [recoverd: 7786]: Take the recovery lock
>2014/10/30 02:59:16.009161 [ 7503]: Freeze priority 1
>2014/10/30 02:59:16.027925 [ 7503]: Freeze priority 2
>2014/10/30 02:59:16.029506 [ 7503]: Freeze priority 3
>2014/10/30 02:59:19.235865 [ 7503]: Thawing priority 1
>2014/10/30 02:59:19.235929 [ 7503]: Release freeze handler for prio 1
>2014/10/30 02:59:19.235970 [ 7503]: Thawing priority 2
>2014/10/30 02:59:19.235990 [ 7503]: Release freeze handler for prio 2
>2014/10/30 02:59:19.236024 [ 7503]: Thawing priority 3
>2014/10/30 02:59:19.236044 [ 7503]: Release freeze handler for prio 3
>2014/10/30 02:59:19.927863 [recoverd: 7786]: Resetting ban count to 0 for all nodes
>2014/10/30 03:02:08.156267 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 03:02:08.156350 [recoverd: 7786]: Take the recovery lock
>2014/10/30 03:02:08.305469 [ 7503]: Freeze priority 1
>2014/10/30 03:02:08.320271 [ 7503]: Freeze priority 2
>2014/10/30 03:02:08.324911 [ 7503]: Freeze priority 3
>2014/10/30 03:02:11.180106 [ 7503]: Thawing priority 1
>2014/10/30 03:02:11.180158 [ 7503]: Release freeze handler for prio 1
>2014/10/30 03:02:11.180192 [ 7503]: Thawing priority 2
>2014/10/30 03:02:11.180228 [ 7503]: Release freeze handler for prio 2
>2014/10/30 03:02:11.180260 [ 7503]: Thawing priority 3
>2014/10/30 03:02:11.180279 [ 7503]: Release freeze handler for prio 3
>2014/10/30 03:02:11.563575 [recoverd: 7786]: Resetting ban count to 0 for all nodes
>2014/10/30 03:02:29.694994 [ 7503]: 10.interface: Killing TCP connection 10.10.10.205:54728 10.10.10.182:445
>2014/10/30 03:02:29.708249 [ 7503]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.182
>2014/10/30 03:02:29.718685 [ 7503]: 10.interface: Re-adding secondary address 10.10.10.22/24 to dev bond1
>2014/10/30 03:02:30.117155 [ 7503]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 03:06:47.925902 [recoverd: 7786]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 4 vs 3
>2014/10/30 03:06:47.925984 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 03:06:47.926000 [recoverd: 7786]: Take the recovery lock
>2014/10/30 03:06:47.942371 [ 7503]: Freeze priority 1
>2014/10/30 03:06:57.570492 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 03:07:12.570763 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 03:07:27.571127 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 03:07:42.572238 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 03:07:47.917346 [ 7503]: Recovery daemon ping timeout. Count : 0
>2014/10/30 03:07:47.942548 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 03:07:47.942598 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 03:07:47.942620 [recoverd: 7786]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/30 03:07:47.942634 [recoverd: 7786]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits
>2014/10/30 03:07:47.942652 [recoverd: 7786]: Async wait failed - fail_count=1
>2014/10/30 03:07:47.942666 [recoverd: 7786]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/30 03:07:47.942680 [recoverd: 7786]: server/ctdb_recoverd.c:1833 Unable to set recovery mode to active on cluster
>2014/10/30 03:07:47.946158 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 03:07:47.946210 [recoverd: 7786]: Take the recovery lock
>2014/10/30 03:07:47.960600 [ 7503]: Freeze priority 1
>2014/10/30 03:07:57.573259 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 03:08:12.574041 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 03:08:27.574320 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 03:08:42.575168 [ 7503]: Skip monitoring since databases are frozen
>2014/10/30 03:08:47.943575 [ 7503]: Recovery daemon ping timeout. Count : 0
>2014/10/30 03:08:47.960779 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 03:08:47.960836 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 03:08:47.960857 [recoverd: 7786]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/30 03:08:47.960870 [recoverd: 7786]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits
>2014/10/30 03:08:47.960885 [recoverd: 7786]: Async wait failed - fail_count=1
>2014/10/30 03:08:47.960898 [recoverd: 7786]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/30 03:08:47.960910 [recoverd: 7786]: server/ctdb_recoverd.c:1833 Unable to set recovery mode to active on cluster
>2014/10/30 03:08:48.181408 [ 7503]: pnn 2 Invalid reqid 84199 in ctdb_reply_control
>2014/10/30 03:08:48.181452 [ 7503]: pnn 2 Invalid reqid 83859 in ctdb_reply_control
>2014/10/30 03:08:48.183494 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 03:08:48.183543 [recoverd: 7786]: Take the recovery lock
>2014/10/30 03:08:48.197266 [ 7503]: Freeze priority 1
>2014/10/30 03:08:48.197593 [ 7503]: Freeze priority 2
>2014/10/30 03:08:48.198573 [ 7503]: Freeze priority 3
>2014/10/30 03:08:49.975840 [ 7503]: Thawing priority 1
>2014/10/30 03:08:49.975891 [ 7503]: Release freeze handler for prio 1
>2014/10/30 03:08:49.975939 [ 7503]: Thawing priority 2
>2014/10/30 03:08:49.975975 [ 7503]: Release freeze handler for prio 2
>2014/10/30 03:08:49.976006 [ 7503]: Thawing priority 3
>2014/10/30 03:08:49.976037 [ 7503]: Release freeze handler for prio 3
>2014/10/30 03:08:50.334773 [ 7503]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 03:08:50.555091 [recoverd: 7786]: Resetting ban count to 0 for all nodes
>2014/10/30 03:09:18.593542 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 03:09:18.593595 [recoverd: 7786]: Take the recovery lock
>2014/10/30 03:09:18.662372 [ 7503]: Freeze priority 1
>2014/10/30 03:09:18.679631 [ 7503]: Freeze priority 2
>2014/10/30 03:09:18.682498 [ 7503]: Freeze priority 3
>2014/10/30 03:09:20.515392 [ 7503]: Thawing priority 1
>2014/10/30 03:09:20.515454 [ 7503]: Release freeze handler for prio 1
>2014/10/30 03:09:20.515494 [ 7503]: Thawing priority 2
>2014/10/30 03:09:20.515512 [ 7503]: Release freeze handler for prio 2
>2014/10/30 03:09:20.515538 [ 7503]: Thawing priority 3
>2014/10/30 03:09:20.515563 [ 7503]: Release freeze handler for prio 3
>2014/10/30 03:09:22.294252 [recoverd: 7786]: Resetting ban count to 0 for all nodes
>2014/10/30 03:13:26.620019 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 03:13:26.620069 [recoverd: 7786]: Take the recovery lock
>2014/10/30 03:13:26.752064 [ 7503]: Freeze priority 1
>2014/10/30 03:13:26.769881 [ 7503]: Freeze priority 2
>2014/10/30 03:13:26.774293 [ 7503]: Freeze priority 3
>2014/10/30 03:13:30.744914 [ 7503]: Thawing priority 1
>2014/10/30 03:13:30.745021 [ 7503]: Release freeze handler for prio 1
>2014/10/30 03:13:30.745103 [ 7503]: Thawing priority 2
>2014/10/30 03:13:30.745124 [ 7503]: Release freeze handler for prio 2
>2014/10/30 03:13:30.745163 [ 7503]: Thawing priority 3
>2014/10/30 03:13:30.745179 [ 7503]: Release freeze handler for prio 3
>2014/10/30 03:13:31.118725 [recoverd: 7786]: Resetting ban count to 0 for all nodes
>2014/10/30 03:13:41.353980 [recoverd: 7786]: server/ctdb_recoverd.c:3933 Remote node:1 has different flags for node 0. It has 0x02 vs our 0x00
>2014/10/30 03:13:41.354026 [recoverd: 7786]: Use flags 0x00 from local recmaster node for cluster update of node 0 flags
>2014/10/30 03:13:41.359547 [recoverd: 7786]: Taking out recovery lock from recovery daemon
>2014/10/30 03:13:41.359574 [recoverd: 7786]: Take the recovery lock
>2014/10/30 03:13:41.479985 [ 7503]: Freeze priority 1
>2014/10/30 03:13:41.486114 [ 7503]: Freeze priority 2
>2014/10/30 03:13:41.491669 [ 7503]: Freeze priority 3
>2014/10/30 03:13:44.303373 [ 7503]: Thawing priority 1
>2014/10/30 03:13:44.303438 [ 7503]: Release freeze handler for prio 1
>2014/10/30 03:13:44.303477 [ 7503]: Thawing priority 2
>2014/10/30 03:13:44.303495 [ 7503]: Release freeze handler for prio 2
>2014/10/30 03:13:44.303522 [ 7503]: Thawing priority 3
>2014/10/30 03:13:44.303537 [ 7503]: Release freeze handler for prio 3
>2014/10/30 03:13:44.723345 [recoverd: 7786]: Resetting ban count to 0 for all nodes
>2014/10/30 03:14:03.370953 [ 7503]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 03:16:51.010354 [ 7503]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 03:16:51.022907 [ 7503]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 03:18:15.406944 [ 1533]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 03:18:15.407046 [ 1533]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 03:18:15.407060 [ 1533]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 03:22:17.671433 [ 7588]: Starting CTDBD (Version 2.5.3) as PID: 7588
>2014/10/30 03:22:19.234406 [ 7588]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 03:22:19.258034 [ 7588]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 03:22:19.272258 [ 7588]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 03:22:19.286191 [ 7588]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 03:22:19.286210 [ 7588]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 03:22:19.286219 [ 7588]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 03:22:19.286242 [ 7588]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 03:22:19.286251 [ 7588]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 03:22:19.286259 [ 7588]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 03:22:19.300836 [ 7588]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 03:22:19.315182 [ 7588]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 03:22:19.315220 [ 7588]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 03:22:19.315229 [ 7588]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 03:22:19.329221 [ 7588]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 03:22:19.329289 [ 7588]: Freeze priority 1
>2014/10/30 03:22:19.346548 [ 7588]: Freeze priority 2
>2014/10/30 03:22:19.346937 [ 7588]: Freeze priority 3
>2014/10/30 03:22:19.510306 [ 7588]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 03:22:19.511273 [ 7588]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 03:22:19.515314 [ 7588]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 03:22:19.519011 [ 7588]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 03:22:19.639953 [ 7588]: Freeze priority 1
>2014/10/30 03:22:19.640029 [ 7588]: Freeze priority 2
>2014/10/30 03:22:19.640083 [ 7588]: Freeze priority 3
>2014/10/30 03:22:20.074367 [ 7588]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 03:22:25.264022 [ 7588]: Freeze priority 1
>2014/10/30 03:22:25.268474 [ 7588]: Freeze priority 2
>2014/10/30 03:22:25.271899 [ 7588]: Freeze priority 3
>2014/10/30 03:22:25.474461 [ 7588]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 03:22:25.477839 [ 7588]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 03:22:25.487353 [ 7588]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 03:22:29.050428 [ 7588]: Thawing priority 1
>2014/10/30 03:22:29.050471 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:22:29.050499 [ 7588]: Thawing priority 2
>2014/10/30 03:22:29.050509 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:22:29.050525 [ 7588]: Thawing priority 3
>2014/10/30 03:22:29.050546 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:22:43.066351 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:22:43.348291 [ 7588]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 03:22:43.606709 [ 7588]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 03:22:43.620142 [ 7588]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 03:22:43.647203 [ 7588]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 03:22:43.847006 [ 7588]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 03:22:46.276453 [ 7588]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 03:22:47.039955 [ 7588]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 03:26:55.376077 [ 7588]: Freeze priority 1
>2014/10/30 03:26:55.382059 [ 7588]: Freeze priority 1
>2014/10/30 03:26:55.392137 [ 7588]: Freeze priority 1
>2014/10/30 03:26:55.394287 [ 7588]: Freeze priority 2
>2014/10/30 03:26:55.394613 [ 7588]: Freeze priority 2
>2014/10/30 03:26:55.394662 [ 7588]: Freeze priority 2
>2014/10/30 03:26:55.395532 [ 7588]: Freeze priority 3
>2014/10/30 03:26:55.395684 [ 7588]: Freeze priority 3
>2014/10/30 03:26:55.395778 [ 7588]: Freeze priority 3
>2014/10/30 03:26:58.916488 [ 7588]: Freeze priority 1
>2014/10/30 03:26:58.916825 [ 7588]: Freeze priority 2
>2014/10/30 03:26:58.917196 [ 7588]: Freeze priority 3
>2014/10/30 03:27:01.618601 [ 7588]: Thawing priority 1
>2014/10/30 03:27:01.618657 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:27:01.618704 [ 7588]: Thawing priority 2
>2014/10/30 03:27:01.618738 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:27:01.618776 [ 7588]: Thawing priority 3
>2014/10/30 03:27:01.618798 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:27:01.621713 [ 7588]: pnn 2 Invalid reqid 14251 in ctdb_become_dmaster from node 0
>2014/10/30 03:27:01.621814 [ 7588]: server/ctdb_call.c:1005 reqid 14253 not found
>2014/10/30 03:27:01.621967 [ 7588]: pnn 2 Invalid reqid 14252 in ctdb_become_dmaster from node 1
>2014/10/30 03:27:01.622025 [ 7588]: server/ctdb_call.c:1005 reqid 14254 not found
>2014/10/30 03:31:59.045042 [recoverd: 7955]: Taking out recovery lock from recovery daemon
>2014/10/30 03:31:59.099112 [recoverd: 7955]: Take the recovery lock
>2014/10/30 03:31:59.257530 [ 7588]: Freeze priority 1
>2014/10/30 03:31:59.335449 [ 7588]: Freeze priority 2
>2014/10/30 03:31:59.338024 [ 7588]: Freeze priority 3
>2014/10/30 03:32:04.090335 [ 7588]: Thawing priority 1
>2014/10/30 03:32:04.090402 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:32:04.090451 [ 7588]: Thawing priority 2
>2014/10/30 03:32:04.090473 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:32:04.090504 [ 7588]: Thawing priority 3
>2014/10/30 03:32:04.090523 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:32:04.103670 [ 7588]: Freeze priority 1
>2014/10/30 03:32:04.105126 [ 7588]: Freeze priority 2
>2014/10/30 03:32:04.106378 [ 7588]: Freeze priority 3
>2014/10/30 03:32:04.300413 [ 7588]: Refusing to run event scripts call 'recovered' while in recovery
>2014/10/30 03:32:04.300460 [ 7588]: server/ctdb_recover.c:952 Failed to end recovery
>2014/10/30 03:32:04.300496 [recoverd: 7955]: Async operation failed with ret=0 res=-1 opcode=71
>2014/10/30 03:32:04.300527 [recoverd: 7955]: server/ctdb_recoverd.c:239 Node 2 failed the recovered event. Setting it as recovery fail culprit
>2014/10/30 03:32:04.300710 [recoverd: 7955]: Async operation failed with ret=0 res=-1 opcode=71
>2014/10/30 03:32:04.300722 [recoverd: 7955]: server/ctdb_recoverd.c:239 Node 0 failed the recovered event. Setting it as recovery fail culprit
>2014/10/30 03:32:04.300743 [recoverd: 7955]: Async operation failed with ret=0 res=-1 opcode=71
>2014/10/30 03:32:04.300754 [recoverd: 7955]: server/ctdb_recoverd.c:239 Node 3 failed the recovered event. Setting it as recovery fail culprit
>2014/10/30 03:32:04.300767 [recoverd: 7955]: Async operation failed with ret=0 res=-1 opcode=71
>2014/10/30 03:32:04.300776 [recoverd: 7955]: server/ctdb_recoverd.c:239 Node 1 failed the recovered event. Setting it as recovery fail culprit
>2014/10/30 03:32:04.300787 [recoverd: 7955]: Async wait failed - fail_count=4
>2014/10/30 03:32:04.300797 [recoverd: 7955]: server/ctdb_recoverd.c:262 Unable to run the 'recovered' event when called from do_recovery
>2014/10/30 03:32:04.300806 [recoverd: 7955]: server/ctdb_recoverd.c:2016 Unable to run the 'recovered' event on cluster. Recovery process failed.
>2014/10/30 03:32:04.319310 [recoverd: 7955]: Taking out recovery lock from recovery daemon
>2014/10/30 03:32:04.319372 [recoverd: 7955]: Take the recovery lock
>2014/10/30 03:32:04.322003 [recoverd: 7955]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile'
>2014/10/30 03:32:04.322034 [recoverd: 7955]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds
>2014/10/30 03:32:04.322090 [ 7588]: Banning this node for 30 seconds
>2014/10/30 03:32:04.497384 [ 7588]: 10.interface: Killing TCP connection 10.10.10.208:57594 10.10.10.183:445
>2014/10/30 03:32:04.497498 [ 7588]: 10.interface: Killing TCP connection 10.10.10.206:49708 10.10.10.183:445
>2014/10/30 03:32:04.517070 [ 7588]: 10.interface: Killed 2 TCP connections to released IP 10.10.10.183
>2014/10/30 03:32:06.891513 [ 7588]: Thawing priority 1
>2014/10/30 03:32:06.891553 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:32:06.891584 [ 7588]: Thawing priority 2
>2014/10/30 03:32:06.891597 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:32:06.891617 [ 7588]: Thawing priority 3
>2014/10/30 03:32:06.891628 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:32:07.226393 [ 7588]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 03:32:08.304351 [recoverd: 7955]: Node is stopped or banned but recovery mode is not active. Activate recovery mode and lock databases
>2014/10/30 03:32:08.304437 [ 7588]: Freeze priority 1
>2014/10/30 03:32:10.102476 [ 7588]: DB Attach to database ctdb.tdb refused since node is inactive (flags=0x8)
>2014/10/30 03:32:17.432653 [ 7588]: Freeze priority 2
>2014/10/30 03:32:17.432838 [ 7588]: Freeze priority 3
>2014/10/30 03:32:17.476718 [ 7588]: 10.interface: Killing TCP connection 10.10.10.208:57788 10.10.10.183:445
>2014/10/30 03:32:17.482592 [ 7588]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.183
>2014/10/30 03:32:34.322951 [ 7588]: Banning timedout
>2014/10/30 03:32:34.343571 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:34.343714 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:35.344859 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:35.344996 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:36.347636 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:36.347816 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:37.349615 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:37.349817 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:38.350890 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:38.351115 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:39.352767 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:39.352990 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:40.354511 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:40.354745 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:41.355307 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:41.355448 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:42.358255 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:42.358465 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:43.359842 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:43.360015 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:44.361524 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:44.361683 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:45.363477 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:45.363635 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:46.364399 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:46.364513 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:47.366990 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:47.367118 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:48.367994 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:48.368186 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:49.369828 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:49.370001 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:50.371810 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:50.371957 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:51.372780 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:51.372886 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:52.375139 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:52.375250 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:53.377136 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:53.377254 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:54.378530 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:54.378715 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:55.380459 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:55.380636 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:56.382373 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:56.382511 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:57.384189 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:57.384322 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:58.385483 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:58.385606 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:32:59.387322 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:32:59.387500 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:00.389156 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:00.389315 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:01.391140 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:01.391339 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:02.392959 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:02.393144 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:03.395003 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:03.395214 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:04.395798 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:04.395974 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:05.397532 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:05.397675 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:06.399294 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:06.399452 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:07.400647 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:07.400883 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:08.401717 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:08.401919 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:09.404185 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:09.404342 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:10.405469 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:10.405582 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:11.407345 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:11.407541 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:12.409329 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:12.409534 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:13.411189 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:13.411378 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:14.412247 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:14.412444 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:15.413163 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:15.413309 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:16.414874 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:16.415025 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:17.416997 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:17.417192 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:18.418613 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:18.418818 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:18.472498 [ 7588]: Freeze priority 1
>2014/10/30 03:33:19.420103 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:19.420279 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:20.421870 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:20.422052 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:21.423959 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:21.424165 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:22.424305 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:22.424415 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:23.426092 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:23.426226 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:24.428039 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:24.428227 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:25.429934 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:25.430134 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:26.601429 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:26.601623 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:27.432826 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:27.432963 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:28.434699 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:28.434920 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:29.436054 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:29.436177 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:30.437574 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:30.437724 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:31.439025 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:31.439154 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:32.440731 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:32.440844 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:33.442712 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:33.442893 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:34.444529 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:34.444694 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:35.446108 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:35.446263 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:36.447759 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:36.447940 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:37.450006 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:37.450205 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:38.451004 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:38.451178 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:39.452875 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:39.453057 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:40.454592 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:40.454756 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:41.455187 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:41.455398 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:42.457234 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:42.457438 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:43.459649 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:43.459790 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:44.461537 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:44.461758 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:45.463211 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:45.463376 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:46.465040 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:46.465246 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:47.466843 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:47.467039 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:48.467611 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:48.467837 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:49.469478 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:49.469711 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:50.470409 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:50.470514 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:51.472879 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:51.473020 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:52.473280 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:52.473364 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:53.475945 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:53.476150 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:54.477872 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:54.478044 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:55.479638 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:55.479779 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:56.481257 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:56.481404 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:57.483065 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:57.483256 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:58.484610 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:58.484742 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:33:59.486371 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:33:59.486581 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:00.487767 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:00.487911 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:01.489396 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:01.489546 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:02.491169 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:02.491316 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:03.492951 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:03.493087 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:04.494404 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:04.494531 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:05.496204 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:05.496358 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:06.498060 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:06.498183 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:07.499578 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:07.499733 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:08.306355 [ 7588]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS
>2014/10/30 03:34:08.501924 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:08.502083 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:09.503735 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:09.503946 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:10.505610 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:10.505741 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:11.507143 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:11.507296 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:12.509282 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:12.509442 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:13.510987 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:13.511153 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:14.511876 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:14.512048 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:15.512955 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:15.513103 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:16.514449 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:16.514574 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:17.515783 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it
>2014/10/30 03:34:17.516016 [recoverd: 7955]: Trigger takeoverrun
>2014/10/30 03:34:18.517832 [ 7588]: Freeze priority 1
>2014/10/30 03:34:18.518371 [ 7588]: Freeze priority 2
>2014/10/30 03:34:18.519562 [ 7588]: Freeze priority 3
>2014/10/30 03:34:22.100215 [ 7588]: Freeze priority 1
>2014/10/30 03:34:22.100630 [ 7588]: Freeze priority 2
>2014/10/30 03:34:22.100944 [ 7588]: Freeze priority 3
>2014/10/30 03:34:24.700205 [ 7588]: Thawing priority 1
>2014/10/30 03:34:24.700247 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:34:24.700287 [ 7588]: Thawing priority 2
>2014/10/30 03:34:24.700310 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:34:24.700343 [ 7588]: Thawing priority 3
>2014/10/30 03:34:24.700362 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:34:24.709824 [ 7588]: pnn 2 Invalid reqid 32094 in ctdb_become_dmaster from node 1
>2014/10/30 03:34:24.709990 [ 7588]: server/ctdb_call.c:1005 reqid 32095 not found
>2014/10/30 03:34:25.298089 [ 7588]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 03:34:36.082492 [ 7588]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 03:34:48.770118 [ 7588]: Freeze priority 1
>2014/10/30 03:34:48.778944 [ 7588]: Freeze priority 2
>2014/10/30 03:34:48.785468 [ 7588]: Freeze priority 3
>2014/10/30 03:34:52.036874 [ 7588]: Thawing priority 1
>2014/10/30 03:34:52.036923 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:34:52.036960 [ 7588]: Thawing priority 2
>2014/10/30 03:34:52.036980 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:34:52.037010 [ 7588]: Thawing priority 3
>2014/10/30 03:34:52.037029 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:42:01.561496 [ 7588]: Freeze priority 1
>2014/10/30 03:42:01.662581 [ 7588]: Freeze priority 2
>2014/10/30 03:42:01.668700 [ 7588]: Freeze priority 3
>2014/10/30 03:42:04.914056 [ 7588]: Thawing priority 1
>2014/10/30 03:42:04.914113 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:42:04.914142 [ 7588]: Thawing priority 2
>2014/10/30 03:42:04.914169 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:42:04.914194 [ 7588]: Thawing priority 3
>2014/10/30 03:42:04.914209 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:47:22.946969 [ 7588]: Freeze priority 1
>2014/10/30 03:47:23.009728 [ 7588]: Freeze priority 2
>2014/10/30 03:47:23.012149 [ 7588]: Freeze priority 3
>2014/10/30 03:47:28.389886 [ 7588]: Thawing priority 1
>2014/10/30 03:47:28.389937 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:47:28.389971 [ 7588]: Thawing priority 2
>2014/10/30 03:47:28.389990 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:47:28.390017 [ 7588]: Thawing priority 3
>2014/10/30 03:47:28.390033 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:52:06.238108 [ 7588]: Freeze priority 1
>2014/10/30 03:52:06.325681 [ 7588]: Freeze priority 2
>2014/10/30 03:52:06.327546 [ 7588]: Freeze priority 3
>2014/10/30 03:52:06.761278 [ 7588]: Freeze priority 1
>2014/10/30 03:52:06.761736 [ 7588]: Freeze priority 2
>2014/10/30 03:52:06.762447 [ 7588]: Freeze priority 3
>2014/10/30 03:52:09.785923 [ 7588]: Freeze priority 1
>2014/10/30 03:52:09.786240 [ 7588]: Freeze priority 2
>2014/10/30 03:52:09.786581 [ 7588]: Freeze priority 3
>2014/10/30 03:52:12.148658 [ 7588]: Thawing priority 1
>2014/10/30 03:52:12.148699 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:52:12.148734 [ 7588]: Thawing priority 2
>2014/10/30 03:52:12.148755 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:52:12.148785 [ 7588]: Thawing priority 3
>2014/10/30 03:52:12.148804 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:58:50.345012 [ 7588]: Freeze priority 1
>2014/10/30 03:58:50.436617 [ 7588]: Freeze priority 2
>2014/10/30 03:58:50.438775 [ 7588]: Freeze priority 3
>2014/10/30 03:58:53.746872 [ 7588]: Thawing priority 1
>2014/10/30 03:58:53.746966 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:58:53.747024 [ 7588]: Thawing priority 2
>2014/10/30 03:58:53.747043 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:58:53.747081 [ 7588]: Thawing priority 3
>2014/10/30 03:58:53.747098 [ 7588]: Release freeze handler for prio 3
>2014/10/30 03:59:04.441232 [ 7588]: Freeze priority 1
>2014/10/30 03:59:04.463043 [ 7588]: Freeze priority 2
>2014/10/30 03:59:04.465833 [ 7588]: Freeze priority 3
>2014/10/30 03:59:07.613588 [ 7588]: Thawing priority 1
>2014/10/30 03:59:07.613626 [ 7588]: Release freeze handler for prio 1
>2014/10/30 03:59:07.613659 [ 7588]: Thawing priority 2
>2014/10/30 03:59:07.613680 [ 7588]: Release freeze handler for prio 2
>2014/10/30 03:59:07.613710 [ 7588]: Thawing priority 3
>2014/10/30 03:59:07.613729 [ 7588]: Release freeze handler for prio 3
>2014/10/30 04:02:15.338221 [ 7588]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 04:02:15.414907 [ 7588]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/30 04:02:15.414943 [ 7588]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 04:03:44.616786 [ 1551]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 04:03:44.616903 [ 1551]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 04:03:44.616917 [ 1551]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 04:07:46.192240 [ 7647]: Starting CTDBD (Version 2.5.3) as PID: 7647
>2014/10/30 04:07:47.785871 [ 7647]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 04:07:47.811655 [ 7647]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 04:07:47.826072 [ 7647]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 04:07:47.840102 [ 7647]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 04:07:47.840121 [ 7647]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 04:07:47.840131 [ 7647]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 04:07:47.840140 [ 7647]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 04:07:47.840149 [ 7647]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 04:07:47.840168 [ 7647]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 04:07:47.854144 [ 7647]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 04:07:47.868074 [ 7647]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 04:07:47.868093 [ 7647]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 04:07:47.868102 [ 7647]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 04:07:47.882256 [ 7647]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 04:07:47.882292 [ 7647]: Freeze priority 1
>2014/10/30 04:07:47.902607 [ 7647]: Freeze priority 2
>2014/10/30 04:07:47.903021 [ 7647]: Freeze priority 3
>2014/10/30 04:07:48.067160 [ 7647]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 04:07:48.068217 [ 7647]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 04:07:48.071283 [ 7647]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 04:07:48.074612 [ 7647]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 04:07:48.194681 [ 7647]: Freeze priority 1
>2014/10/30 04:07:48.194779 [ 7647]: Freeze priority 2
>2014/10/30 04:07:48.194867 [ 7647]: Freeze priority 3
>2014/10/30 04:07:53.398416 [ 7647]: Freeze priority 1
>2014/10/30 04:07:53.409735 [ 7647]: Freeze priority 2
>2014/10/30 04:07:53.412494 [ 7647]: Freeze priority 3
>2014/10/30 04:07:53.591233 [ 7647]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 04:07:53.592423 [ 7647]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 04:07:53.596675 [ 7647]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 04:07:55.940766 [ 7647]: Thawing priority 1
>2014/10/30 04:07:55.940819 [ 7647]: Release freeze handler for prio 1
>2014/10/30 04:07:55.940853 [ 7647]: Thawing priority 2
>2014/10/30 04:07:55.940882 [ 7647]: Release freeze handler for prio 2
>2014/10/30 04:07:55.940907 [ 7647]: Thawing priority 3
>2014/10/30 04:07:55.940922 [ 7647]: Release freeze handler for prio 3
>2014/10/30 04:08:09.960816 [recoverd: 7985]: Trigger takeoverrun
>2014/10/30 04:08:10.439709 [ 7647]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 04:08:10.752987 [ 7647]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 04:08:10.763745 [ 7647]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 04:08:10.811286 [ 7647]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 04:08:10.974705 [ 7647]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 04:08:13.376196 [ 7647]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/30 04:08:14.896121 [ 7647]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 04:12:18.715733 [ 7647]: Freeze priority 1
>2014/10/30 04:12:18.716787 [ 7647]: Freeze priority 2
>2014/10/30 04:12:18.717632 [ 7647]: Freeze priority 3
>2014/10/30 04:12:20.947401 [ 7647]: Thawing priority 1
>2014/10/30 04:12:20.947445 [ 7647]: Release freeze handler for prio 1
>2014/10/30 04:12:20.947475 [ 7647]: Thawing priority 2
>2014/10/30 04:12:20.947491 [ 7647]: Release freeze handler for prio 2
>2014/10/30 04:12:20.947527 [ 7647]: Thawing priority 3
>2014/10/30 04:12:20.947543 [ 7647]: Release freeze handler for prio 3
>2014/10/30 04:17:33.022404 [ 7647]: Freeze priority 1
>2014/10/30 04:17:33.032998 [ 7647]: Freeze priority 2
>2014/10/30 04:17:33.034769 [ 7647]: Freeze priority 3
>2014/10/30 04:17:36.086530 [ 7647]: Thawing priority 1
>2014/10/30 04:17:36.086598 [ 7647]: Release freeze handler for prio 1
>2014/10/30 04:17:36.086635 [ 7647]: Thawing priority 2
>2014/10/30 04:17:36.086655 [ 7647]: Release freeze handler for prio 2
>2014/10/30 04:17:36.086684 [ 7647]: Thawing priority 3
>2014/10/30 04:17:36.086714 [ 7647]: Release freeze handler for prio 3
>2014/10/30 04:27:31.878797 [ 7647]: Freeze priority 1
>2014/10/30 04:27:31.899677 [ 7647]: Freeze priority 1
>2014/10/30 04:27:31.956293 [ 7647]: Freeze priority 2
>2014/10/30 04:27:31.957087 [ 7647]: Freeze priority 2
>2014/10/30 04:27:31.961223 [ 7647]: Freeze priority 3
>2014/10/30 04:27:31.962039 [ 7647]: Freeze priority 3
>2014/10/30 04:27:34.981999 [ 7647]: Freeze priority 1
>2014/10/30 04:27:34.982392 [ 7647]: Freeze priority 2
>2014/10/30 04:27:34.982681 [ 7647]: Freeze priority 3
>2014/10/30 04:27:39.839753 [ 7647]: Thawing priority 1
>2014/10/30 04:27:39.839820 [ 7647]: Release freeze handler for prio 1
>2014/10/30 04:27:39.839862 [ 7647]: Thawing priority 2
>2014/10/30 04:27:39.839900 [ 7647]: Release freeze handler for prio 2
>2014/10/30 04:27:39.839939 [ 7647]: Thawing priority 3
>2014/10/30 04:27:39.839971 [ 7647]: Release freeze handler for prio 3
>2014/10/30 04:32:54.023580 [ 7647]: Freeze priority 1
>2014/10/30 04:32:54.143548 [ 7647]: Freeze priority 2
>2014/10/30 04:32:54.146889 [ 7647]: Freeze priority 3
>2014/10/30 04:32:59.106296 [ 7647]: Thawing priority 1
>2014/10/30 04:32:59.106355 [ 7647]: Release freeze handler for prio 1
>2014/10/30 04:32:59.106401 [ 7647]: Thawing priority 2
>2014/10/30 04:32:59.106426 [ 7647]: Release freeze handler for prio 2
>2014/10/30 04:32:59.106460 [ 7647]: Thawing priority 3
>2014/10/30 04:32:59.106484 [ 7647]: Release freeze handler for prio 3
>2014/10/30 04:37:42.159685 [ 7647]: Freeze priority 1
>2014/10/30 04:37:42.161784 [ 7647]: Freeze priority 1
>2014/10/30 04:37:42.176155 [ 7647]: Freeze priority 2
>2014/10/30 04:37:42.178375 [ 7647]: Freeze priority 2
>2014/10/30 04:37:42.181139 [ 7647]: Freeze priority 3
>2014/10/30 04:37:42.184336 [ 7647]: Freeze priority 3
>2014/10/30 04:37:45.707207 [recoverd: 7985]: Taking out recovery lock from recovery daemon
>2014/10/30 04:37:45.707300 [recoverd: 7985]: Take the recovery lock
>2014/10/30 04:37:45.719799 [ 7647]: Freeze priority 1
>2014/10/30 04:37:45.720139 [ 7647]: Freeze priority 2
>2014/10/30 04:37:45.720435 [ 7647]: Freeze priority 3
>2014/10/30 04:37:47.636054 [ 7647]: Thawing priority 1
>2014/10/30 04:37:47.636101 [ 7647]: Release freeze handler for prio 1
>2014/10/30 04:37:47.636135 [ 7647]: Thawing priority 2
>2014/10/30 04:37:47.636164 [ 7647]: Release freeze handler for prio 2
>2014/10/30 04:37:47.636189 [ 7647]: Thawing priority 3
>2014/10/30 04:37:47.636215 [ 7647]: Release freeze handler for prio 3
>2014/10/30 04:37:48.232306 [recoverd: 7985]: Resetting ban count to 0 for all nodes
>2014/10/30 04:44:25.247222 [ 7647]: Freeze priority 1
>2014/10/30 04:44:25.395776 [ 7647]: Freeze priority 2
>2014/10/30 04:44:25.397074 [ 7647]: Freeze priority 3
>2014/10/30 04:44:28.875769 [ 7647]: Thawing priority 1
>2014/10/30 04:44:28.875838 [ 7647]: Release freeze handler for prio 1
>2014/10/30 04:44:28.875891 [ 7647]: Thawing priority 2
>2014/10/30 04:44:28.875912 [ 7647]: Release freeze handler for prio 2
>2014/10/30 04:44:28.875942 [ 7647]: Thawing priority 3
>2014/10/30 04:44:28.875961 [ 7647]: Release freeze handler for prio 3
>2014/10/30 04:44:32.036422 [ 7647]: Freeze priority 1
>2014/10/30 04:44:32.041913 [ 7647]: Freeze priority 2
>2014/10/30 04:44:32.045477 [ 7647]: Freeze priority 3
>2014/10/30 04:44:35.136655 [ 7647]: Thawing priority 1
>2014/10/30 04:44:35.136724 [ 7647]: Release freeze handler for prio 1
>2014/10/30 04:44:35.136772 [ 7647]: Thawing priority 2
>2014/10/30 04:44:35.136791 [ 7647]: Release freeze handler for prio 2
>2014/10/30 04:44:35.136818 [ 7647]: Thawing priority 3
>2014/10/30 04:44:35.136835 [ 7647]: Release freeze handler for prio 3
>2014/10/30 04:47:47.417472 [ 7647]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 04:47:47.428464 [ 7647]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 04:49:12.932961 [ 1531]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 04:49:12.933103 [ 1531]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 04:49:12.933125 [ 1531]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 04:53:13.680727 [ 7771]: Starting CTDBD (Version 2.5.3) as PID: 7771
>2014/10/30 04:53:15.121045 [ 7771]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 04:53:15.145409 [ 7771]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 04:53:15.159569 [ 7771]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 04:53:15.173540 [ 7771]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 04:53:15.173559 [ 7771]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 04:53:15.173567 [ 7771]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 04:53:15.173576 [ 7771]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 04:53:15.173584 [ 7771]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 04:53:15.173592 [ 7771]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 04:53:15.187917 [ 7771]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 04:53:15.202114 [ 7771]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 04:53:15.202137 [ 7771]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 04:53:15.202146 [ 7771]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 04:53:15.216102 [ 7771]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 04:53:15.216134 [ 7771]: Freeze priority 1
>2014/10/30 04:53:15.232896 [ 7771]: Freeze priority 2
>2014/10/30 04:53:15.233244 [ 7771]: Freeze priority 3
>2014/10/30 04:53:15.397452 [ 7771]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 04:53:15.401512 [ 7771]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 04:53:15.405162 [ 7771]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 04:53:15.527117 [ 7771]: Freeze priority 1
>2014/10/30 04:53:15.527191 [ 7771]: Freeze priority 2
>2014/10/30 04:53:15.527245 [ 7771]: Freeze priority 3
>2014/10/30 04:53:15.855657 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 04:53:15.857534 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 04:53:15.858167 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 04:53:15.858488 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 04:53:15.900048 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x6afb8c09 on local node
>2014/10/30 04:53:15.900116 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node
>2014/10/30 04:53:15.900133 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x4e66c2b2 on local node
>2014/10/30 04:53:15.900146 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x6afb8c09 on local node
>2014/10/30 04:53:15.900158 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node
>2014/10/30 04:53:15.900173 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x4e66c2b2 on local node
>2014/10/30 04:53:15.902425 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 04:53:19.534430 [recoverd: 8056]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 04:53:19.534487 [ 7771]: Freeze priority 1
>2014/10/30 04:53:19.534549 [ 7771]: Freeze priority 2
>2014/10/30 04:53:19.534602 [ 7771]: Freeze priority 3
>2014/10/30 04:53:24.032569 [ 7771]: Freeze priority 1
>2014/10/30 04:53:24.043522 [recoverd: 8056]: Trigger takeoverrun
>2014/10/30 04:53:24.045707 [ 7771]: Freeze priority 2
>2014/10/30 04:53:24.047282 [ 7771]: Freeze priority 3
>2014/10/30 04:53:24.222051 [ 7771]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 04:53:24.224411 [ 7771]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 04:53:24.228830 [ 7771]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 04:53:26.626588 [ 7771]: Thawing priority 1
>2014/10/30 04:53:26.626637 [ 7771]: Release freeze handler for prio 1
>2014/10/30 04:53:26.626695 [ 7771]: Thawing priority 2
>2014/10/30 04:53:26.626717 [ 7771]: Release freeze handler for prio 2
>2014/10/30 04:53:26.626745 [ 7771]: Thawing priority 3
>2014/10/30 04:53:26.626762 [ 7771]: Release freeze handler for prio 3
>2014/10/30 04:53:28.633226 [recoverd: 8056]: Trigger takeoverrun
>2014/10/30 04:53:28.785751 [ 7771]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 04:53:29.043945 [ 7771]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 04:53:29.057538 [ 7771]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 04:53:29.083971 [ 7771]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 04:53:29.272008 [ 7771]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 04:53:31.724700 [ 7771]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation
>2014/10/30 04:53:37.567001 [ 7771]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 04:57:50.473555 [ 7771]: Freeze priority 1
>2014/10/30 04:57:50.499675 [ 7771]: Freeze priority 2
>2014/10/30 04:57:50.500750 [ 7771]: Freeze priority 3
>2014/10/30 04:57:52.919220 [ 7771]: Thawing priority 1
>2014/10/30 04:57:52.919265 [ 7771]: Release freeze handler for prio 1
>2014/10/30 04:57:52.919304 [ 7771]: Thawing priority 2
>2014/10/30 04:57:52.919327 [ 7771]: Release freeze handler for prio 2
>2014/10/30 04:57:52.919372 [ 7771]: Thawing priority 3
>2014/10/30 04:57:52.919390 [ 7771]: Release freeze handler for prio 3
>2014/10/30 05:02:52.883034 [ 7771]: Freeze priority 1
>2014/10/30 05:02:52.987170 [ 7771]: Freeze priority 2
>2014/10/30 05:02:52.988812 [ 7771]: Freeze priority 3
>2014/10/30 05:02:58.771496 [ 7771]: Thawing priority 1
>2014/10/30 05:02:58.771570 [ 7771]: Release freeze handler for prio 1
>2014/10/30 05:02:58.771608 [ 7771]: Thawing priority 2
>2014/10/30 05:02:58.771635 [ 7771]: Release freeze handler for prio 2
>2014/10/30 05:02:58.771684 [ 7771]: Thawing priority 3
>2014/10/30 05:02:58.771703 [ 7771]: Release freeze handler for prio 3
>2014/10/30 05:03:09.183111 [ 7771]: Freeze priority 1
>2014/10/30 05:03:09.185291 [ 7771]: Freeze priority 2
>2014/10/30 05:03:09.187527 [ 7771]: Freeze priority 3
>2014/10/30 05:03:11.867896 [ 7771]: Thawing priority 1
>2014/10/30 05:03:11.867950 [ 7771]: Release freeze handler for prio 1
>2014/10/30 05:03:11.867988 [ 7771]: Thawing priority 2
>2014/10/30 05:03:11.868008 [ 7771]: Release freeze handler for prio 2
>2014/10/30 05:03:11.868038 [ 7771]: Thawing priority 3
>2014/10/30 05:03:11.868056 [ 7771]: Release freeze handler for prio 3
>2014/10/30 05:12:59.002841 [ 7771]: Freeze priority 1
>2014/10/30 05:12:59.003116 [ 7771]: Freeze priority 1
>2014/10/30 05:12:59.048410 [ 7771]: Freeze priority 1
>2014/10/30 05:12:59.179316 [ 7771]: Freeze priority 2
>2014/10/30 05:12:59.179524 [ 7771]: Freeze priority 2
>2014/10/30 05:12:59.180441 [ 7771]: Freeze priority 2
>2014/10/30 05:12:59.180683 [ 7771]: Freeze priority 3
>2014/10/30 05:12:59.181563 [ 7771]: Freeze priority 3
>2014/10/30 05:12:59.182681 [ 7771]: Freeze priority 3
>2014/10/30 05:13:02.245321 [ 7771]: Freeze priority 1
>2014/10/30 05:13:02.245697 [ 7771]: Freeze priority 2
>2014/10/30 05:13:02.245992 [ 7771]: Freeze priority 3
>2014/10/30 05:13:07.586542 [ 7771]: Thawing priority 1
>2014/10/30 05:13:07.586607 [ 7771]: Release freeze handler for prio 1
>2014/10/30 05:13:07.586643 [ 7771]: Thawing priority 2
>2014/10/30 05:13:07.586676 [ 7771]: Release freeze handler for prio 2
>2014/10/30 05:13:07.586705 [ 7771]: Thawing priority 3
>2014/10/30 05:13:07.586736 [ 7771]: Release freeze handler for prio 3
>2014/10/30 05:18:21.744705 [ 7771]: Freeze priority 1
>2014/10/30 05:18:21.858673 [ 7771]: Freeze priority 2
>2014/10/30 05:18:21.862773 [ 7771]: Freeze priority 3
>2014/10/30 05:18:28.282940 [ 7771]: Thawing priority 1
>2014/10/30 05:18:28.283003 [ 7771]: Release freeze handler for prio 1
>2014/10/30 05:18:28.283040 [ 7771]: Thawing priority 2
>2014/10/30 05:18:28.283060 [ 7771]: Release freeze handler for prio 2
>2014/10/30 05:18:28.283089 [ 7771]: Thawing priority 3
>2014/10/30 05:18:28.283108 [ 7771]: Release freeze handler for prio 3
>2014/10/30 05:23:10.398652 [ 7771]: Freeze priority 1
>2014/10/30 05:23:10.417348 [ 7771]: Freeze priority 1
>2014/10/30 05:23:10.419947 [ 7771]: Freeze priority 2
>2014/10/30 05:23:10.420109 [ 7771]: Freeze priority 2
>2014/10/30 05:23:10.421070 [ 7771]: Freeze priority 3
>2014/10/30 05:23:10.421293 [ 7771]: Freeze priority 3
>2014/10/30 05:23:13.933107 [recoverd: 8056]: Taking out recovery lock from recovery daemon
>2014/10/30 05:23:13.933178 [recoverd: 8056]: Take the recovery lock
>2014/10/30 05:23:13.945108 [ 7771]: Freeze priority 1
>2014/10/30 05:23:13.945478 [ 7771]: Freeze priority 2
>2014/10/30 05:23:13.946153 [ 7771]: Freeze priority 3
>2014/10/30 05:23:15.813040 [ 7771]: Thawing priority 1
>2014/10/30 05:23:15.813085 [ 7771]: Release freeze handler for prio 1
>2014/10/30 05:23:15.813131 [ 7771]: Thawing priority 2
>2014/10/30 05:23:15.813150 [ 7771]: Release freeze handler for prio 2
>2014/10/30 05:23:15.813177 [ 7771]: Thawing priority 3
>2014/10/30 05:23:15.813194 [ 7771]: Release freeze handler for prio 3
>2014/10/30 05:23:16.368044 [recoverd: 8056]: Resetting ban count to 0 for all nodes
>2014/10/30 05:29:57.927301 [recoverd: 8056]: Taking out recovery lock from recovery daemon
>2014/10/30 05:29:57.927377 [recoverd: 8056]: Take the recovery lock
>2014/10/30 05:29:58.056676 [ 7771]: Freeze priority 1
>2014/10/30 05:29:58.072482 [ 7771]: Freeze priority 2
>2014/10/30 05:29:58.076227 [ 7771]: Freeze priority 3
>2014/10/30 05:30:00.671686 [ 7771]: Thawing priority 1
>2014/10/30 05:30:00.671727 [ 7771]: Release freeze handler for prio 1
>2014/10/30 05:30:00.671758 [ 7771]: Thawing priority 2
>2014/10/30 05:30:00.671775 [ 7771]: Release freeze handler for prio 2
>2014/10/30 05:30:00.671810 [ 7771]: Thawing priority 3
>2014/10/30 05:30:00.671829 [ 7771]: Release freeze handler for prio 3
>2014/10/30 05:30:01.029404 [recoverd: 8056]: Resetting ban count to 0 for all nodes
>2014/10/30 05:33:11.044634 [ 7771]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/30 05:33:11.128218 [ 7771]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 05:33:11.139454 [ 7771]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 05:34:36.323395 [ 1540]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 05:34:36.323539 [ 1540]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 05:34:36.323563 [ 1540]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 05:38:38.353089 [ 7506]: Starting CTDBD (Version 2.5.3) as PID: 7506
>2014/10/30 05:38:40.043395 [ 7506]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 05:38:40.067059 [ 7506]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 05:38:40.081267 [ 7506]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 05:38:40.095497 [ 7506]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 05:38:40.095517 [ 7506]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 05:38:40.095527 [ 7506]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 05:38:40.095536 [ 7506]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 05:38:40.095545 [ 7506]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 05:38:40.095555 [ 7506]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 05:38:40.109613 [ 7506]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 05:38:40.123674 [ 7506]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 05:38:40.123693 [ 7506]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 05:38:40.123704 [ 7506]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 05:38:40.137713 [ 7506]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 05:38:40.137745 [ 7506]: Freeze priority 1
>2014/10/30 05:38:40.155074 [ 7506]: Freeze priority 2
>2014/10/30 05:38:40.155412 [ 7506]: Freeze priority 3
>2014/10/30 05:38:40.343792 [ 7506]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 05:38:40.343933 [ 7506]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 05:38:40.343956 [ 7506]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 05:38:40.344169 [ 7506]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 05:38:40.344972 [ 7506]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 05:38:40.348992 [ 7506]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 05:38:40.352577 [ 7506]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 05:38:40.490609 [ 7506]: Freeze priority 1
>2014/10/30 05:38:40.490695 [ 7506]: Freeze priority 2
>2014/10/30 05:38:40.490754 [ 7506]: Freeze priority 3
>2014/10/30 05:38:44.197140 [ 7506]: Freeze priority 1
>2014/10/30 05:38:44.321422 [ 7506]: Freeze priority 2
>2014/10/30 05:38:44.322816 [ 7506]: Freeze priority 3
>2014/10/30 05:38:44.497289 [recoverd: 7843]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 05:38:44.497376 [ 7506]: Freeze priority 1
>2014/10/30 05:38:44.497436 [ 7506]: Freeze priority 2
>2014/10/30 05:38:44.497488 [ 7506]: Freeze priority 3
>2014/10/30 05:38:48.460242 [ 7506]: Thawing priority 1
>2014/10/30 05:38:48.460291 [ 7506]: Release freeze handler for prio 1
>2014/10/30 05:38:48.460324 [ 7506]: Thawing priority 2
>2014/10/30 05:38:48.460341 [ 7506]: Release freeze handler for prio 2
>2014/10/30 05:38:48.460377 [ 7506]: Thawing priority 3
>2014/10/30 05:38:48.460393 [ 7506]: Release freeze handler for prio 3
>2014/10/30 05:38:49.505487 [recoverd: 7843]: Trigger takeoverrun
>2014/10/30 05:38:59.033521 [ 7506]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 05:38:59.086695 [ 7506]: Freeze priority 1
>2014/10/30 05:38:59.090976 [ 7506]: Freeze priority 2
>2014/10/30 05:38:59.093362 [ 7506]: Freeze priority 3
>2014/10/30 05:38:59.269732 [ 7506]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 05:38:59.273421 [ 7506]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 05:39:01.508278 [ 7506]: Thawing priority 1
>2014/10/30 05:39:01.508376 [ 7506]: Release freeze handler for prio 1
>2014/10/30 05:39:01.508435 [ 7506]: Thawing priority 2
>2014/10/30 05:39:01.508455 [ 7506]: Release freeze handler for prio 2
>2014/10/30 05:39:01.508493 [ 7506]: Thawing priority 3
>2014/10/30 05:39:01.508512 [ 7506]: Release freeze handler for prio 3
>2014/10/30 05:39:15.545467 [recoverd: 7843]: Trigger takeoverrun
>2014/10/30 05:39:15.880479 [ 7506]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 05:39:16.236680 [ 7506]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 05:39:16.248322 [ 7506]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 05:39:16.271276 [ 7506]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 05:39:16.485987 [ 7506]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 05:39:18.872663 [ 7506]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 05:39:19.407094 [ 7506]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 05:43:13.855039 [ 7506]: Freeze priority 1
>2014/10/30 05:43:13.867353 [ 7506]: Freeze priority 1
>2014/10/30 05:43:13.873364 [ 7506]: Freeze priority 1
>2014/10/30 05:43:13.906402 [ 7506]: Freeze priority 2
>2014/10/30 05:43:13.906604 [ 7506]: Freeze priority 2
>2014/10/30 05:43:13.906673 [ 7506]: Freeze priority 2
>2014/10/30 05:43:13.907455 [ 7506]: Freeze priority 3
>2014/10/30 05:43:13.908298 [ 7506]: Freeze priority 3
>2014/10/30 05:43:13.908371 [ 7506]: Freeze priority 3
>2014/10/30 05:43:16.927830 [ 7506]: Freeze priority 1
>2014/10/30 05:43:16.928260 [ 7506]: Freeze priority 2
>2014/10/30 05:43:16.928552 [ 7506]: Freeze priority 3
>2014/10/30 05:43:21.547204 [ 7506]: Thawing priority 1
>2014/10/30 05:43:21.547277 [ 7506]: Release freeze handler for prio 1
>2014/10/30 05:43:21.547324 [ 7506]: Thawing priority 2
>2014/10/30 05:43:21.547340 [ 7506]: Release freeze handler for prio 2
>2014/10/30 05:43:21.547377 [ 7506]: Thawing priority 3
>2014/10/30 05:43:21.547392 [ 7506]: Release freeze handler for prio 3
>2014/10/30 05:48:21.663997 [ 7506]: Freeze priority 1
>2014/10/30 05:48:21.685963 [ 7506]: Freeze priority 2
>2014/10/30 05:48:21.690421 [ 7506]: Freeze priority 3
>2014/10/30 05:48:24.857674 [ 7506]: Thawing priority 1
>2014/10/30 05:48:24.857751 [ 7506]: Release freeze handler for prio 1
>2014/10/30 05:48:24.857789 [ 7506]: Thawing priority 2
>2014/10/30 05:48:24.857836 [ 7506]: Release freeze handler for prio 2
>2014/10/30 05:48:24.857874 [ 7506]: Thawing priority 3
>2014/10/30 05:48:24.857894 [ 7506]: Release freeze handler for prio 3
>2014/10/30 05:58:21.993530 [ 7506]: Freeze priority 1
>2014/10/30 05:58:21.995952 [ 7506]: Freeze priority 1
>2014/10/30 05:58:21.997938 [ 7506]: Freeze priority 1
>2014/10/30 05:58:22.007652 [ 7506]: Freeze priority 2
>2014/10/30 05:58:22.008899 [ 7506]: Freeze priority 3
>2014/10/30 05:58:22.009747 [ 7506]: Freeze priority 2
>2014/10/30 05:58:22.010155 [ 7506]: Freeze priority 2
>2014/10/30 05:58:22.012996 [ 7506]: Freeze priority 3
>2014/10/30 05:58:22.013409 [ 7506]: Freeze priority 3
>2014/10/30 05:58:25.038498 [ 7506]: Freeze priority 1
>2014/10/30 05:58:25.038829 [ 7506]: Freeze priority 2
>2014/10/30 05:58:25.039155 [ 7506]: Freeze priority 3
>2014/10/30 05:58:28.618570 [ 7506]: Thawing priority 1
>2014/10/30 05:58:28.618687 [ 7506]: Release freeze handler for prio 1
>2014/10/30 05:58:28.618747 [ 7506]: Thawing priority 2
>2014/10/30 05:58:28.618770 [ 7506]: Release freeze handler for prio 2
>2014/10/30 05:58:28.618813 [ 7506]: Thawing priority 3
>2014/10/30 05:58:28.618844 [ 7506]: Release freeze handler for prio 3
>2014/10/30 06:03:44.900153 [ 7506]: Freeze priority 1
>2014/10/30 06:03:44.968881 [ 7506]: Freeze priority 2
>2014/10/30 06:03:44.973331 [ 7506]: Freeze priority 3
>2014/10/30 06:03:49.812350 [ 7506]: Thawing priority 1
>2014/10/30 06:03:49.812417 [ 7506]: Release freeze handler for prio 1
>2014/10/30 06:03:49.812453 [ 7506]: Thawing priority 2
>2014/10/30 06:03:49.812473 [ 7506]: Release freeze handler for prio 2
>2014/10/30 06:03:49.812501 [ 7506]: Thawing priority 3
>2014/10/30 06:03:49.812518 [ 7506]: Release freeze handler for prio 3
>2014/10/30 06:08:29.886867 [ 7506]: Freeze priority 1
>2014/10/30 06:08:29.891091 [ 7506]: Freeze priority 1
>2014/10/30 06:08:29.908299 [ 7506]: Freeze priority 2
>2014/10/30 06:08:29.909390 [ 7506]: Freeze priority 2
>2014/10/30 06:08:29.912770 [ 7506]: Freeze priority 3
>2014/10/30 06:08:29.915378 [ 7506]: Freeze priority 3
>2014/10/30 06:08:33.430903 [recoverd: 7843]: Taking out recovery lock from recovery daemon
>2014/10/30 06:08:33.430978 [recoverd: 7843]: Take the recovery lock
>2014/10/30 06:08:33.442472 [ 7506]: Freeze priority 1
>2014/10/30 06:08:33.442815 [ 7506]: Freeze priority 2
>2014/10/30 06:08:33.443101 [ 7506]: Freeze priority 3
>2014/10/30 06:08:35.251880 [ 7506]: Thawing priority 1
>2014/10/30 06:08:35.251947 [ 7506]: Release freeze handler for prio 1
>2014/10/30 06:08:35.251996 [ 7506]: Thawing priority 2
>2014/10/30 06:08:35.252025 [ 7506]: Release freeze handler for prio 2
>2014/10/30 06:08:35.252054 [ 7506]: Thawing priority 3
>2014/10/30 06:08:35.252070 [ 7506]: Release freeze handler for prio 3
>2014/10/30 06:08:35.859673 [recoverd: 7843]: Resetting ban count to 0 for all nodes
>2014/10/30 06:15:12.431357 [recoverd: 7843]: Taking out recovery lock from recovery daemon
>2014/10/30 06:15:12.431415 [recoverd: 7843]: Take the recovery lock
>2014/10/30 06:15:12.569408 [ 7506]: Freeze priority 1
>2014/10/30 06:15:12.579404 [ 7506]: Freeze priority 2
>2014/10/30 06:15:12.583450 [ 7506]: Freeze priority 3
>2014/10/30 06:15:15.440627 [ 7506]: Thawing priority 1
>2014/10/30 06:15:15.440687 [ 7506]: Release freeze handler for prio 1
>2014/10/30 06:15:15.440727 [ 7506]: Thawing priority 2
>2014/10/30 06:15:15.440748 [ 7506]: Release freeze handler for prio 2
>2014/10/30 06:15:15.440779 [ 7506]: Thawing priority 3
>2014/10/30 06:15:15.440798 [ 7506]: Release freeze handler for prio 3
>2014/10/30 06:15:15.816301 [recoverd: 7843]: Resetting ban count to 0 for all nodes
>2014/10/30 06:15:26.059699 [recoverd: 7843]: server/ctdb_recoverd.c:3933 Remote node:1 has different flags for node 0. It has 0x02 vs our 0x00
>2014/10/30 06:15:26.059755 [recoverd: 7843]: Use flags 0x00 from local recmaster node for cluster update of node 0 flags
>2014/10/30 06:15:26.061228 [recoverd: 7843]: Taking out recovery lock from recovery daemon
>2014/10/30 06:15:26.061262 [recoverd: 7843]: Take the recovery lock
>2014/10/30 06:15:26.120270 [ 7506]: Freeze priority 1
>2014/10/30 06:15:26.123676 [ 7506]: Freeze priority 2
>2014/10/30 06:15:26.126432 [ 7506]: Freeze priority 3
>2014/10/30 06:15:29.033628 [ 7506]: Thawing priority 1
>2014/10/30 06:15:29.033682 [ 7506]: Release freeze handler for prio 1
>2014/10/30 06:15:29.033724 [ 7506]: Thawing priority 2
>2014/10/30 06:15:29.033760 [ 7506]: Release freeze handler for prio 2
>2014/10/30 06:15:29.033790 [ 7506]: Thawing priority 3
>2014/10/30 06:15:29.033809 [ 7506]: Release freeze handler for prio 3
>2014/10/30 06:15:29.431851 [recoverd: 7843]: Resetting ban count to 0 for all nodes
>2014/10/30 06:18:33.436332 [ 7506]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 06:18:33.447338 [ 7506]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 06:20:00.686246 [ 1524]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 06:20:00.686345 [ 1524]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 06:20:00.686359 [ 1524]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 06:24:01.760413 [ 7740]: Starting CTDBD (Version 2.5.3) as PID: 7740
>2014/10/30 06:24:03.209028 [ 7740]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 06:24:03.232597 [ 7740]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 06:24:03.246818 [ 7740]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 06:24:03.261033 [ 7740]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 06:24:03.261051 [ 7740]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 06:24:03.261060 [ 7740]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 06:24:03.261068 [ 7740]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 06:24:03.261077 [ 7740]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 06:24:03.261086 [ 7740]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 06:24:03.275028 [ 7740]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 06:24:03.289051 [ 7740]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 06:24:03.289070 [ 7740]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 06:24:03.289079 [ 7740]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 06:24:03.302991 [ 7740]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 06:24:03.303023 [ 7740]: Freeze priority 1
>2014/10/30 06:24:03.314474 [ 7740]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 06:24:03.321069 [ 7740]: Freeze priority 2
>2014/10/30 06:24:03.321419 [ 7740]: Freeze priority 3
>2014/10/30 06:24:03.485305 [ 7740]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 06:24:03.489111 [ 7740]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 06:24:03.492929 [ 7740]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 06:24:03.612759 [ 7740]: Freeze priority 1
>2014/10/30 06:24:03.612840 [ 7740]: Freeze priority 2
>2014/10/30 06:24:03.612898 [ 7740]: Freeze priority 3
>2014/10/30 06:24:04.146435 [ 7740]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 06:24:08.817452 [ 7740]: Freeze priority 1
>2014/10/30 06:24:08.832580 [ 7740]: Freeze priority 2
>2014/10/30 06:24:08.834854 [ 7740]: Freeze priority 3
>2014/10/30 06:24:09.012037 [ 7740]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 06:24:09.016217 [ 7740]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 06:24:09.019699 [ 7740]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 06:24:11.228067 [ 7740]: Thawing priority 1
>2014/10/30 06:24:11.228098 [ 7740]: Release freeze handler for prio 1
>2014/10/30 06:24:11.228127 [ 7740]: Thawing priority 2
>2014/10/30 06:24:11.228156 [ 7740]: Release freeze handler for prio 2
>2014/10/30 06:24:11.228179 [ 7740]: Thawing priority 3
>2014/10/30 06:24:11.228206 [ 7740]: Release freeze handler for prio 3
>2014/10/30 06:24:26.242720 [recoverd: 8014]: Trigger takeoverrun
>2014/10/30 06:24:26.449108 [ 7740]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 06:24:26.715391 [ 7740]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 06:24:26.728906 [ 7740]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 06:24:26.765692 [ 7740]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 06:24:26.961194 [ 7740]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 06:24:29.417969 [ 7740]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 06:24:30.179131 [ 7740]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 06:28:37.580154 [ 7740]: Freeze priority 1
>2014/10/30 06:28:37.598806 [ 7740]: Freeze priority 1
>2014/10/30 06:28:37.613160 [ 7740]: Freeze priority 1
>2014/10/30 06:28:37.647510 [ 7740]: Freeze priority 2
>2014/10/30 06:28:37.647737 [ 7740]: Freeze priority 2
>2014/10/30 06:28:37.648610 [ 7740]: Freeze priority 2
>2014/10/30 06:28:37.649073 [ 7740]: Freeze priority 3
>2014/10/30 06:28:37.650173 [ 7740]: Freeze priority 3
>2014/10/30 06:28:37.651324 [ 7740]: Freeze priority 3
>2014/10/30 06:28:40.688247 [ 7740]: Freeze priority 1
>2014/10/30 06:28:40.688620 [ 7740]: Freeze priority 2
>2014/10/30 06:28:40.688998 [ 7740]: Freeze priority 3
>2014/10/30 06:28:44.228453 [ 7740]: Thawing priority 1
>2014/10/30 06:28:44.228515 [ 7740]: Release freeze handler for prio 1
>2014/10/30 06:28:44.228552 [ 7740]: Thawing priority 2
>2014/10/30 06:28:44.228585 [ 7740]: Release freeze handler for prio 2
>2014/10/30 06:28:44.228620 [ 7740]: Thawing priority 3
>2014/10/30 06:28:44.228653 [ 7740]: Release freeze handler for prio 3
>2014/10/30 06:33:45.372666 [ 7740]: Freeze priority 1
>2014/10/30 06:33:45.473092 [ 7740]: Freeze priority 2
>2014/10/30 06:33:45.477459 [ 7740]: Freeze priority 3
>2014/10/30 06:33:49.677700 [ 7740]: Thawing priority 1
>2014/10/30 06:33:49.677765 [ 7740]: Release freeze handler for prio 1
>2014/10/30 06:33:49.677801 [ 7740]: Thawing priority 2
>2014/10/30 06:33:49.677822 [ 7740]: Release freeze handler for prio 2
>2014/10/30 06:33:49.677884 [ 7740]: Thawing priority 3
>2014/10/30 06:33:49.677907 [ 7740]: Release freeze handler for prio 3
>2014/10/30 06:34:00.093115 [ 7740]: Freeze priority 1
>2014/10/30 06:34:00.121706 [ 7740]: Freeze priority 2
>2014/10/30 06:34:00.124975 [ 7740]: Freeze priority 3
>2014/10/30 06:34:04.124920 [ 7740]: Thawing priority 1
>2014/10/30 06:34:04.124966 [ 7740]: Release freeze handler for prio 1
>2014/10/30 06:34:04.124998 [ 7740]: Thawing priority 2
>2014/10/30 06:34:04.125018 [ 7740]: Release freeze handler for prio 2
>2014/10/30 06:34:04.125049 [ 7740]: Thawing priority 3
>2014/10/30 06:34:04.125067 [ 7740]: Release freeze handler for prio 3
>2014/10/30 06:43:49.971404 [ 7740]: Freeze priority 1
>2014/10/30 06:43:50.083436 [ 7740]: Freeze priority 2
>2014/10/30 06:43:50.084581 [ 7740]: Freeze priority 3
>2014/10/30 06:43:53.931395 [ 7740]: Freeze priority 1
>2014/10/30 06:43:53.931793 [ 7740]: Freeze priority 2
>2014/10/30 06:43:53.932142 [ 7740]: Freeze priority 3
>2014/10/30 06:43:57.699025 [ 7740]: Thawing priority 1
>2014/10/30 06:43:57.699066 [ 7740]: Release freeze handler for prio 1
>2014/10/30 06:43:57.699099 [ 7740]: Thawing priority 2
>2014/10/30 06:43:57.699120 [ 7740]: Release freeze handler for prio 2
>2014/10/30 06:43:57.699148 [ 7740]: Thawing priority 3
>2014/10/30 06:43:57.699167 [ 7740]: Release freeze handler for prio 3
>2014/10/30 06:49:07.908515 [ 7740]: Freeze priority 1
>2014/10/30 06:49:07.930351 [ 7740]: Freeze priority 2
>2014/10/30 06:49:07.933707 [ 7740]: Freeze priority 3
>2014/10/30 06:49:13.989635 [ 7740]: Thawing priority 1
>2014/10/30 06:49:13.989686 [ 7740]: Release freeze handler for prio 1
>2014/10/30 06:49:13.989736 [ 7740]: Thawing priority 2
>2014/10/30 06:49:13.989752 [ 7740]: Release freeze handler for prio 2
>2014/10/30 06:49:13.989778 [ 7740]: Thawing priority 3
>2014/10/30 06:49:13.989794 [ 7740]: Release freeze handler for prio 3
>2014/10/30 06:49:24.434469 [ 7740]: Freeze priority 1
>2014/10/30 06:49:24.622295 [ 7740]: Freeze priority 2
>2014/10/30 06:49:24.624051 [ 7740]: Freeze priority 3
>2014/10/30 06:49:28.572269 [ 7740]: Thawing priority 1
>2014/10/30 06:49:28.572313 [ 7740]: Release freeze handler for prio 1
>2014/10/30 06:49:28.572341 [ 7740]: Thawing priority 2
>2014/10/30 06:49:28.572369 [ 7740]: Release freeze handler for prio 2
>2014/10/30 06:49:28.572393 [ 7740]: Thawing priority 3
>2014/10/30 06:49:28.572408 [ 7740]: Release freeze handler for prio 3
>2014/10/30 06:53:59.942698 [ 7740]: Freeze priority 1
>2014/10/30 06:53:59.949313 [ 7740]: Freeze priority 1
>2014/10/30 06:53:59.958253 [ 7740]: Freeze priority 1
>2014/10/30 06:53:59.965143 [ 7740]: Freeze priority 2
>2014/10/30 06:53:59.965362 [ 7740]: Freeze priority 2
>2014/10/30 06:53:59.965405 [ 7740]: Freeze priority 2
>2014/10/30 06:53:59.966903 [ 7740]: Freeze priority 3
>2014/10/30 06:53:59.967093 [ 7740]: Freeze priority 3
>2014/10/30 06:53:59.968248 [ 7740]: Freeze priority 3
>2014/10/30 06:54:03.477701 [recoverd: 8014]: Taking out recovery lock from recovery daemon
>2014/10/30 06:54:03.477746 [recoverd: 8014]: Take the recovery lock
>2014/10/30 06:54:03.490690 [ 7740]: Freeze priority 1
>2014/10/30 06:54:03.491048 [ 7740]: Freeze priority 2
>2014/10/30 06:54:03.491368 [ 7740]: Freeze priority 3
>2014/10/30 06:54:05.169504 [ 7740]: Thawing priority 1
>2014/10/30 06:54:05.169559 [ 7740]: Release freeze handler for prio 1
>2014/10/30 06:54:05.169610 [ 7740]: Thawing priority 2
>2014/10/30 06:54:05.169632 [ 7740]: Release freeze handler for prio 2
>2014/10/30 06:54:05.169676 [ 7740]: Thawing priority 3
>2014/10/30 06:54:05.169699 [ 7740]: Release freeze handler for prio 3
>2014/10/30 06:54:05.701963 [recoverd: 8014]: Resetting ban count to 0 for all nodes
>2014/10/30 07:00:49.252909 [recoverd: 8014]: Taking out recovery lock from recovery daemon
>2014/10/30 07:00:49.252971 [recoverd: 8014]: Take the recovery lock
>2014/10/30 07:00:49.310761 [ 7740]: Freeze priority 1
>2014/10/30 07:00:49.327202 [ 7740]: Freeze priority 2
>2014/10/30 07:00:49.330718 [ 7740]: Freeze priority 3
>2014/10/30 07:00:51.833262 [ 7740]: Thawing priority 1
>2014/10/30 07:00:51.833314 [ 7740]: Release freeze handler for prio 1
>2014/10/30 07:00:51.833349 [ 7740]: Thawing priority 2
>2014/10/30 07:00:51.833370 [ 7740]: Release freeze handler for prio 2
>2014/10/30 07:00:51.833401 [ 7740]: Thawing priority 3
>2014/10/30 07:00:51.833420 [ 7740]: Release freeze handler for prio 3
>2014/10/30 07:00:52.205210 [recoverd: 8014]: Resetting ban count to 0 for all nodes
>2014/10/30 07:04:02.298016 [ 7740]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 07:04:02.308978 [ 7740]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 07:05:26.786064 [ 1520]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 07:05:26.786168 [ 1520]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 07:05:26.786186 [ 1520]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 07:09:29.236277 [ 7610]: Starting CTDBD (Version 2.5.3) as PID: 7610
>2014/10/30 07:09:30.841650 [ 7610]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 07:09:30.865220 [ 7610]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 07:09:30.879845 [ 7610]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 07:09:30.893995 [ 7610]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 07:09:30.894015 [ 7610]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 07:09:30.894025 [ 7610]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 07:09:30.894034 [ 7610]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 07:09:30.894043 [ 7610]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 07:09:30.894052 [ 7610]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 07:09:30.908049 [ 7610]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 07:09:30.921990 [ 7610]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 07:09:30.922011 [ 7610]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 07:09:30.922021 [ 7610]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 07:09:30.936065 [ 7610]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 07:09:30.936115 [ 7610]: Freeze priority 1
>2014/10/30 07:09:30.949730 [ 7610]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 07:09:30.953431 [ 7610]: Freeze priority 2
>2014/10/30 07:09:30.953839 [ 7610]: Freeze priority 3
>2014/10/30 07:09:31.117913 [ 7610]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 07:09:31.121654 [ 7610]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 07:09:31.125068 [ 7610]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 07:09:31.243666 [ 7610]: Freeze priority 1
>2014/10/30 07:09:31.243741 [ 7610]: Freeze priority 2
>2014/10/30 07:09:31.243818 [ 7610]: Freeze priority 3
>2014/10/30 07:09:31.378081 [ 7610]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 07:09:31.378142 [ 7610]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 07:09:31.773661 [ 7610]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 07:09:35.218080 [ 7610]: Freeze priority 1
>2014/10/30 07:09:35.251389 [recoverd: 7904]: server/ctdb_recoverd.c:3692 Current recmaster node 1 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 07:09:35.251462 [ 7610]: Freeze priority 1
>2014/10/30 07:09:35.251518 [ 7610]: Freeze priority 2
>2014/10/30 07:09:35.251569 [ 7610]: Freeze priority 3
>2014/10/30 07:09:35.287411 [ 7610]: Freeze priority 2
>2014/10/30 07:09:35.294649 [ 7610]: Freeze priority 3
>2014/10/30 07:09:42.004406 [ 7610]: Freeze priority 1
>2014/10/30 07:09:42.007031 [ 7610]: Thawing priority 1
>2014/10/30 07:09:42.007065 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:09:42.007106 [ 7610]: Thawing priority 2
>2014/10/30 07:09:42.007125 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:09:42.007153 [ 7610]: Thawing priority 3
>2014/10/30 07:09:42.007170 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:09:42.009966 [ 7610]: Freeze priority 2
>2014/10/30 07:09:42.011084 [ 7610]: Freeze priority 3
>2014/10/30 07:09:42.014010 [set_recmode: 9303]: ERROR: recovery lock file /mnt/lock/lockfile not locked when recovering!
>2014/10/30 07:09:45.030081 [ 7610]: Freeze priority 1
>2014/10/30 07:09:45.031157 [ 7610]: Freeze priority 2
>2014/10/30 07:09:45.031481 [ 7610]: Freeze priority 3
>2014/10/30 07:09:45.190508 [ 7610]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 07:09:45.191365 [ 7610]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 07:09:45.192560 [ 7610]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 07:09:47.202078 [ 7610]: Thawing priority 1
>2014/10/30 07:09:47.202133 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:09:47.202168 [ 7610]: Thawing priority 2
>2014/10/30 07:09:47.202189 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:09:47.202232 [ 7610]: Thawing priority 3
>2014/10/30 07:09:47.202251 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:10:01.227523 [recoverd: 7904]: Trigger takeoverrun
>2014/10/30 07:10:01.683618 [ 7610]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 07:10:02.248791 [ 7610]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 07:10:02.262318 [ 7610]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 07:10:02.290866 [ 7610]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 07:10:02.477620 [ 7610]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 07:10:04.943567 [ 7610]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 07:10:06.094236 [ 7610]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 07:14:06.609719 [ 7610]: Freeze priority 1
>2014/10/30 07:14:06.609934 [ 7610]: Freeze priority 1
>2014/10/30 07:14:06.610059 [ 7610]: Freeze priority 1
>2014/10/30 07:14:06.779425 [ 7610]: Freeze priority 2
>2014/10/30 07:14:06.779641 [ 7610]: Freeze priority 2
>2014/10/30 07:14:06.779689 [ 7610]: Freeze priority 2
>2014/10/30 07:14:06.780507 [ 7610]: Freeze priority 3
>2014/10/30 07:14:06.780661 [ 7610]: Freeze priority 3
>2014/10/30 07:14:06.780798 [ 7610]: Freeze priority 3
>2014/10/30 07:14:09.802458 [ 7610]: Freeze priority 1
>2014/10/30 07:14:09.802840 [ 7610]: Freeze priority 2
>2014/10/30 07:14:09.803214 [ 7610]: Freeze priority 3
>2014/10/30 07:14:14.742248 [ 7610]: Thawing priority 1
>2014/10/30 07:14:14.742287 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:14:14.742321 [ 7610]: Thawing priority 2
>2014/10/30 07:14:14.742343 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:14:14.742375 [ 7610]: Thawing priority 3
>2014/10/30 07:14:14.742396 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:19:13.733797 [ 7610]: Freeze priority 1
>2014/10/30 07:19:13.846895 [ 7610]: Freeze priority 2
>2014/10/30 07:19:13.848199 [ 7610]: Freeze priority 3
>2014/10/30 07:19:16.638831 [ 7610]: Thawing priority 1
>2014/10/30 07:19:16.638897 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:19:16.638947 [ 7610]: Thawing priority 2
>2014/10/30 07:19:16.638966 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:19:16.638994 [ 7610]: Thawing priority 3
>2014/10/30 07:19:16.639021 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:19:27.107699 [ 7610]: Freeze priority 1
>2014/10/30 07:19:27.122193 [ 7610]: Freeze priority 2
>2014/10/30 07:19:27.124853 [ 7610]: Freeze priority 3
>2014/10/30 07:19:29.752674 [ 7610]: Thawing priority 1
>2014/10/30 07:19:29.752727 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:19:29.752778 [ 7610]: Thawing priority 2
>2014/10/30 07:19:29.752800 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:19:29.752839 [ 7610]: Thawing priority 3
>2014/10/30 07:19:29.752854 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:29:16.527007 [ 7610]: Freeze priority 1
>2014/10/30 07:29:16.556096 [ 7610]: Freeze priority 1
>2014/10/30 07:29:16.600786 [ 7610]: Freeze priority 1
>2014/10/30 07:29:27.062102 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:29:42.062395 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:29:57.062537 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:30:12.062961 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:30:16.530216 [ 7610]: Freeze priority 1
>2014/10/30 07:30:16.540339 [ 7610]: Recovery daemon ping timeout. Count : 0
>2014/10/30 07:30:16.556515 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 07:30:16.556561 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 07:30:16.556587 [recoverd: 7904]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/30 07:30:16.556603 [recoverd: 7904]: Failed to freeze node 1 during recovery. Set it as ban culprit for 4 credits
>2014/10/30 07:30:16.556623 [recoverd: 7904]: Async wait failed - fail_count=1
>2014/10/30 07:30:16.556638 [recoverd: 7904]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/30 07:30:16.556655 [recoverd: 7904]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/30 07:30:16.558255 [ 7610]: Freeze priority 1
>2014/10/30 07:30:16.602992 [ 7610]: Freeze priority 1
>2014/10/30 07:30:27.063305 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=8307 =====
>5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W
>5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=5505 -----
>#0  0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f90287ac744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=8307 =====
>===== Start of debug locks PID=8859 =====
>5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W
>5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=5505 -----
>#0  0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f90287ac744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=8859 =====
>2014/10/30 07:30:42.063778 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=9201 =====
>5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W
>5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=5505 -----
>#0  0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f90287ac744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=9201 =====
>2014/10/30 07:30:57.064304 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=9672 =====
>5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W
>5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=5505 -----
>#0  0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f90287ac744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=9672 =====
>===== Start of debug locks PID=10101 =====
>5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>10259 /usr/sbin/smbd printer_list.tdb.2 2984 2984 W
>5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W
>5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=5505 -----
>#0  0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f90287ac744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=10101 =====
>2014/10/30 07:31:12.064709 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:31:16.557795 [ 7610]: Recovery daemon ping timeout. Count : 0
>2014/10/30 07:31:16.559004 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 07:31:16.559053 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 07:31:16.559077 [recoverd: 7904]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/30 07:31:16.559092 [recoverd: 7904]: Failed to freeze node 1 during recovery. Set it as ban culprit for 4 credits
>2014/10/30 07:31:16.559112 [recoverd: 7904]: Async wait failed - fail_count=1
>2014/10/30 07:31:16.559128 [recoverd: 7904]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/30 07:31:16.559144 [recoverd: 7904]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster
>2014/10/30 07:31:16.773102 [ 7610]: Freeze priority 1
>2014/10/30 07:31:16.773203 [ 7610]: Freeze priority 1
>2014/10/30 07:31:16.823226 [ 7610]: pnn 2 Invalid reqid 45672 in ctdb_reply_control
>2014/10/30 07:31:16.823262 [ 7610]: pnn 2 Invalid reqid 45455 in ctdb_reply_control
>2014/10/30 07:31:16.823335 [ 7610]: Freeze priority 2
>2014/10/30 07:31:16.823479 [ 7610]: Freeze priority 2
>2014/10/30 07:31:16.824328 [ 7610]: Freeze priority 3
>2014/10/30 07:31:16.824473 [ 7610]: Freeze priority 3
>2014/10/30 07:31:19.832708 [recoverd: 7904]: Taking out recovery lock from recovery daemon
>2014/10/30 07:31:19.832749 [recoverd: 7904]: Take the recovery lock
>2014/10/30 07:31:19.845824 [ 7610]: Freeze priority 1
>2014/10/30 07:31:19.846127 [ 7610]: Freeze priority 2
>2014/10/30 07:31:19.846445 [ 7610]: Freeze priority 3
>2014/10/30 07:31:21.530602 [ 7610]: Thawing priority 1
>2014/10/30 07:31:21.530655 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:31:21.530698 [ 7610]: Thawing priority 2
>2014/10/30 07:31:21.530720 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:31:21.530761 [ 7610]: Thawing priority 3
>2014/10/30 07:31:21.530786 [ 7610]: Release freeze handler for prio 3
>===== Start of debug locks PID=10541 =====
>===== End of debug locks PID=10541 =====
>2014/10/30 07:31:26.879924 [ 7610]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 07:31:27.081816 [recoverd: 7904]: Resetting ban count to 0 for all nodes
>2014/10/30 07:31:47.105533 [recoverd: 7904]: Taking out recovery lock from recovery daemon
>2014/10/30 07:31:47.105571 [recoverd: 7904]: Take the recovery lock
>2014/10/30 07:31:47.117516 [ 7610]: Freeze priority 1
>2014/10/30 07:31:47.118872 [ 7610]: Freeze priority 2
>2014/10/30 07:31:47.119920 [ 7610]: Freeze priority 3
>2014/10/30 07:31:50.735331 [ 7610]: Thawing priority 1
>2014/10/30 07:31:50.735373 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:31:50.735405 [ 7610]: Thawing priority 2
>2014/10/30 07:31:50.735426 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:31:50.735457 [ 7610]: Thawing priority 3
>2014/10/30 07:31:50.735476 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:31:56.373950 [recoverd: 7904]: Resetting ban count to 0 for all nodes
>2014/10/30 07:34:33.032954 [ 7610]: Freeze priority 1
>2014/10/30 07:34:33.034431 [ 7610]: Freeze priority 2
>2014/10/30 07:34:33.035341 [ 7610]: Freeze priority 3
>2014/10/30 07:34:37.416416 [recoverd: 7904]: Taking out recovery lock from recovery daemon
>2014/10/30 07:34:37.416541 [recoverd: 7904]: Take the recovery lock
>2014/10/30 07:34:37.416673 [ 7610]: Thawing priority 1
>2014/10/30 07:34:37.416717 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:34:37.416806 [ 7610]: Thawing priority 2
>2014/10/30 07:34:37.416833 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:34:37.416869 [ 7610]: Thawing priority 3
>2014/10/30 07:34:37.416888 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:34:37.429293 [ 7610]: Freeze priority 1
>2014/10/30 07:34:37.431502 [ 7610]: Freeze priority 2
>2014/10/30 07:34:37.433209 [ 7610]: Freeze priority 3
>2014/10/30 07:34:37.609490 [ 7610]: Refusing to run event scripts call 'recovered' while in recovery
>2014/10/30 07:34:37.609527 [ 7610]: server/ctdb_recover.c:952 Failed to end recovery
>2014/10/30 07:34:40.526609 [ 7610]: Thawing priority 1
>2014/10/30 07:34:40.526652 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:34:40.526681 [ 7610]: Thawing priority 2
>2014/10/30 07:34:40.526700 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:34:40.526726 [ 7610]: Thawing priority 3
>2014/10/30 07:34:40.526743 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:34:40.528143 [ 7610]: Freeze priority 1
>2014/10/30 07:34:40.532745 [ 7610]: Freeze priority 2
>2014/10/30 07:34:40.534611 [ 7610]: Freeze priority 3
>2014/10/30 07:34:40.786081 [recoverd: 7904]: Async operation failed with ret=0 res=-1 opcode=71
>2014/10/30 07:34:40.786131 [recoverd: 7904]: server/ctdb_recoverd.c:239 Node 1 failed the recovered event. Setting it as recovery fail culprit
>2014/10/30 07:34:40.927466 [recoverd: 7904]: Async wait failed - fail_count=1
>2014/10/30 07:34:40.927520 [recoverd: 7904]: server/ctdb_recoverd.c:262 Unable to run the 'recovered' event when called from do_recovery
>2014/10/30 07:34:40.927533 [recoverd: 7904]: server/ctdb_recoverd.c:2016 Unable to run the 'recovered' event on cluster. Recovery process failed.
>2014/10/30 07:34:43.934177 [recoverd: 7904]: Taking out recovery lock from recovery daemon
>2014/10/30 07:34:43.934229 [recoverd: 7904]: Take the recovery lock
>2014/10/30 07:34:43.945996 [ 7610]: Freeze priority 1
>2014/10/30 07:34:46.308336 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=22007 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=22007 =====
>2014/10/30 07:34:51.309222 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:34:56.309861 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=22378 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=22378 =====
>2014/10/30 07:35:01.310906 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:35:06.311314 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=22878 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=22878 =====
>2014/10/30 07:35:11.312360 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:35:16.312911 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=23263 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=23263 =====
>2014/10/30 07:35:21.313408 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:35:26.313560 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=23618 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=23618 =====
>2014/10/30 07:35:31.314102 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:35:36.314651 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=24143 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=24143 =====
>2014/10/30 07:35:41.315706 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:35:43.932392 [ 7610]: Recovery daemon ping timeout. Count : 0
>2014/10/30 07:35:43.946543 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 07:35:43.946604 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out'
>2014/10/30 07:35:43.946640 [recoverd: 7904]: Async operation failed with ret=-1 res=-1 opcode=33
>2014/10/30 07:35:43.946667 [recoverd: 7904]: Failed to freeze node 3 during recovery. Set it as ban culprit for 4 credits
>2014/10/30 07:35:43.946688 [recoverd: 7904]: Async wait failed - fail_count=1
>2014/10/30 07:35:43.946700 [recoverd: 7904]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed.
>2014/10/30 07:35:43.946713 [recoverd: 7904]: server/ctdb_recoverd.c:1833 Unable to set recovery mode to active on cluster
>2014/10/30 07:35:43.958196 [recoverd: 7904]: Taking out recovery lock from recovery daemon
>2014/10/30 07:35:43.958235 [recoverd: 7904]: Take the recovery lock
>2014/10/30 07:35:43.994158 [ 7610]: Freeze priority 1
>2014/10/30 07:35:46.315985 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=24780 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=24780 =====
>2014/10/30 07:35:51.316892 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:35:56.318008 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=25143 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=25143 =====
>2014/10/30 07:36:01.318718 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:36:06.319414 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=25650 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=25650 =====
>2014/10/30 07:36:11.320130 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:36:16.320764 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=26021 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=26021 =====
>===== Start of debug locks PID=26142 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=26142 =====
>2014/10/30 07:36:21.321265 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:36:26.322244 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=26503 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=26503 =====
>===== Start of debug locks PID=26626 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=26626 =====
>2014/10/30 07:36:31.322929 [ 7610]: Skip monitoring since databases are frozen
>2014/10/30 07:36:36.324011 [ 7610]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=27048 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=27048 =====
>2014/10/30 07:36:40.538616 [ 7610]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS
>===== Start of debug locks PID=27242 =====
>21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF
>21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W
>25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W
>21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF
>21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF
>----- Stack trace for PID=21453 -----
>#0  0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6
>#1  0x00007f008d132744 in sleep () from /lib64/libc.so.6
>#2  0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145
>===== End of debug locks PID=27242 =====
>2014/10/30 07:36:40.704508 [ 7610]: pnn 2 Invalid reqid 53706 in ctdb_reply_control
>2014/10/30 07:36:40.704817 [ 7610]: Freeze priority 2
>2014/10/30 07:36:40.705312 [ 7610]: Freeze priority 3
>2014/10/30 07:36:43.313289 [ 7610]: Thawing priority 1
>2014/10/30 07:36:43.313326 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:36:43.313363 [ 7610]: Thawing priority 2
>2014/10/30 07:36:43.313384 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:36:43.313414 [ 7610]: Thawing priority 3
>2014/10/30 07:36:43.313433 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:36:43.617253 [ 7610]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 07:36:43.799970 [recoverd: 7904]: Resetting ban count to 0 for all nodes
>2014/10/30 07:36:43.947685 [ 7610]: Recovery daemon ping timeout. Count : 0
>2014/10/30 07:39:25.026633 [recoverd: 7904]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 4 vs 3
>2014/10/30 07:39:25.067517 [recoverd: 7904]: Taking out recovery lock from recovery daemon
>2014/10/30 07:39:25.067551 [recoverd: 7904]: Take the recovery lock
>2014/10/30 07:39:25.233241 [ 7610]: Freeze priority 1
>2014/10/30 07:39:25.334714 [ 7610]: Freeze priority 2
>2014/10/30 07:39:25.336778 [ 7610]: Freeze priority 3
>2014/10/30 07:39:25.339993 [ 7610]: Monitoring event was cancelled
>2014/10/30 07:39:25.340032 [ 7610]: server/eventscript.c:569 Sending SIGTERM to child pid:3866
>2014/10/30 07:39:28.097089 [ 7610]: Thawing priority 1
>2014/10/30 07:39:28.097132 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:39:28.097164 [ 7610]: Thawing priority 2
>2014/10/30 07:39:28.097184 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:39:28.097213 [ 7610]: Thawing priority 3
>2014/10/30 07:39:28.097231 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:39:28.835411 [recoverd: 7904]: Resetting ban count to 0 for all nodes
>2014/10/30 07:46:38.435309 [recoverd: 7904]: Taking out recovery lock from recovery daemon
>2014/10/30 07:46:38.435363 [recoverd: 7904]: Take the recovery lock
>2014/10/30 07:46:38.486660 [ 7610]: Freeze priority 1
>2014/10/30 07:46:38.507490 [ 7610]: Freeze priority 2
>2014/10/30 07:46:38.508776 [ 7610]: Freeze priority 3
>2014/10/30 07:46:41.464641 [ 7610]: Thawing priority 1
>2014/10/30 07:46:41.464679 [ 7610]: Release freeze handler for prio 1
>2014/10/30 07:46:41.464706 [ 7610]: Thawing priority 2
>2014/10/30 07:46:41.464725 [ 7610]: Release freeze handler for prio 2
>2014/10/30 07:46:41.464762 [ 7610]: Thawing priority 3
>2014/10/30 07:46:41.464782 [ 7610]: Release freeze handler for prio 3
>2014/10/30 07:46:41.835633 [recoverd: 7904]: Resetting ban count to 0 for all nodes
>2014/10/30 07:49:34.964116 [ 7610]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 07:49:35.021317 [ 7610]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/30 07:49:35.021351 [ 7610]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 07:51:00.837394 [ 1487]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 07:51:00.837489 [ 1487]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 07:51:00.837503 [ 1487]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 07:55:00.654581 [ 7845]: Starting CTDBD (Version 2.5.3) as PID: 7845
>2014/10/30 07:55:02.528277 [ 7845]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 07:55:02.552227 [ 7845]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 07:55:02.567580 [ 7845]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 07:55:02.582547 [ 7845]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 07:55:02.582573 [ 7845]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 07:55:02.582582 [ 7845]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 07:55:02.582590 [ 7845]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 07:55:02.582599 [ 7845]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 07:55:02.582607 [ 7845]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 07:55:02.597961 [ 7845]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 07:55:02.613320 [ 7845]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 07:55:02.613347 [ 7845]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 07:55:02.613357 [ 7845]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 07:55:02.628585 [ 7845]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 07:55:02.631128 [ 7845]: Freeze priority 1
>2014/10/30 07:55:02.665985 [ 7845]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 07:55:02.681036 [ 7845]: Freeze priority 2
>2014/10/30 07:55:02.681416 [ 7845]: Freeze priority 3
>2014/10/30 07:55:02.862259 [ 7845]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 07:55:02.871303 [ 7845]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 07:55:02.874920 [ 7845]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 07:55:02.965682 [ 7845]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 07:55:03.005942 [ 7845]: Freeze priority 1
>2014/10/30 07:55:03.006024 [ 7845]: Freeze priority 2
>2014/10/30 07:55:03.006081 [ 7845]: Freeze priority 3
>2014/10/30 07:55:03.217031 [ 7845]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 07:55:07.015239 [recoverd: 8565]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 07:55:07.015319 [ 7845]: Freeze priority 1
>2014/10/30 07:55:07.015378 [ 7845]: Freeze priority 2
>2014/10/30 07:55:07.015429 [ 7845]: Freeze priority 3
>2014/10/30 07:55:11.513694 [ 7845]: Freeze priority 1
>2014/10/30 07:55:11.522696 [ 7845]: Freeze priority 2
>2014/10/30 07:55:11.523607 [ 7845]: Freeze priority 3
>2014/10/30 07:55:11.693095 [ 7845]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 07:55:11.693624 [ 7845]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 07:55:11.694440 [ 7845]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 07:55:14.835355 [ 7845]: Thawing priority 1
>2014/10/30 07:55:14.835406 [ 7845]: Release freeze handler for prio 1
>2014/10/30 07:55:14.835440 [ 7845]: Thawing priority 2
>2014/10/30 07:55:14.835461 [ 7845]: Release freeze handler for prio 2
>2014/10/30 07:55:14.835490 [ 7845]: Thawing priority 3
>2014/10/30 07:55:14.835508 [ 7845]: Release freeze handler for prio 3
>2014/10/30 07:55:28.860157 [recoverd: 8565]: Trigger takeoverrun
>2014/10/30 07:55:29.422806 [ 7845]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 07:55:29.746964 [ 7845]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 07:55:29.760258 [ 7845]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 07:55:29.788972 [ 7845]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 07:55:29.967240 [ 7845]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 07:55:32.433374 [ 7845]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 07:55:33.851886 [ 7845]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 07:59:38.204235 [ 7845]: Freeze priority 1
>2014/10/30 07:59:38.218143 [ 7845]: Freeze priority 1
>2014/10/30 07:59:38.220797 [ 7845]: Freeze priority 1
>2014/10/30 07:59:38.221414 [ 7845]: Freeze priority 2
>2014/10/30 07:59:38.221804 [ 7845]: Freeze priority 2
>2014/10/30 07:59:38.221891 [ 7845]: Freeze priority 2
>2014/10/30 07:59:38.223066 [ 7845]: Freeze priority 3
>2014/10/30 07:59:38.223372 [ 7845]: Freeze priority 3
>2014/10/30 07:59:38.223988 [ 7845]: Freeze priority 3
>2014/10/30 07:59:41.266544 [ 7845]: Freeze priority 1
>2014/10/30 07:59:41.267703 [ 7845]: Freeze priority 2
>2014/10/30 07:59:41.268624 [ 7845]: Freeze priority 3
>2014/10/30 07:59:45.249536 [ 7845]: Thawing priority 1
>2014/10/30 07:59:45.249588 [ 7845]: Release freeze handler for prio 1
>2014/10/30 07:59:45.249624 [ 7845]: Thawing priority 2
>2014/10/30 07:59:45.249645 [ 7845]: Release freeze handler for prio 2
>2014/10/30 07:59:45.249676 [ 7845]: Thawing priority 3
>2014/10/30 07:59:45.249694 [ 7845]: Release freeze handler for prio 3
>2014/10/30 08:04:45.491472 [ 7845]: Freeze priority 1
>2014/10/30 08:04:45.512095 [ 7845]: Freeze priority 2
>2014/10/30 08:04:45.516010 [ 7845]: Freeze priority 3
>2014/10/30 08:04:48.839050 [ 7845]: Thawing priority 1
>2014/10/30 08:04:48.839108 [ 7845]: Release freeze handler for prio 1
>2014/10/30 08:04:48.839159 [ 7845]: Thawing priority 2
>2014/10/30 08:04:48.839182 [ 7845]: Release freeze handler for prio 2
>2014/10/30 08:04:48.839225 [ 7845]: Thawing priority 3
>2014/10/30 08:04:48.839246 [ 7845]: Release freeze handler for prio 3
>2014/10/30 08:14:43.633345 [ 7845]: Freeze priority 1
>2014/10/30 08:14:43.642585 [ 7845]: Freeze priority 2
>2014/10/30 08:14:43.644376 [ 7845]: Freeze priority 3
>2014/10/30 08:14:46.716939 [ 7845]: Freeze priority 1
>2014/10/30 08:14:46.717895 [ 7845]: Freeze priority 2
>2014/10/30 08:14:46.718923 [ 7845]: Freeze priority 3
>2014/10/30 08:14:48.791472 [ 7845]: Thawing priority 1
>2014/10/30 08:14:48.791517 [ 7845]: Release freeze handler for prio 1
>2014/10/30 08:14:48.791544 [ 7845]: Thawing priority 2
>2014/10/30 08:14:48.791560 [ 7845]: Release freeze handler for prio 2
>2014/10/30 08:14:48.791585 [ 7845]: Thawing priority 3
>2014/10/30 08:14:48.791600 [ 7845]: Release freeze handler for prio 3
>2014/10/30 08:20:01.439712 [ 7845]: Freeze priority 1
>2014/10/30 08:20:01.582705 [ 7845]: Freeze priority 2
>2014/10/30 08:20:01.587253 [ 7845]: Freeze priority 3
>2014/10/30 08:20:04.899225 [ 7845]: Thawing priority 1
>2014/10/30 08:20:04.899272 [ 7845]: Release freeze handler for prio 1
>2014/10/30 08:20:04.899306 [ 7845]: Thawing priority 2
>2014/10/30 08:20:04.899324 [ 7845]: Release freeze handler for prio 2
>2014/10/30 08:20:04.899365 [ 7845]: Thawing priority 3
>2014/10/30 08:20:04.899381 [ 7845]: Release freeze handler for prio 3
>2014/10/30 08:24:46.847320 [ 7845]: Freeze priority 1
>2014/10/30 08:24:49.269653 [ 7845]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=4673 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=4673 =====
>2014/10/30 08:25:04.270366 [ 7845]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=5541 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=5541 =====
>===== Start of debug locks PID=6277 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=6277 =====
>2014/10/30 08:25:19.271062 [ 7845]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=6646 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=6646 =====
>2014/10/30 08:25:34.271397 [ 7845]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=7132 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=7132 =====
>2014/10/30 08:25:46.865237 [ 7845]: Freeze priority 1
>===== Start of debug locks PID=7589 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=7589 =====
>2014/10/30 08:25:49.271899 [ 7845]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=7955 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=7955 =====
>2014/10/30 08:26:04.272070 [ 7845]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=8429 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=8429 =====
>===== Start of debug locks PID=9001 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=9001 =====
>2014/10/30 08:26:19.272136 [ 7845]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=9405 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=9405 =====
>2014/10/30 08:26:34.273157 [ 7845]: Skip monitoring since databases are frozen
>===== Start of debug locks PID=9848 =====
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>#1  0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0
>#2  0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0
>#3  0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0
>#4  0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0
>#5  0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0
>#6  0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so
>#7  0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so
>#8  0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0
>#9  0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so
>#10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so
>#11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2
>#13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2
>#14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so
>#15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so
>#16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so
>#17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so
>#18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so
>#19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so
>#20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so
>#21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so
>#22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so
>#24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0
>#25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0
>#26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so
>#29 0x00007f9d18ee91b4 in smbd_accept_connection ()
>#30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0
>#31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0
>#32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0
>#33 0x00007f9d18ee5d01 in main ()
>===== End of debug locks PID=9848 =====
>2014/10/30 08:26:46.866203 [ 7845]: Banning this node for 30 seconds
>2014/10/30 08:26:46.866272 [ 7845]: Freeze priority 1
>2014/10/30 08:26:46.866286 [ 7845]: Freeze priority 2
>2014/10/30 08:26:46.866420 [ 7845]: Freeze priority 3
>===== Start of debug locks PID=10292 =====
>10282 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF
>10282 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF
>5800 /usr/sbin/smbd locking.tdb.2 353844 353844
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W
>10281 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF
>10281 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF
>10281 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843
>5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W
>4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF
>4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF
>----- Stack trace for PID=4206 -----
>2014/10/30 08:26:47.008997 [ 7845]: 10.interface: Killing TCP connection 10.10.10.208:63875 10.10.10.183:445
>2014/10/30 08:26:47.009171 [ 7845]: 10.interface: Killing TCP connection 10.10.10.206:49739 10.10.10.183:445
>2014/10/30 08:26:47.009332 [ 7845]: 10.interface: Killing TCP connection 10.10.10.205:54766 10.10.10.183:445
>#0  0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6
>#1  0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47
>#2  0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156
>#3  0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527
>#4  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537
>#5  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537
>#6  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541
>#7  0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537
>#8  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541
>#9  0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541
>#10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537
>#11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541
>#12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537
>#13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537
>#14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541
>#15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537
>#16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537
>#17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537
>#18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541
>#19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541
>#20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541
>#21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570
>#22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650
>#23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86
>#24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129
>----- Stack trace for PID=5800 -----
>2014/10/30 08:26:47.033499 [ 7845]: 10.interface: Killed 3 TCP connections to released IP 10.10.10.183
>2014/10/30 08:26:47.156700 [ 7845]: Freeze priority 1
>#0  0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6
>===== End of debug locks PID=10292 =====
>2014/10/30 08:26:49.139034 [ 7845]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 08:26:51.976804 [ 7845]: DB Attach to database ctdb.tdb refused since node is inactive (flags=0x8)
>2014/10/30 08:27:16.866346 [ 7845]: Banning timedout
>2014/10/30 08:27:17.596298 [ 7845]: Freeze priority 1
>2014/10/30 08:27:17.602515 [ 7845]: Freeze priority 2
>2014/10/30 08:27:17.606564 [ 7845]: Freeze priority 3
>2014/10/30 08:27:20.159126 [ 7845]: Thawing priority 1
>2014/10/30 08:27:20.159183 [ 7845]: Release freeze handler for prio 1
>2014/10/30 08:27:20.159231 [ 7845]: Thawing priority 2
>2014/10/30 08:27:20.159248 [ 7845]: Release freeze handler for prio 2
>2014/10/30 08:27:20.159272 [ 7845]: Thawing priority 3
>2014/10/30 08:27:20.159285 [ 7845]: Release freeze handler for prio 3
>2014/10/30 08:27:20.162423 [ 7845]: pnn 2 Invalid reqid 60681 in ctdb_become_dmaster from node 3
>2014/10/30 08:27:20.162505 [ 7845]: server/ctdb_call.c:1005 reqid 60682 not found
>2014/10/30 08:27:20.693942 [ 7845]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 08:31:22.559339 [vacuum-smbXsrv_open_global.tdb:23765]: Error storing record copies on node 1: ret[0] res[-1]
>2014/10/30 08:31:22.769552 [vacuum-locking.tdb:23774]: Error storing record copies on node 1: ret[0] res[-1]
>2014/10/30 08:31:26.360084 [ 7845]: Freeze priority 1
>2014/10/30 08:31:26.368578 [ 7845]: Freeze priority 2
>2014/10/30 08:31:26.371792 [ 7845]: Freeze priority 3
>2014/10/30 08:31:28.723884 [ 7845]: Thawing priority 1
>2014/10/30 08:31:28.723944 [ 7845]: Release freeze handler for prio 1
>2014/10/30 08:31:28.723983 [ 7845]: Thawing priority 2
>2014/10/30 08:31:28.724004 [ 7845]: Release freeze handler for prio 2
>2014/10/30 08:31:28.724035 [ 7845]: Thawing priority 3
>2014/10/30 08:31:28.724052 [ 7845]: Release freeze handler for prio 3
>2014/10/30 08:34:57.804563 [ 7845]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 08:34:57.849022 [ 7845]: common/ctdb_fork.c:131 waitpid() returned error. errno:10
>2014/10/30 08:34:57.849096 [ 7845]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 08:36:25.589060 [ 1526]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 08:36:25.589166 [ 1526]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 08:36:25.589180 [ 1526]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 08:40:26.596007 [ 7759]: Starting CTDBD (Version 2.5.3) as PID: 7759
>2014/10/30 08:40:28.044766 [ 7759]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 08:40:28.068204 [ 7759]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 08:40:28.082637 [ 7759]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 08:40:28.096741 [ 7759]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 08:40:28.096762 [ 7759]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 08:40:28.096771 [ 7759]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 08:40:28.096780 [ 7759]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 08:40:28.096789 [ 7759]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 08:40:28.096798 [ 7759]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 08:40:28.110786 [ 7759]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 08:40:28.124709 [ 7759]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 08:40:28.124729 [ 7759]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 08:40:28.124739 [ 7759]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 08:40:28.138720 [ 7759]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 08:40:28.138751 [ 7759]: Freeze priority 1
>2014/10/30 08:40:28.148213 [ 7759]: Freeze priority 2
>2014/10/30 08:40:28.148565 [ 7759]: Freeze priority 3
>2014/10/30 08:40:28.312516 [ 7759]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 08:40:28.316310 [ 7759]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 08:40:28.320031 [ 7759]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 08:40:28.348871 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 08:40:28.349040 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 08:40:28.349066 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 08:40:28.387846 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 08:40:28.387880 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 08:40:28.438846 [ 7759]: Freeze priority 1
>2014/10/30 08:40:28.438916 [ 7759]: Freeze priority 2
>2014/10/30 08:40:28.438971 [ 7759]: Freeze priority 3
>2014/10/30 08:40:28.917467 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 08:40:32.447109 [recoverd: 8031]: server/ctdb_recoverd.c:3692 Current recmaster node 1 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 08:40:32.447491 [ 7759]: Freeze priority 1
>2014/10/30 08:40:32.447619 [ 7759]: Freeze priority 2
>2014/10/30 08:40:32.447687 [ 7759]: Freeze priority 3
>2014/10/30 08:40:36.807698 [ 7759]: Freeze priority 1
>2014/10/30 08:40:36.827438 [ 7759]: Freeze priority 2
>2014/10/30 08:40:36.828699 [ 7759]: Freeze priority 3
>2014/10/30 08:40:37.039072 [ 7759]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 08:40:37.043614 [ 7759]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 08:40:37.049247 [ 7759]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 08:40:38.708169 [ 7759]: Thawing priority 1
>2014/10/30 08:40:38.708241 [ 7759]: Release freeze handler for prio 1
>2014/10/30 08:40:38.708288 [ 7759]: Thawing priority 2
>2014/10/30 08:40:38.708309 [ 7759]: Release freeze handler for prio 2
>2014/10/30 08:40:38.708348 [ 7759]: Thawing priority 3
>2014/10/30 08:40:38.708366 [ 7759]: Release freeze handler for prio 3
>2014/10/30 08:40:53.491994 [recoverd: 8031]: Trigger takeoverrun
>2014/10/30 08:40:53.641782 [ 7759]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 08:40:53.981692 [ 7759]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 08:40:53.994968 [ 7759]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 08:40:54.034498 [ 7759]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 08:40:54.213206 [ 7759]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 08:40:56.627598 [ 7759]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 08:40:57.544915 [ 7759]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 08:45:00.118554 [ 7759]: Freeze priority 1
>2014/10/30 08:45:00.151997 [ 7759]: Freeze priority 1
>2014/10/30 08:45:00.165676 [ 7759]: Freeze priority 2
>2014/10/30 08:45:00.166043 [ 7759]: Freeze priority 2
>2014/10/30 08:45:00.166668 [ 7759]: Freeze priority 3
>2014/10/30 08:45:00.167604 [ 7759]: Freeze priority 3
>2014/10/30 08:45:03.187457 [ 7759]: Freeze priority 1
>2014/10/30 08:45:03.187852 [ 7759]: Freeze priority 2
>2014/10/30 08:45:03.188121 [ 7759]: Freeze priority 3
>2014/10/30 08:45:07.520410 [ 7759]: Thawing priority 1
>2014/10/30 08:45:07.520468 [ 7759]: Release freeze handler for prio 1
>2014/10/30 08:45:07.520501 [ 7759]: Thawing priority 2
>2014/10/30 08:45:07.520538 [ 7759]: Release freeze handler for prio 2
>2014/10/30 08:45:07.520584 [ 7759]: Thawing priority 3
>2014/10/30 08:45:07.520603 [ 7759]: Release freeze handler for prio 3
>2014/10/30 08:50:04.558334 [ 7759]: Freeze priority 1
>2014/10/30 08:50:04.623344 [ 7759]: Freeze priority 2
>2014/10/30 08:50:04.627513 [ 7759]: Freeze priority 3
>2014/10/30 08:50:07.198975 [ 7759]: Thawing priority 1
>2014/10/30 08:50:07.199019 [ 7759]: Release freeze handler for prio 1
>2014/10/30 08:50:07.199053 [ 7759]: Thawing priority 2
>2014/10/30 08:50:07.199071 [ 7759]: Release freeze handler for prio 2
>2014/10/30 08:50:07.199108 [ 7759]: Thawing priority 3
>2014/10/30 08:50:07.199127 [ 7759]: Release freeze handler for prio 3
>2014/10/30 08:50:17.829321 [ 7759]: Freeze priority 1
>2014/10/30 08:50:17.833224 [ 7759]: Freeze priority 2
>2014/10/30 08:50:17.836824 [ 7759]: Freeze priority 3
>2014/10/30 08:50:19.909921 [ 7759]: Thawing priority 1
>2014/10/30 08:50:19.909955 [ 7759]: Release freeze handler for prio 1
>2014/10/30 08:50:19.909984 [ 7759]: Thawing priority 2
>2014/10/30 08:50:19.910002 [ 7759]: Release freeze handler for prio 2
>2014/10/30 08:50:19.910028 [ 7759]: Thawing priority 3
>2014/10/30 08:50:19.910045 [ 7759]: Release freeze handler for prio 3
>2014/10/30 09:00:10.353603 [ 7759]: Freeze priority 1
>2014/10/30 09:00:10.375699 [ 7759]: Freeze priority 2
>2014/10/30 09:00:10.378329 [ 7759]: Freeze priority 3
>2014/10/30 09:00:12.223107 [ 7759]: Thawing priority 1
>2014/10/30 09:00:12.223145 [ 7759]: Release freeze handler for prio 1
>2014/10/30 09:00:12.223174 [ 7759]: Thawing priority 2
>2014/10/30 09:00:12.223191 [ 7759]: Release freeze handler for prio 2
>2014/10/30 09:00:12.223216 [ 7759]: Thawing priority 3
>2014/10/30 09:00:12.223231 [ 7759]: Release freeze handler for prio 3
>2014/10/30 09:05:26.379123 [ 7759]: Freeze priority 1
>2014/10/30 09:05:26.850015 [ 7759]: Freeze priority 2
>2014/10/30 09:05:26.858254 [ 7759]: Freeze priority 3
>2014/10/30 09:05:31.276777 [ 7759]: Thawing priority 1
>2014/10/30 09:05:31.276837 [ 7759]: Release freeze handler for prio 1
>2014/10/30 09:05:31.276892 [ 7759]: Thawing priority 2
>2014/10/30 09:05:31.276913 [ 7759]: Release freeze handler for prio 2
>2014/10/30 09:05:31.276941 [ 7759]: Thawing priority 3
>2014/10/30 09:05:31.276959 [ 7759]: Release freeze handler for prio 3
>2014/10/30 09:05:41.689545 [ 7759]: Freeze priority 1
>2014/10/30 09:05:41.701314 [ 7759]: Freeze priority 2
>2014/10/30 09:05:41.702487 [ 7759]: Freeze priority 3
>2014/10/30 09:05:45.964701 [ 7759]: Thawing priority 1
>2014/10/30 09:05:45.964745 [ 7759]: Release freeze handler for prio 1
>2014/10/30 09:05:45.964781 [ 7759]: Thawing priority 2
>2014/10/30 09:05:45.964802 [ 7759]: Release freeze handler for prio 2
>2014/10/30 09:05:45.964831 [ 7759]: Thawing priority 3
>2014/10/30 09:05:45.964850 [ 7759]: Release freeze handler for prio 3
>2014/10/30 09:06:04.827134 [ 7759]: Monitoring event was cancelled
>2014/10/30 09:10:17.338793 [ 7759]: Freeze priority 1
>2014/10/30 09:10:17.343182 [ 7759]: Freeze priority 1
>2014/10/30 09:10:17.345028 [ 7759]: Freeze priority 1
>2014/10/30 09:10:17.347100 [ 7759]: Freeze priority 2
>2014/10/30 09:10:17.347443 [ 7759]: Freeze priority 2
>2014/10/30 09:10:17.348444 [ 7759]: Freeze priority 2
>2014/10/30 09:10:17.348890 [ 7759]: Freeze priority 3
>2014/10/30 09:10:17.349077 [ 7759]: Freeze priority 3
>2014/10/30 09:10:17.349114 [ 7759]: Freeze priority 3
>2014/10/30 09:10:20.358270 [recoverd: 8031]: Taking out recovery lock from recovery daemon
>2014/10/30 09:10:20.358329 [recoverd: 8031]: Take the recovery lock
>2014/10/30 09:10:20.370813 [ 7759]: Freeze priority 1
>2014/10/30 09:10:20.371134 [ 7759]: Freeze priority 2
>2014/10/30 09:10:20.371427 [ 7759]: Freeze priority 3
>2014/10/30 09:10:22.084892 [ 7759]: Thawing priority 1
>2014/10/30 09:10:22.084935 [ 7759]: Release freeze handler for prio 1
>2014/10/30 09:10:22.084975 [ 7759]: Thawing priority 2
>2014/10/30 09:10:22.084996 [ 7759]: Release freeze handler for prio 2
>2014/10/30 09:10:22.085042 [ 7759]: Thawing priority 3
>2014/10/30 09:10:22.085059 [ 7759]: Release freeze handler for prio 3
>2014/10/30 09:10:22.086537 [ 7759]: server/ctdb_call.c:1005 reqid 50535 not found
>2014/10/30 09:10:22.086601 [ 7759]: server/ctdb_call.c:1005 reqid 50536 not found
>2014/10/30 09:10:22.617950 [recoverd: 8031]: Resetting ban count to 0 for all nodes
>2014/10/30 09:17:21.218849 [recoverd: 8031]: Taking out recovery lock from recovery daemon
>2014/10/30 09:17:21.218902 [recoverd: 8031]: Take the recovery lock
>2014/10/30 09:17:21.315009 [ 7759]: Freeze priority 1
>2014/10/30 09:17:21.327635 [ 7759]: Freeze priority 2
>2014/10/30 09:17:21.329597 [ 7759]: Freeze priority 3
>2014/10/30 09:17:23.213696 [ 7759]: Thawing priority 1
>2014/10/30 09:17:23.213739 [ 7759]: Release freeze handler for prio 1
>2014/10/30 09:17:23.213778 [ 7759]: Thawing priority 2
>2014/10/30 09:17:23.213801 [ 7759]: Release freeze handler for prio 2
>2014/10/30 09:17:23.213846 [ 7759]: Thawing priority 3
>2014/10/30 09:17:23.213865 [ 7759]: Release freeze handler for prio 3
>2014/10/30 09:17:23.539615 [recoverd: 8031]: Resetting ban count to 0 for all nodes
>2014/10/30 09:17:33.749489 [recoverd: 8031]: server/ctdb_recoverd.c:3933 Remote node:1 has different flags for node 0. It has 0x02 vs our 0x00
>2014/10/30 09:17:33.749545 [recoverd: 8031]: Use flags 0x00 from local recmaster node for cluster update of node 0 flags
>2014/10/30 09:17:33.753336 [recoverd: 8031]: Taking out recovery lock from recovery daemon
>2014/10/30 09:17:33.753368 [recoverd: 8031]: Take the recovery lock
>2014/10/30 09:17:33.882711 [ 7759]: Freeze priority 1
>2014/10/30 09:17:33.890523 [ 7759]: Freeze priority 2
>2014/10/30 09:17:33.898881 [ 7759]: Freeze priority 3
>2014/10/30 09:17:35.654818 [ 7759]: Thawing priority 1
>2014/10/30 09:17:35.654916 [ 7759]: Release freeze handler for prio 1
>2014/10/30 09:17:35.654984 [ 7759]: Thawing priority 2
>2014/10/30 09:17:35.655005 [ 7759]: Release freeze handler for prio 2
>2014/10/30 09:17:35.655034 [ 7759]: Thawing priority 3
>2014/10/30 09:17:35.655051 [ 7759]: Release freeze handler for prio 3
>2014/10/30 09:17:36.066372 [recoverd: 8031]: Resetting ban count to 0 for all nodes
>2014/10/30 09:20:22.995568 [ 7759]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 09:20:23.008057 [ 7759]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 09:21:47.278429 [ 1518]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 09:21:47.278529 [ 1518]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 09:21:47.278543 [ 1518]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 09:25:48.759922 [ 8158]: Starting CTDBD (Version 2.5.3) as PID: 8158
>2014/10/30 09:25:50.191843 [ 8158]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 09:25:50.215412 [ 8158]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 09:25:50.229591 [ 8158]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 09:25:50.243583 [ 8158]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 09:25:50.243601 [ 8158]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 09:25:50.243610 [ 8158]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 09:25:50.243619 [ 8158]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 09:25:50.243627 [ 8158]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 09:25:50.243635 [ 8158]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 09:25:50.257675 [ 8158]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 09:25:50.271652 [ 8158]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 09:25:50.271671 [ 8158]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 09:25:50.271680 [ 8158]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 09:25:50.285592 [ 8158]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 09:25:50.285625 [ 8158]: Freeze priority 1
>2014/10/30 09:25:50.303790 [ 8158]: Freeze priority 2
>2014/10/30 09:25:50.304142 [ 8158]: Freeze priority 3
>2014/10/30 09:25:50.468063 [ 8158]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 09:25:50.471820 [ 8158]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 09:25:50.475348 [ 8158]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 09:25:50.594568 [ 8158]: Freeze priority 1
>2014/10/30 09:25:50.594659 [ 8158]: Freeze priority 2
>2014/10/30 09:25:50.594732 [ 8158]: Freeze priority 3
>2014/10/30 09:25:50.832826 [ 8158]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 09:25:50.937223 [ 8158]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 09:25:50.943320 [ 8158]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 09:25:54.602479 [recoverd: 8440]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 09:25:54.602559 [ 8158]: Freeze priority 1
>2014/10/30 09:25:54.602621 [ 8158]: Freeze priority 2
>2014/10/30 09:25:54.602685 [ 8158]: Freeze priority 3
>2014/10/30 09:25:58.699242 [ 8158]: Freeze priority 1
>2014/10/30 09:25:58.707178 [ 8158]: Freeze priority 2
>2014/10/30 09:25:58.708009 [ 8158]: Freeze priority 3
>2014/10/30 09:25:58.865421 [ 8158]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 09:25:58.866126 [ 8158]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 09:25:58.866957 [ 8158]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 09:26:01.408850 [ 8158]: Thawing priority 1
>2014/10/30 09:26:01.408926 [ 8158]: Release freeze handler for prio 1
>2014/10/30 09:26:01.408968 [ 8158]: Thawing priority 2
>2014/10/30 09:26:01.409003 [ 8158]: Release freeze handler for prio 2
>2014/10/30 09:26:01.409029 [ 8158]: Thawing priority 3
>2014/10/30 09:26:01.409046 [ 8158]: Release freeze handler for prio 3
>2014/10/30 09:26:16.404396 [ 8158]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 09:26:16.436195 [recoverd: 8440]: Trigger takeoverrun
>2014/10/30 09:26:16.778567 [ 8158]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 09:26:16.791969 [ 8158]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 09:26:16.820562 [ 8158]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 09:26:17.003659 [ 8158]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 09:26:19.472638 [ 8158]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation
>2014/10/30 09:26:20.387370 [ 8158]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 09:30:24.770985 [ 8158]: Freeze priority 1
>2014/10/30 09:30:24.792197 [ 8158]: Freeze priority 1
>2014/10/30 09:30:24.795917 [ 8158]: Freeze priority 1
>2014/10/30 09:30:24.840950 [ 8158]: Freeze priority 2
>2014/10/30 09:30:24.841073 [ 8158]: Freeze priority 2
>2014/10/30 09:30:24.841110 [ 8158]: Freeze priority 2
>2014/10/30 09:30:24.841784 [ 8158]: Freeze priority 3
>2014/10/30 09:30:24.841917 [ 8158]: Freeze priority 3
>2014/10/30 09:30:24.841965 [ 8158]: Freeze priority 3
>2014/10/30 09:30:27.859902 [ 8158]: Freeze priority 1
>2014/10/30 09:30:27.860211 [ 8158]: Freeze priority 2
>2014/10/30 09:30:27.860422 [ 8158]: Freeze priority 3
>2014/10/30 09:30:30.816961 [ 8158]: Thawing priority 1
>2014/10/30 09:30:30.817006 [ 8158]: Release freeze handler for prio 1
>2014/10/30 09:30:30.817035 [ 8158]: Thawing priority 2
>2014/10/30 09:30:30.817054 [ 8158]: Release freeze handler for prio 2
>2014/10/30 09:30:30.817082 [ 8158]: Thawing priority 3
>2014/10/30 09:30:30.817101 [ 8158]: Release freeze handler for prio 3
>2014/10/30 09:35:35.853417 [ 8158]: Freeze priority 1
>2014/10/30 09:35:35.955426 [ 8158]: Freeze priority 2
>2014/10/30 09:35:35.956890 [ 8158]: Freeze priority 3
>2014/10/30 09:35:39.191937 [ 8158]: Thawing priority 1
>2014/10/30 09:35:39.191973 [ 8158]: Release freeze handler for prio 1
>2014/10/30 09:35:39.192007 [ 8158]: Thawing priority 2
>2014/10/30 09:35:39.192028 [ 8158]: Release freeze handler for prio 2
>2014/10/30 09:35:39.192061 [ 8158]: Thawing priority 3
>2014/10/30 09:35:39.192081 [ 8158]: Release freeze handler for prio 3
>2014/10/30 09:45:34.038341 [ 8158]: Freeze priority 1
>2014/10/30 09:45:34.162299 [ 8158]: Freeze priority 2
>2014/10/30 09:45:34.163782 [ 8158]: Freeze priority 3
>2014/10/30 09:45:38.086931 [ 8158]: Freeze priority 1
>2014/10/30 09:45:38.089472 [ 8158]: Freeze priority 2
>2014/10/30 09:45:38.092109 [ 8158]: Freeze priority 3
>2014/10/30 09:45:42.002518 [ 8158]: Thawing priority 1
>2014/10/30 09:45:42.002566 [ 8158]: Release freeze handler for prio 1
>2014/10/30 09:45:42.002597 [ 8158]: Thawing priority 2
>2014/10/30 09:45:42.002617 [ 8158]: Release freeze handler for prio 2
>2014/10/30 09:45:42.002657 [ 8158]: Thawing priority 3
>2014/10/30 09:45:42.002679 [ 8158]: Release freeze handler for prio 3
>2014/10/30 09:50:51.431212 [recoverd: 8440]: Taking out recovery lock from recovery daemon
>2014/10/30 09:50:51.431265 [recoverd: 8440]: Take the recovery lock
>2014/10/30 09:50:51.494324 [ 8158]: Freeze priority 1
>2014/10/30 09:50:51.595151 [ 8158]: Freeze priority 2
>2014/10/30 09:50:51.597727 [ 8158]: Freeze priority 3
>2014/10/30 09:50:55.750899 [ 8158]: Thawing priority 1
>2014/10/30 09:50:55.750959 [ 8158]: Release freeze handler for prio 1
>2014/10/30 09:50:55.751019 [ 8158]: Thawing priority 2
>2014/10/30 09:50:55.751043 [ 8158]: Release freeze handler for prio 2
>2014/10/30 09:50:55.751079 [ 8158]: Thawing priority 3
>2014/10/30 09:50:55.751116 [ 8158]: Release freeze handler for prio 3
>2014/10/30 09:50:56.150909 [recoverd: 8440]: Resetting ban count to 0 for all nodes
>2014/10/30 09:53:37.381277 [recoverd: 8440]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 4 vs 3
>2014/10/30 09:53:37.381348 [recoverd: 8440]: Taking out recovery lock from recovery daemon
>2014/10/30 09:53:37.381362 [recoverd: 8440]: Take the recovery lock
>2014/10/30 09:53:37.391351 [ 8158]: Freeze priority 1
>2014/10/30 09:53:37.418153 [ 8158]: Freeze priority 2
>2014/10/30 09:53:37.419106 [ 8158]: Freeze priority 3
>2014/10/30 09:53:39.741314 [ 8158]: Thawing priority 1
>2014/10/30 09:53:39.741354 [ 8158]: Release freeze handler for prio 1
>2014/10/30 09:53:39.741394 [ 8158]: Thawing priority 2
>2014/10/30 09:53:39.741428 [ 8158]: Release freeze handler for prio 2
>2014/10/30 09:53:39.741469 [ 8158]: Thawing priority 3
>2014/10/30 09:53:39.741490 [ 8158]: Release freeze handler for prio 3
>2014/10/30 09:53:39.742528 [ 8158]: server/ctdb_call.c:1005 reqid 52646 not found
>2014/10/30 09:53:39.742582 [ 8158]: server/ctdb_call.c:1005 reqid 52648 not found
>2014/10/30 09:53:39.743615 [ 8158]: pnn 2 Invalid reqid 52645 in ctdb_become_dmaster from node 1
>2014/10/30 09:53:39.743675 [ 8158]: server/ctdb_call.c:1005 reqid 52647 not found
>2014/10/30 09:53:40.204678 [recoverd: 8440]: Resetting ban count to 0 for all nodes
>2014/10/30 09:58:53.462851 [ 8158]: Freeze priority 1
>2014/10/30 09:58:53.618452 [ 8158]: Freeze priority 2
>2014/10/30 09:58:53.621452 [ 8158]: Freeze priority 3
>2014/10/30 09:58:53.627339 [ 8158]: Monitoring event was cancelled
>2014/10/30 09:58:53.627371 [ 8158]: server/eventscript.c:569 Sending SIGTERM to child pid:18191
>2014/10/30 09:58:58.639383 [ 8158]: Thawing priority 1
>2014/10/30 09:58:58.639427 [ 8158]: Release freeze handler for prio 1
>2014/10/30 09:58:58.639462 [ 8158]: Thawing priority 2
>2014/10/30 09:58:58.639480 [ 8158]: Release freeze handler for prio 2
>2014/10/30 09:58:58.639509 [ 8158]: Thawing priority 3
>2014/10/30 09:58:58.639526 [ 8158]: Release freeze handler for prio 3
>2014/10/30 09:59:09.191231 [ 8158]: Freeze priority 1
>2014/10/30 09:59:09.256792 [ 8158]: Freeze priority 2
>2014/10/30 09:59:09.260339 [ 8158]: Freeze priority 3
>2014/10/30 09:59:09.264789 [ 8158]: Monitoring event was cancelled
>2014/10/30 09:59:09.264824 [ 8158]: server/eventscript.c:569 Sending SIGTERM to child pid:19107
>2014/10/30 09:59:13.953342 [ 8158]: Thawing priority 1
>2014/10/30 09:59:13.953397 [ 8158]: Release freeze handler for prio 1
>2014/10/30 09:59:13.953443 [ 8158]: Thawing priority 2
>2014/10/30 09:59:13.953462 [ 8158]: Release freeze handler for prio 2
>2014/10/30 09:59:13.953498 [ 8158]: Thawing priority 3
>2014/10/30 09:59:13.953514 [ 8158]: Release freeze handler for prio 3
>2014/10/30 10:11:08.019433 [ 8158]: 60.nfs: ERROR: rquotad failed RPC check:
>2014/10/30 10:11:08.046885 [ 8158]: 60.nfs: rpcinfo: RPC: Program not registered
>2014/10/30 10:11:08.046917 [ 8158]: 60.nfs: program 100011 version 1 is not available
>2014/10/30 10:11:08.046964 [ 8158]: 60.nfs: Trying to restart rquotad [rpc.rquotad]
>2014/10/30 10:11:14.716921 [ 8158]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 10:11:14.729115 [ 8158]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 10:14:05.724484 [ 1502]: Recovery lock file set to "". Disabling recovery lock checking
>2014/10/30 10:14:05.724603 [ 1502]: ctdb error: Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 10:14:05.724618 [ 1502]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes'
>
>2014/10/30 10:18:05.538175 [ 7589]: Starting CTDBD (Version 2.5.3) as PID: 7589
>2014/10/30 10:18:07.378398 [ 7589]: Vacuuming is disabled for persistent database registry.tdb
>2014/10/30 10:18:07.401642 [ 7589]: Vacuuming is disabled for persistent database passdb.tdb
>2014/10/30 10:18:07.415797 [ 7589]: Vacuuming is disabled for persistent database secrets.tdb
>2014/10/30 10:18:07.429977 [ 7589]: Vacuuming is disabled for persistent database share_info.tdb
>2014/10/30 10:18:07.429996 [ 7589]: Ignoring persistent database 'account_policy.tdb.1'
>2014/10/30 10:18:07.430006 [ 7589]: Ignoring persistent database 'ctdb.tdb.1'
>2014/10/30 10:18:07.430015 [ 7589]: Ignoring persistent database 'group_mapping.tdb.1'
>2014/10/30 10:18:07.430024 [ 7589]: Ignoring persistent database 'secrets.tdb.1'
>2014/10/30 10:18:07.430033 [ 7589]: Ignoring persistent database 'share_info.tdb.1'
>2014/10/30 10:18:07.444550 [ 7589]: Vacuuming is disabled for persistent database ctdb.tdb
>2014/10/30 10:18:07.459083 [ 7589]: Vacuuming is disabled for persistent database account_policy.tdb
>2014/10/30 10:18:07.459103 [ 7589]: Ignoring persistent database 'passdb.tdb.1'
>2014/10/30 10:18:07.459112 [ 7589]: Ignoring persistent database 'registry.tdb.1'
>2014/10/30 10:18:07.473599 [ 7589]: Vacuuming is disabled for persistent database group_mapping.tdb
>2014/10/30 10:18:07.473642 [ 7589]: Freeze priority 1
>2014/10/30 10:18:07.504497 [ 7589]: Freeze priority 2
>2014/10/30 10:18:07.504916 [ 7589]: Freeze priority 3
>2014/10/30 10:18:07.679590 [ 7589]: 00.ctdb: Set EventScriptTimeout to 60
>2014/10/30 10:18:07.683268 [ 7589]: 00.ctdb: Set RecoverTimeout to 60
>2014/10/30 10:18:07.686581 [ 7589]: 00.ctdb: Set RecoveryBanPeriod to 30
>2014/10/30 10:18:07.813379 [ 7589]: Freeze priority 1
>2014/10/30 10:18:07.813455 [ 7589]: Freeze priority 2
>2014/10/30 10:18:07.813508 [ 7589]: Freeze priority 3
>2014/10/30 10:18:08.227865 [ 7589]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum
>2014/10/30 10:18:11.820204 [recoverd: 7872]: server/ctdb_recoverd.c:3692 Current recmaster node 1 does not have CAP_RECMASTER, but we (node 2) have - force an election
>2014/10/30 10:18:11.820284 [ 7589]: Freeze priority 1
>2014/10/30 10:18:11.820344 [ 7589]: Freeze priority 2
>2014/10/30 10:18:11.820394 [ 7589]: Freeze priority 3
>2014/10/30 10:18:16.150196 [ 7589]: Freeze priority 1
>2014/10/30 10:18:16.161389 [ 7589]: Freeze priority 2
>2014/10/30 10:18:16.162263 [ 7589]: Freeze priority 3
>2014/10/30 10:18:16.297203 [ 7589]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup
>2014/10/30 10:18:16.297738 [ 7589]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup
>2014/10/30 10:18:16.298502 [ 7589]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup
>2014/10/30 10:18:19.087828 [ 7589]: Thawing priority 1
>2014/10/30 10:18:19.087875 [ 7589]: Release freeze handler for prio 1
>2014/10/30 10:18:19.087909 [ 7589]: Thawing priority 2
>2014/10/30 10:18:19.087929 [ 7589]: Release freeze handler for prio 2
>2014/10/30 10:18:19.087972 [ 7589]: Thawing priority 3
>2014/10/30 10:18:19.087988 [ 7589]: Release freeze handler for prio 3
>2014/10/30 10:18:34.032164 [ 7589]: 50.samba: Redirecting to /bin/systemctl start  smb.service
>2014/10/30 10:18:34.114149 [recoverd: 7872]: Trigger takeoverrun
>2014/10/30 10:18:34.347415 [ 7589]: 60.nfs: Stopping nfslock (via systemctl):  [  OK  ]
>2014/10/30 10:18:34.360712 [ 7589]: 60.nfs: Redirecting to /bin/systemctl stop  nfs.service
>2014/10/30 10:18:34.389691 [ 7589]: 60.nfs: Redirecting to /bin/systemctl start  nfs.service
>2014/10/30 10:18:34.614431 [ 7589]: 60.nfs: Starting nfslock (via systemctl):  [  OK  ]
>2014/10/30 10:18:37.023882 [ 7589]: Node became HEALTHY. Ask recovery master 1 to perform ip reallocation
>2014/10/30 10:18:37.904380 [ 7589]: 60.nfs: Reconfiguring service "nfs"...
>2014/10/30 10:28:30.805726 [ 7589]: Freeze priority 1
>2014/10/30 10:28:31.291855 [ 7589]: Freeze priority 1
>2014/10/30 10:28:31.309339 [ 7589]: Freeze priority 2
>2014/10/30 10:28:31.310567 [ 7589]: Freeze priority 3
>2014/10/30 10:28:31.817334 [recoverd: 7872]: server/ctdb_recoverd.c:2343 Reload nodes file from recovery daemon
>2014/10/30 10:28:32.243238 [ 7589]: Freeze priority 1
>2014/10/30 10:28:32.243710 [ 7589]: Freeze priority 2
>2014/10/30 10:28:32.244090 [ 7589]: Freeze priority 3
>2014/10/30 10:28:33.974842 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:28:33.974906 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:28:34.137396 [ 7589]: Thawing priority 1
>2014/10/30 10:28:34.137441 [ 7589]: Release freeze handler for prio 1
>2014/10/30 10:28:34.137476 [ 7589]: Thawing priority 2
>2014/10/30 10:28:34.137499 [ 7589]: Release freeze handler for prio 2
>2014/10/30 10:28:34.137530 [ 7589]: Thawing priority 3
>2014/10/30 10:28:34.137551 [ 7589]: Release freeze handler for prio 3
>2014/10/30 10:28:56.156817 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:28:59.159370 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:02.161617 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:05.162924 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:08.164475 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:11.168203 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:14.170780 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:17.171837 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:20.174829 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:23.177413 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:26.178249 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:29:29.179542 [ 7589]: Refused connection from unknown node 10.192.55.182
>2014/10/30 10:39:35.274922 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:39:35.275819 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:39:35.277700 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:39:35.278954 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:39:35.280102 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:39:35.280431 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:52:35.989398 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:52:35.992143 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:52:36.000357 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:52:36.004212 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:52:36.006268 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0
>2014/10/30 10:52:36.006639 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0

Actions: View

Attachments on bug 10934: 10432 | 10853