Created attachment 15122 [details]
ctdb_strace

We have ctdb - glusterfs setup in our organization with 2 replicas. 
Recently "/var" partition of 1st node got full and ctdb crashed in 1st node.

We are not able to startup CTDB service after that. 
Startup fails with the following error:

2019/05/03 13:01:45.337679 [10182]: Event script '00.ctdb init ' timed out after 30.0s, count: 0, pid: 10184
2019/05/03 13:01:45.337883 [10182]: Added timed event " ": 0x7f92c983b2a4
2019/05/03 13:01:45.337901 [10182]: Ending timer event 0x7ffe707654c8 "Ending timer event %p "%s"
2019/05/03 13:01:45.337908 [10182]: "
2019/05/03 13:01:45.337916 [10182]: ../ctdb/server/eventscript.c:889 eventscript for 'init' timedout. Immediately banning ourself for 300 seconds
2019/05/03 13:01:45.337930 [10182]: ctdb exiting with error: Failed to run init event
2019/05/03 13:01:45.337937 [10182]: 
2019/05/03 13:01:45.337947 [10182]: CTDB daemon shutting down
2019/05/03 13:01:45.361538 [10182]: Removed PID file /run/ctdb/ctdbd.pid

CTDB config file:

[root@nfs01 ~]# cat /etc/sysconfig/ctdb 
CTDB_NODES=/etc/ctdb/nodes
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_RECOVERY_LOCK="/mnt/gluster/ctdb_lock/.ctdb.lock"
CTDB_MANAGES_SAMBA=yes
CTDB_MANAGES_WINBIND=yes
CTDB_MANAGES_NFS=yes
CTDB_DEBUGLEVEL=DEBUG




We have no clue on how to resolve this issue. Please help.

- Strace  output  has been attached