The Samba-Bugzilla – Attachment 10853 Details for
Bug 10934
ctdb banned error,while reboot one node
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Requests
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
Log of node that get's banned, low loglevel
ctdb_banned_loglevel2 (text/plain), 6.80 KB, created by
Systemen
on 2015-03-11 13:15:24 UTC
(
hide
)
Description:
Log of node that get's banned, low loglevel
Filename:
MIME Type:
Creator:
Systemen
Created:
2015-03-11 13:15:24 UTC
Size:
6.80 KB
patch
obsolete
>2015/03/11 13:52:40.571327 [16667]: 172.16.223.60:4379: node 172.16.223.61:4379 is dead: 0 connected >2015/03/11 13:52:40.571367 [16667]: Tearing down connection to dead node :1 >2015/03/11 13:52:41.104273 [recoverd:16839]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 2 vs 1 >2015/03/11 13:52:41.104314 [recoverd:16839]: server/ctdb_recoverd.c:1765 Starting do_recovery >2015/03/11 13:52:41.104320 [recoverd:16839]: Taking out recovery lock from recovery daemon >2015/03/11 13:52:41.104325 [recoverd:16839]: Take the recovery lock >2015/03/11 13:52:41.104510 [recoverd:16839]: Recovery lock taken successfully >2015/03/11 13:52:41.104521 [recoverd:16839]: ctdb_recovery_lock: Got recovery lock on '/users/local/ctdb.lock' >2015/03/11 13:52:41.104582 [recoverd:16839]: Recovery lock taken successfully by recovery daemon >2015/03/11 13:52:41.104592 [recoverd:16839]: server/ctdb_recoverd.c:1790 Recovery initiated due to problem with node 0 >2015/03/11 13:52:41.104646 [recoverd:16839]: server/ctdb_recoverd.c:1815 Recovery - created remote databases >2015/03/11 13:52:41.105334 [recoverd:16839]: server/ctdb_recoverd.c:1822 Recovery - updated db priority for all databases >2015/03/11 13:52:41.105382 [16667]: Freeze priority 1 >2015/03/11 13:52:51.106125 [16667]: Unable to get ALLDB locks for 10 seconds >2015/03/11 13:52:52.005566 [16667]: 172.16.223.60:4379: connected to 172.16.223.61:4379 - 1 connected >2015/03/11 13:52:52.611872 [16667]: Skip monitoring since databases are frozen >2015/03/11 13:52:53.311494 [recoverd:16839]: server/ctdb_recoverd.c:1139 Election timed out >2015/03/11 13:52:56.816067 [recoverd:16839]: server/ctdb_recoverd.c:1139 Election timed out >2015/03/11 13:53:01.106923 [16667]: Unable to get ALLDB locks for 20 seconds >2015/03/11 13:53:07.612334 [16667]: Skip monitoring since databases are frozen >2015/03/11 13:53:11.108204 [16667]: Unable to get ALLDB locks for 30 seconds >2015/03/11 13:53:21.108581 [16667]: Unable to get ALLDB locks for 40 seconds >2015/03/11 13:53:22.612606 [16667]: Skip monitoring since databases are frozen >2015/03/11 13:53:31.109680 [16667]: Unable to get ALLDB locks for 50 seconds >2015/03/11 13:53:37.613527 [16667]: Skip monitoring since databases are frozen >2015/03/11 13:53:41.104076 [16667]: Recovery daemon ping timeout. Count : 0 >2015/03/11 13:53:41.106207 [recoverd:16839]: ctdb_control error: 'ctdb_control timed out' >2015/03/11 13:53:41.106223 [recoverd:16839]: ctdb_control error: 'ctdb_control timed out' >2015/03/11 13:53:41.106262 [recoverd:16839]: Async operation failed with ret=-1 res=-1 opcode=33 >2015/03/11 13:53:41.106272 [recoverd:16839]: Failed to freeze node 0 during recovery. Set it as ban culprit for 2 credits >2015/03/11 13:53:41.106281 [recoverd:16839]: Async wait failed - fail_count=1 >2015/03/11 13:53:41.106287 [recoverd:16839]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. >2015/03/11 13:53:41.106292 [recoverd:16839]: server/ctdb_recoverd.c:1833 Unable to set recovery mode to active on cluster >2015/03/11 13:53:41.107767 [recoverd:16839]: server/ctdb_recoverd.c:1765 Starting do_recovery >2015/03/11 13:53:41.107780 [recoverd:16839]: Taking out recovery lock from recovery daemon >2015/03/11 13:53:41.107787 [recoverd:16839]: Take the recovery lock >2015/03/11 13:53:41.108131 [recoverd:16839]: Recovery lock taken successfully >2015/03/11 13:53:41.108146 [recoverd:16839]: ctdb_recovery_lock: Got recovery lock on '/users/local/ctdb.lock' >2015/03/11 13:53:41.108205 [recoverd:16839]: Recovery lock taken successfully by recovery daemon >2015/03/11 13:53:41.108215 [recoverd:16839]: server/ctdb_recoverd.c:1790 Recovery initiated due to problem with node 0 >2015/03/11 13:53:41.110943 [16667]: Unable to get ALLDB locks for 60 seconds >2015/03/11 13:53:41.133244 [recoverd:16839]: server/ctdb_recoverd.c:1815 Recovery - created remote databases >2015/03/11 13:53:41.137952 [recoverd:16839]: server/ctdb_recoverd.c:1822 Recovery - updated db priority for all databases >2015/03/11 13:53:41.138251 [16667]: Freeze priority 1 >2015/03/11 13:53:51.111205 [16667]: Unable to get ALLDB locks for 70 seconds >2015/03/11 13:53:52.614432 [16667]: Skip monitoring since databases are frozen >2015/03/11 13:54:01.111639 [16667]: Unable to get ALLDB locks for 80 seconds >2015/03/11 13:54:07.614943 [16667]: Skip monitoring since databases are frozen >2015/03/11 13:54:11.112299 [16667]: Unable to get ALLDB locks for 90 seconds >2015/03/11 13:54:21.113215 [16667]: Unable to get ALLDB locks for 100 seconds >2015/03/11 13:54:22.615444 [16667]: Skip monitoring since databases are frozen >2015/03/11 13:54:31.114016 [16667]: Unable to get ALLDB locks for 110 seconds >2015/03/11 13:54:37.616152 [16667]: Skip monitoring since databases are frozen >2015/03/11 13:54:41.106864 [16667]: Recovery daemon ping timeout. Count : 0 >2015/03/11 13:54:41.114980 [16667]: Unable to get ALLDB locks for 120 seconds >2015/03/11 13:54:41.138963 [recoverd:16839]: ctdb_control error: 'ctdb_control timed out' >2015/03/11 13:54:41.138985 [recoverd:16839]: ctdb_control error: 'ctdb_control timed out' >2015/03/11 13:54:41.138993 [recoverd:16839]: Async operation failed with ret=-1 res=-1 opcode=33 >2015/03/11 13:54:41.138998 [recoverd:16839]: Failed to freeze node 0 during recovery. Set it as ban culprit for 2 credits >2015/03/11 13:54:41.139005 [recoverd:16839]: Async wait failed - fail_count=1 >2015/03/11 13:54:41.139010 [recoverd:16839]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. >2015/03/11 13:54:41.139037 [recoverd:16839]: server/ctdb_recoverd.c:1833 Unable to set recovery mode to active on cluster >2015/03/11 13:54:41.139245 [recoverd:16839]: Node 0 reached 5 banning credits - banning it for 300 seconds >2015/03/11 13:54:41.139255 [recoverd:16839]: Banning node 0 for 300 seconds >2015/03/11 13:54:41.139282 [16667]: Banning this node for 300 seconds >2015/03/11 13:54:41.139292 [16667]: This node has been banned - forcing freeze and recovery >2015/03/11 13:54:41.139297 [16667]: Freeze priority 1 >2015/03/11 13:54:41.139304 [16667]: Freeze priority 2 >2015/03/11 13:54:41.139437 [16667]: Freeze priority 3 >2015/03/11 13:54:41.139538 [16667]: server/ctdb_takeover.c:3291 Released 0 public IPs >2015/03/11 13:54:41.139569 [recoverd:16839]: This node was banned, restart main_loop >2015/03/11 13:54:41.493575 [16667]: This node (0) is no longer the recovery master >2015/03/11 13:54:44.493295 [recoverd:16839]: server/ctdb_recoverd.c:1139 Election timed out >2015/03/11 13:54:44.498007 [16667]: This node has been banned - forcing freeze and recovery >2015/03/11 13:54:44.498027 [16667]: Freeze priority 1 >2015/03/11 13:54:44.498034 [16667]: server/ctdb_takeover.c:3291 Released 0 public IPs >2015/03/11 13:54:44.498122 [recoverd:16839]: Node 0 has changed flags - now 0x8 was 0x0 >2015/03/11 13:54:44.723373 [recoverd:16839]: Disabling takeover runs for 60 seconds >2015/03/11 13:54:44.771501 [recoverd:16839]: Reenabling takeover runs
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 10934
:
10432
| 10853