Attachment 16191 Details for Bug 14475 – access after free full log

access after free full log

ctdb-eventd access after free.log (text/x-log), 12.38 KB, created by Olivier Garaud on 2020-09-01 08:01:47 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Olivier Garaud

Created: 2020-09-01 08:01:47 UTC

Size: 12.38 KB

patch

obsolete

>Aug 19 12:41:28 NFSConnector01 ctdbd[6289]: CTDB starting on node
>Aug 19 12:41:28 NFSConnector01 ctdbd[6289]: Recovery lock not set
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Starting CTDBD (Version 4.10.8-SerNet-RedHat-12.el7) as PID: 6336
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Created PID file /var/run/ctdb/ctdbd.pid
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Listening to ctdb socket /var/run/ctdb/ctdbd.socket
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Set real-time scheduler priority
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Starting event daemon /usr/libexec/ctdb/ctdb-eventd -P 6336 -S 13
>Aug 19 12:41:28 NFSConnector01 ctdb-eventd[6338]: daemon started, pid=6338
>Aug 19 12:41:28 NFSConnector01 ctdb-eventd[6338]: startup completed successfully
>Aug 19 12:41:28 NFSConnector01 ctdb-eventd[6338]: listening on /var/run/ctdb/eventd.socket
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Set runstate to INIT (1)
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: PNN is 0
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Loaded public addresses from /etc/ctdb/public_addresses
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Vacuuming is disabled for non-volatile database ctdb.tdb
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Attached to database '/var/lib/ctdb/persistent/ctdb.tdb.0' with flags 0x400
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Freeze db: ctdb.tdb
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Set lock helper to "/usr/libexec/ctdb/ctdb_lock_helper"
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Set runstate to SETUP (2)
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Keepalive monitoring has been started
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: Set runstate to FIRST_RECOVERY (3)
>Aug 19 12:41:28 NFSConnector01 ctdb-recoverd[6512]: monitor_cluster starting
>Aug 19 12:41:28 NFSConnector01 ctdb-recoverd[6512]: Initial recovery master set - forcing election
>Aug 19 12:41:28 NFSConnector01 ctdbd[6336]: This node (0) is now the recovery master
>Aug 19 12:41:29 NFSConnector01 sernet-samba-ctdb[6227]: Starting CTDB ctdbd : [  OK  ]
>Aug 19 12:41:29 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:30 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:31 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Election period ended
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Node:0 was in recovery mode. Start recovery process
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_recoverd.c:1342 Starting do_recovery
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_recoverd.c:1417 Recovery initiated due to problem with node 0
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_recoverd.c:1442 Recovery - created remote databases
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_recoverd.c:1471 Recovery - updated flags
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Set recovery_helper to "/usr/libexec/ctdb/ctdb_recovery_helper"
>Aug 19 12:41:31 NFSConnector01 ctdb-recovery[6989]: Set recovery mode to ACTIVE
>Aug 19 12:41:31 NFSConnector01 ctdbd[6336]: Recovery has started
>Aug 19 12:41:31 NFSConnector01 ctdb-recovery[6989]: start_recovery event finished
>Aug 19 12:41:31 NFSConnector01 ctdb-recovery[6989]: updated VNNMAP
>Aug 19 12:41:31 NFSConnector01 ctdb-recovery[6989]: recover database 0x6645c6c4
>Aug 19 12:41:31 NFSConnector01 ctdbd[6336]: Freeze db: ctdb.tdb frozen
>Aug 19 12:41:31 NFSConnector01 ctdbd[6336]: Thaw db: ctdb.tdb generation 1389672539
>Aug 19 12:41:31 NFSConnector01 ctdbd[6336]: Release freeze handle for db ctdb.tdb
>Aug 19 12:41:31 NFSConnector01 ctdb-recovery[6989]: 1 of 1 databases recovered
>Aug 19 12:41:31 NFSConnector01 ctdbd[6336]: Recovery mode set to NORMAL
>Aug 19 12:41:31 NFSConnector01 ctdb-recovery[6989]: Set recovery mode to NORMAL
>Aug 19 12:41:31 NFSConnector01 ctdbd[6336]: Recovery has finished
>Aug 19 12:41:31 NFSConnector01 ctdbd[6336]: Set runstate to STARTUP (4)
>Aug 19 12:41:31 NFSConnector01 ctdb-recovery[6989]: recovered event finished
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Takeover run starting
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Set takeover_helper to "/usr/libexec/ctdb/ctdb_takeover_helper"
>Aug 19 12:41:31 NFSConnector01 ctdb-takeover[7008]: No nodes available to host public IPs yet
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Takeover run completed successfully
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_recoverd.c:1489 Recovery complete
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Resetting ban count to 0 for all nodes
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Just finished a recovery. New recoveries will now be suppressed for the rerecovery timeout (10 seconds)
>Aug 19 12:41:31 NFSConnector01 ctdb-recoverd[6512]: Disabling recoveries for 10 seconds
>Aug 19 12:41:32 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:32 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:32 NFSConnector01 ctdb-recoverd[6512]: Initial interface fetched
>Aug 19 12:41:32 NFSConnector01 ctdb-recoverd[6512]: Trigger takeoverrun
>Aug 19 12:41:32 NFSConnector01 ctdb-recoverd[6512]: Takeover run starting
>Aug 19 12:41:32 NFSConnector01 ctdb-takeover[7024]: No nodes available to host public IPs yet
>Aug 19 12:41:32 NFSConnector01 ctdb-recoverd[6512]: Takeover run completed successfully
>Aug 19 12:41:33 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:33 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:34 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:34 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:35 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:35 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:36 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:36 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:37 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:37 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:38 NFSConnector01 ctdbd[6336]: 192.168.2.31:4379: connected to 192.168.2.32:4379 - 1 connected
>Aug 19 12:41:38 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:38 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:39 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:39 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:40 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:40 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:41 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:41 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:41 NFSConnector01 ctdb-recoverd[6512]: Reenabling recoveries after timeout
>Aug 19 12:41:42 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:42 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:43 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:43 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:44 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:44 NFSConnector01 ctdbd[6336]: ../../ctdb/server/ctdb_monitor.c:324 wait for pending recoveries to end. Wait one more second.
>Aug 19 12:41:45 NFSConnector01 ctdbd[6336]: CTDB_WAIT_UNTIL_RECOVERED
>Aug 19 12:41:45 NFSConnector01 ctdbd[6336]: ctdb_recheck_persistent_health: OK[1] FAIL[0]
>Aug 19 12:41:45 NFSConnector01 ctdbd[6336]: Running the "startup" event.
>Aug 19 12:42:08 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_client.c:678 control timed out. reqid:2147483574 opcode:80 dstnode:1
>Aug 19 12:42:08 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_client.c:791 ctdb_control_recv failed
>Aug 19 12:42:08 NFSConnector01 ctdb-recoverd[6512]: Async operation failed with state 3, opcode:80
>Aug 19 12:42:08 NFSConnector01 ctdb-recoverd[6512]: Async wait failed - fail_count=1
>Aug 19 12:42:08 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_client.c:1920 Failed to read node capabilities.
>Aug 19 12:42:08 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_recoverd.c:370 Failed to get node capabilities
>Aug 19 12:42:08 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_recoverd.c:2756 Unable to update node capabilities.
>Aug 19 12:42:08 NFSConnector01 ctdb-recoverd[6512]: Node 1 thinks node 1 is recmaster. Need a new recmaster election
>Aug 19 12:42:08 NFSConnector01 ctdbd[6336]: Recovery mode set to ACTIVE
>Aug 19 12:42:09 NFSConnector01 ctdbd[6336]: Remote node (1) is now the recovery master
>Aug 19 12:42:12 NFSConnector01 ctdb-recoverd[6512]: Election period ended
>Aug 19 12:42:15 NFSConnector01 ctdb-eventd[6338]: 08.scality-sfused: Redirecting to /bin/systemctl restart scality-sfused.service
>Aug 19 12:42:15 NFSConnector01 ctdb-eventd[6338]: startup event timed out
>Aug 19 12:42:15 NFSConnector01 ctdbd[6336]: startup event failed
>Aug 19 12:42:20 NFSConnector01 ctdbd[6336]: Running the "startup" event.
>Aug 19 12:42:20 NFSConnector01 ctdbd[6336]: Refusing to run event 'startup' while in recovery
>Aug 19 12:42:20 NFSConnector01 ctdbd[6336]: Unable to launch startup event script
>Aug 19 12:42:25 NFSConnector01 ctdbd[6336]: Running the "startup" event.
>Aug 19 12:42:25 NFSConnector01 ctdbd[6336]: Refusing to run event 'startup' while in recovery
>Aug 19 12:42:25 NFSConnector01 ctdbd[6336]: Unable to launch startup event script
>Aug 19 12:42:30 NFSConnector01 ctdbd[6336]: Running the "startup" event.
>Aug 19 12:42:30 NFSConnector01 ctdbd[6336]: Refusing to run event 'startup' while in recovery
>Aug 19 12:42:30 NFSConnector01 ctdbd[6336]: Unable to launch startup event script
>Aug 19 12:42:32 NFSConnector01 ctdbd[6336]: Recovery has started
>Aug 19 12:42:32 NFSConnector01 ctdbd[6336]: Freeze db: ctdb.tdb
>Aug 19 12:42:32 NFSConnector01 ctdbd[6336]: Thaw db: ctdb.tdb generation 44514481
>Aug 19 12:42:32 NFSConnector01 ctdbd[6336]: Release freeze handle for db ctdb.tdb
>Aug 19 12:42:32 NFSConnector01 ctdbd[6336]: Recovery mode set to NORMAL
>Aug 19 12:42:32 NFSConnector01 ctdbd[6336]: Recovery has finished
>Aug 19 12:42:32 NFSConnector01 ctdb-recoverd[6512]: Disabling takeover runs for 60 seconds
>Aug 19 12:42:32 NFSConnector01 ctdb-recoverd[6512]: Reenabling takeover runs
>Aug 19 12:42:33 NFSConnector01 ctdb-recoverd[6512]: Disabling takeover runs for 60 seconds
>Aug 19 12:42:33 NFSConnector01 ctdb-recoverd[6512]: Reenabling takeover runs
>Aug 19 12:42:35 NFSConnector01 ctdbd[6336]: Running the "startup" event.
>Aug 19 12:42:38 NFSConnector01 ctdb-recoverd[6512]: ../../ctdb/server/ctdb_client.c:623 reqid 2147483574 not found
>Aug 19 12:42:43 NFSConnector01 ctdb-eventd[6338]: 08.scality-sfused: Redirecting to /bin/systemctl restart scality-sfused.service
>Aug 19 12:42:43 NFSConnector01 ctdb-eventd[6338]: 08.scality-sfused: Job for scality-sfused.service failed because the control process exited with error code. See "systemctl status scality-sfused.service" and "journalctl -xe" for details.
>Aug 19 12:42:43 NFSConnector01 ctdb-eventd[6338]: 08.scality-sfused: Failed to restart scality-sfused
>Aug 19 12:42:43 NFSConnector01 ctdb-eventd[6338]: startup event failed
>Aug 19 12:42:43 NFSConnector01 ctdbd[6336]: startup event failed
>Aug 19 12:42:43 NFSConnector01 ctdb-eventd[6338]: talloc: access after free error - first free may be at ../../ctdb/event/event_context.c:144
>Aug 19 12:42:43 NFSConnector01 ctdb-eventd[6338]: Bad talloc magic value - access after free
>Aug 19 12:42:43 NFSConnector01 ctdbd[6336]: Eventd went away - exiting
>Aug 19 12:42:43 NFSConnector01 ctdbd[6336]: CTDB daemon shutting down
>Aug 19 12:42:44 NFSConnector01 ctdb-recoverd[6512]: Daemon has exited - shutting down client

Actions: View

Attachments on bug 14475: 16191 | 16609 | 16610