Bug 11124 - CTDB daemon is non responsive and consuming 100% CPU
Summary: CTDB daemon is non responsive and consuming 100% CPU
Alias: None
Product: CTDB 2.5.x or older
Classification: Unclassified
Component: ctdb (show other bugs)
Version: 4.2.0
Hardware: All Linux
: P5 major
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
Depends on:
Blocks: 10077
  Show dependency treegraph
Reported: 2015-02-27 04:16 UTC by Amitay Isaacs
Modified: 2015-03-02 20:10 UTC (History)
2 users (show)

See Also:

Patch for 4.2 (2.40 KB, patch)
2015-02-27 04:21 UTC, Amitay Isaacs
martins: review+

Note You need to log in before you can comment on or make changes to this bug.
Description Amitay Isaacs 2015-02-27 04:16:06 UTC
Clustered samba is configured on virtual machines with a single CPU.  When running SMB load, ctdb becomes unresonsive and starts consuming 100% of CPU.

Here is the stack trace of the process:

#0  0x00007f763794d1e0 in __write_nocancel () from /lib64/libpthread.so.0
#1  0x00007f7637dc92e6 in sys_write (fd=25, buf=buf@entry=0x7f7638ec9930, count=count@entry=48) at ../ctdb/common/system_util.c:403
#2  0x00007f7637dd4a68 in ctdb_queue_send (queue=0x7f7638bdb5d0, data=0x7f7638ec9930 "0", length=<optimized out>) at ../ctdb/common/ctdb_io.c:313
#3  0x00007f7637d927c8 in daemon_call_from_client_callback (state=<optimized out>) at ../ctdb/server/ctdb_daemon.c:358
#4  0x00007f7637da9339 in ctdb_become_dmaster (ctdb_db=ctdb_db@entry=0x7f7638c350d0, key=..., data=..., rsn=<optimized out>, 
    record_flags=record_flags@entry=0, hdr=0x7f7638e734b0, hdr=0x7f7638e734b0) at ../ctdb/server/ctdb_call.c:408
#5  0x00007f7637da9a23 in ctdb_request_dmaster (ctdb=ctdb@entry=0x7f7638bbaba0, hdr=hdr@entry=0x7f7638e734b0) at ../ctdb/server/ctdb_call.c:638
#6  0x00007f7637da4089 in ctdb_input_pkt (ctdb=0x7f7638bbaba0, hdr=0x7f7638e734b0) at ../ctdb/server/ctdb_server.c:347
#7  0x00007f7637dd4714 in queue_io_read (queue=0x7f7638bdb9b0) at ../ctdb/common/ctdb_io.c:209
#8  queue_io_handler (ev=<optimized out>, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../ctdb/common/ctdb_io.c:279
#9  0x00007f76370ee2bb in epoll_event_loop (tvalp=0x7ffff73f1170, epoll_ev=0x7f7638bcac90) at ../lib/tevent/tevent_epoll.c:728
#10 epoll_event_loop_once (ev=<optimized out>, location=<optimized out>) at ../lib/tevent/tevent_epoll.c:926
#11 0x00007f76370ec787 in std_event_loop_once (ev=0x7f7638bcaa50, location=0x7f7637ddd360 "../ctdb/server/ctdb_daemon.c:1325")
    at ../lib/tevent/tevent_standard.c:114
#12 0x00007f76370e8f5d in _tevent_loop_once (ev=ev@entry=0x7f7638bcaa50, location=location@entry=0x7f7637ddd360 "../ctdb/server/ctdb_daemon.c:1325")
    at ../lib/tevent/tevent.c:530
#13 0x00007f76370e90fb in tevent_common_loop_wait (ev=0x7f7638bcaa50, location=0x7f7637ddd360 "../ctdb/server/ctdb_daemon.c:1325")
    at ../lib/tevent/tevent.c:634
#14 0x00007f76370ec727 in std_event_loop_wait (ev=0x7f7638bcaa50, location=0x7f7637ddd360 "../ctdb/server/ctdb_daemon.c:1325")
    at ../lib/tevent/tevent_standard.c:140
#15 0x00007f7637d93f7b in ctdb_start_daemon (ctdb=ctdb@entry=0x7f7638bbaba0, do_fork=<optimized out>, use_syslog=<optimized out>)
    at ../ctdb/server/ctdb_daemon.c:1325
#16 0x00007f7637d8e4a8 in main (argc=<optimized out>, argv=<optimized out>) at ../ctdb/server/ctdbd.c:321
Comment 1 Amitay Isaacs 2015-02-27 04:21:03 UTC
Created attachment 10800 [details]
Patch for 4.2
Comment 2 Amitay Isaacs 2015-02-27 06:17:02 UTC
Hi Karo,

Here's another ctdb defect for 4.2.

Comment 3 Karolin Seeger 2015-03-01 20:17:10 UTC
Pushed to autobuild-v4-2-test.
Comment 4 Karolin Seeger 2015-03-02 20:10:09 UTC
Pushed to v4-2-test.
Closing out bug report.