Bug 13029 - CTDB consumes lots of memory with record contention workload
Summary: CTDB consumes lots of memory with record contention workload
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: CTDB (show other bugs)
Version: 4.6.7
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-09-13 01:21 UTC by Amitay Isaacs
Modified: 2017-09-17 16:54 UTC (History)
3 users (show)

See Also:


Attachments
Patches for v4-6 (1.30 KB, patch)
2017-09-15 01:16 UTC, Amitay Isaacs
martins: review+
Details
Patches for v4-7 (1.30 KB, patch)
2017-09-15 01:16 UTC, Amitay Isaacs
martins: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Amitay Isaacs 2017-09-13 01:21:28 UTC
If smb workload comprises of multiple clients trying to access the same file, then it gives rise to record contention on locking.tdb.  As a result the record size can become quite large in the database.

When multiple clients send (call) requests to CTDB to migrate the same record, CTDB queues up the migration requests and keeps a single call request in flight.  All the other call requests get queued and deferred waiting for the migration of the record to complete.  Once the record is migrated to the local node, all the deferred call requests are processed again.

During the initial processing of a call request, CTDB fetches the record to check the header.  If the call request gets deferred, then the fetched record data is not freed.  For a contended record this can result in memory usage in GB.

The fix is to free up the record data if a call request gets deferred.
Comment 1 Amitay Isaacs 2017-09-15 01:16:26 UTC
Created attachment 13602 [details]
Patches for v4-6
Comment 2 Amitay Isaacs 2017-09-15 01:16:46 UTC
Created attachment 13603 [details]
Patches for v4-7
Comment 3 Martin Schwenke 2017-09-15 01:46:44 UTC
Hi Karolin,

This is ready for 4.6 and 4.7.

Thanks...
Comment 4 Stefan Metzmacher 2017-09-17 16:54:15 UTC
Pushed to v4-{6,7}-test