Bug 10779 - pthreadpool code can create too many threads under load.
Summary: pthreadpool code can create too many threads under load.
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: unspecified
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
Depends on:
Reported: 2014-08-22 22:29 UTC by Jeremy Allison
Modified: 2014-12-26 16:11 UTC (History)
1 user (show)

See Also:

git-am fix for 4.1.next and 4.0.next. (2.07 KB, patch)
2014-08-25 19:31 UTC, Jeremy Allison
jra: review? (metze)
jra: review? (ddiss)
ira: review+

Note You need to log in before you can comment on or make changes to this bug.
Description Jeremy Allison 2014-08-22 22:29:45 UTC
From Volker's patch I've submitted to master:

Using the new msg_source program with 1.500 instances against a single
msg_sink I found the msg_source process to spawn two worker threads for
synchronously sending the data towards the receiving socket. This should
not happen: Per destination node we only create one queue. We strictly
only add pthreadpool jobs one after the other, so a single helper thread
should be perfectly sufficient.

It turned out that under heavy overload the main sending thread was
scheduled before the thread that just had finished its send() job. So
the helper thread was not able to increment the pool->num_idle variable
indicating that we don't have to create a new thread when the new job
is added.

This patch moves the signalling write under the mutex. This means that
indicating readiness via the pipe and the pool->num_idle variable happen both
under the same mutex lock and thus are atomic. No superfluous threads anymore.


Patch for 4.1.next and 4.0.next to follow once it hits master.
Comment 1 Jeremy Allison 2014-08-25 19:31:09 UTC
Created attachment 10226 [details]
git-am fix for 4.1.next and 4.0.next.

Back-port of Volker's fix that went into master.
Comment 2 Jeremy Allison 2014-10-10 00:35:29 UTC
Comment on attachment 10226 [details]
git-am fix for 4.1.next and 4.0.next.

Someone please review this... :-).
Comment 3 Jeremy Allison 2014-10-10 15:38:02 UTC
Re-assigning to Karolin for inclusion in 4.1.next, 4.0.next.
Comment 4 Karolin Seeger 2014-10-13 19:18:00 UTC
Pushed to autobuild-v4-[0|1]-test.
Comment 5 Karolin Seeger 2014-10-26 20:29:08 UTC
Pushed to both branches.
Closing out bug report.