Bug 11232 - memory leak in the ldap process
memory leak in the ldap process
Status: ASSIGNED
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB
4.1.17
x64 All
: P5 major
: ---
Assigned To: Andrew Bartlett
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-04-23 15:13 UTC by Jan Dvořák
Modified: 2015-07-31 20:49 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jan Dvořák 2015-04-23 15:13:54 UTC
We get frequent OOM kills of the Samba's LDAP server process on our new AD server.

The version is samba-4.1.17, system is rhel7. Build options are the same as in the Fedora package, but with enabled DC and builtin kerberos. System has 8G RAM and 1G swap.

Today I managed to `gdb -p` the process in time and thus have a core dump (8G, compressed to 1G), but I have no idea if that could be of any use to you. Please advise on how to debug this.
Comment 1 Jan Dvořák 2015-04-23 15:15:23 UTC
Oh, I forgot to add: this is for the National Library of Technology, Prague, Czech Republic. We have about 60k user accounts and the database is approximately 1G in size.
Comment 2 Andrew Bartlett 2015-04-23 21:07:21 UTC
If you can reproduce on git master, then (finally!) we have the ability to report the talloc pool usage with:

smbcontrol $PID pool-usage

That would be the single fastest way to find and fix this.

Without being able to do that, the other option is to get into the big process with gdb, and run talloc_report_full().

This link should give you some suggestions on how to run that (but smbcontrol is much easier, if you can):

http://samba.2283325.n4.nabble.com/Possible-samba4-winbind-memory-leak-td4646127.html
Comment 3 Jan Dvořák 2015-05-04 13:44:28 UTC
(In reply to Andrew Bartlett from comment #2)
I've tried several times, but I would literally have to sit in front of htop whole day to catch this one in time. It exhausts main memory in something like 30 seconds and fills the 1G swap soon after.

Would it be possible to hack up some very simple tripwire that would check memory usage on every allocation and abort() with full dump when it hits ~7G? I would then attach gdb in screen and leave it running.

(I've tried just that via gdb + setrlimit now. Let's see where that gets us.)
Comment 4 Jan Dvořák 2015-05-05 13:12:41 UTC
I have failed to reproduce it with the setrlimit method mentioned above. It instead slowly went to crawl. In any case, at the point I had to restart it, memory situation looked something like this:

full talloc report: 1M in 27042 blocks
actual heap usage: 4G
mmapped files: ~1G

Any ideas?
Comment 5 Andrew Bartlett 2015-06-03 00:36:09 UTC
Just touching base on this again.  If we have the core dump, could we run the talloc report on that, re-animated under gdb?  (Well away from your production server, but with the same binaries)

It seems to me that the memory may not be allocated with talloc, it might be something else (eg malloc memory).  

Are you able to get a network trace when it blows up, are there any other clues as to what triggers this?

Thanks,

Andrew Bartlett
Comment 6 Jan Dvořák 2015-06-03 07:51:53 UTC
(In reply to Andrew Bartlett from comment #5)
> Just touching base on this again.  If we have the core dump, could we run the talloc report on that, re-animated under gdb?

AFAIK you can't really do that.

> It seems to me that the memory may not be allocated with talloc, it might be something else (eg malloc memory).  

Well, since talloc report clearly states *much* lower usage than what is observed, that would be my first guess too.

> Are you able to get a network trace when it blows up, are there any other clues as to what triggers this?

Well, we use LSC to sync users from our primary LDAP to Samba DC for the library terminal solution. That's somewhere around 65k accounts, but we do restrict that to changed ones only, which yields around 1k daily. It seems that samba simply slowly accumulates leaked memory and then dies.

Since then, we mitigated the problem by not syncing all accounts, but only around 13k active ones, which is not ideal, but it basically fixed the problem for us.

For debugging / testing I would suggest just creating some script to create a lot of samba users via LDAP and then update them in cycles until samba dies. Alas, I don't have time necessary to write it myself. You could also use valgrind that way.