I have 2 64 bit samba 4.1.5-7 installations on different sites (not linked) running in AD mode using the packages from Sernet. Both installed onto CentOS 6.5 fully updated as of 24/2/2014. The servers don't do any file transfers, have only one machine logging into the domain for admin purposes. Their sole purpose at the moment is authentication for our Zarafa mail servers. Their memory usage increases throughout the day on one server (8GB RAM) by as much as 4GB per day. I have to killall samba each night. There seems to be a memory leak somewhere but I am not an expert in diagnosing this problem.
Upgraded to 4.1.7 and the problem persists.
A first try might be the output of smbcontrol <pid> pool-usage for such a big process. It might be large, so you might compress it before sending it to us.
I get "No replies received"
Is your target process busy in the meantime? If so, you might want to increase the timeout with -t <something>
Increased the log level to 3 but nothing helpful... smbcontrol 18357 pool-usage -t 10 Registered MSG_REQ_POOL_USAGE Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED No replies received smbcontrol 18357 pool-usage -t 100 Registered MSG_REQ_POOL_USAGE Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED No replies received
The server is virtually idle Cpu(s): 0.2%us, 0.2%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3829144k total, 3503692k used, 325452k free, 342760k buffers Swap: 3964920k total, 85752k used, 3879168k free, 1465664k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18357 root 20 0 816m 302m 3508 S 0.0 8.1 14:03.88 samba
Crap, sorry, my fault. smbcontrol is using the wrong messaging mechanism for "samba". Need to find out whether this is possible for the AD DC.
Andrew, do you have an idea how to get the talloc hierarchy out of the AD controller at runtime?
Any more on this? Thanks.
If you run samba with --leak-report-full then on exit, it will print out the full remaining talloc tree when you kill it. This can be a very, very large amount of potentially sensitive data (depending on what is leaked). The best way we have found to chase this down is to use XZ compression: tar --xz -cf logs.tar.xz samba.log You can then attach that as a private attachment here, or for greater security gpg encrypt it to my GPG key below: sec 4096R/C8021865 2012-07-04 [expires: 2018-07-03] Key fingerprint = 8160 9BF8 5375 BA5E 510C CEA1 FE00 1D44 C802 1865 uid Andrew Bartlett <abartlet@abartlet.net> uid Andrew Bartlett <abartlet@samba.org> uid Andrew Bartlett <abartlet@ozlabs.org> uid Andrew Bartlett <abartlet@catalyst.net.nz> ssb 4096R/D899268D 2012-07-04
Send you the log via email.
Please fix this, or I can't help you. Sorry, The mail system <mike@scholes-software.com>: host mail.scholes-software.com[87.81.241.169] said: 550-This mail has been classified as spam and has not been delivered, if you 550-believe this is an error please seek an alternative method of communication 550 and inform the recipient of the problem. (in reply to end of DATA command)
OK I've whitelisted you. Thanks. Mike.
Hi, any luck? I haven't received any emails yet.
Honestly, after you bounced mail from me an Ricky who I asked to help you, we both gave up. I'll try and look again after SambaXP, in between I need to pull some rabbits out of the hat.
That's OK, restarting every night is working and we don't have any immediate plan for 24 hour working.
Hi Any chance for another look at this?
(In reply to comment #17) > Hi > > Any chance for another look at this? Feel free to send it to me.
Created attachment 9993 [details] Results from memory script
I've attached the results of the script that Ricky sent me. This has been running on my home server with only one user (me) using it to authenticate my Zarafa mail server. I have also joined another Samba AD to our main server in work, same OS and it doesn't suffer from the memory leak. No authentication is directed at that machine yet. So I'm still thinking it's a winbind leak.
Created attachment 9995 [details] Results From Work
Also added results for a day and a half from our work server. This has 60 users authenticating from our Zarafa server and again no files transfers.
Thinking about this again the only authentication we are doing is using ldap/AD methods so perhaps nothing to do with winbind.
Any thoughts?
Has this one been abandoned now because the bug remains?
(In reply to comment #25) > Has this one been abandoned now because the bug remains? Well, you might want to talk to whoever sent you the script. This contains no useful information where the memory leak might be. And as according to the history in this bug it is really difficult to get decent logs, yes, this has been pretty much abandoned. If you get us the logs that Andrew requested, we might say more. Alternatively, you might contact Andrew Bartlett directly for help.
Andrew was sent the logs.
(In reply to comment #27) > Andrew was sent the logs. Then it's pretty much upon Andrew I believe. Sorry for the confusion.
The logs as sent sadly don't have any useful information, except to cofirm the process size (we need to know *where* it is). Try getting the leaky process under gdb and running: p talloc_report_full(0, stderr) Andrew Bartlett
(In reply to comment #29) > The logs as sent sadly don't have any useful information, except to cofirm the > process size (we need to know *where* it is). > > Try getting the leaky process under gdb and running: > > p talloc_report_full(0, stderr) > > Andrew Bartlett Andrew, maybe this is the time to add the smbcontrol pool-usage thingy to the "samba" program? That's much less intrusive and easier to use for users. Volker
Thanks. Volker, I think you are right, I wouldn't know how to do what Andrew asked for. I'm just a regular IT Admin. Mike
*** Bug 10519 has been marked as a duplicate of this bug. ***
(In reply to comment #32) > *** Bug 10519 has been marked as a duplicate of this bug. *** I think I've found the bug, the fix is on its way to master.
Created attachment 10117 [details] Patch for v4-1-test
Created attachment 10118 [details] Patch for v4-0-test
Pushed to autobuild-v4-[0|1]-test.
(In reply to comment #36) > Pushed to autobuild-v4-[0|1]-test. Pushed to both branches. Closing out bug report. Thanks!
Just installed 4.1.11 and it has made no difference, the memory leak is still present.
Is there anything in the latest version I can do to help diagnose this problem?
Well, you might want to find someone who can log into your machine and do the required analysis by printing the full talloc hierarchy with gdb.
Unfortunately I don't know anyone that can do this. Anyone there can point to instructions?
(In reply to comment #41) > Unfortunately I don't know anyone that can do this. Anyone there can point to > instructions? Andrew has sent instructions under comment 29. Volker
p talloc_report_full(0, stderr) No symbol "talloc_report_full" in current context.
Thanks to Volker's messaging work smbcontrol <pid> pool-usage now works against the samba process. If this still happens with Samba 4.5.0rc1 or any Samba version since 4.3, then please run that command and give us the output. In the meantime, I'm going to mark this as NEEDINFO, because we can't really go any further with what we have. Sorry,
I'm using 4.2.13-SerNet-RedHat-22.el7 as 4.3 became a paid for product. I am not working for the company any more. smbcontrol <pid> pool-usage produces no output regardless of the pid I enter. I have recently migrated all my servers to xenserver and am now running samba on CentOS 7. I am not seeing any memory leak on this system even though I have the same systems authenticating against Samba. For me the problem has gone away
Thanks. Closing as fixed (for want of a a better tag) as you are unable to still reproduce.