Bug 14399 - Server RAM filling up when writing to share from macOS
Summary: Server RAM filling up when writing to share from macOS
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.12.3
Hardware: All All
: P5 major (vote)
Target Milestone: ---
Assignee: Andreas Schneider
QA Contact: Samba QA Contact
Depends on:
Reported: 2020-06-04 14:31 UTC by Marius Steffen
Modified: 2020-06-08 13:59 UTC (History)
4 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Marius Steffen 2020-06-04 14:31:18 UTC

I noticed an absolutely work-breaking behavior in Samba 4.12 (not present in 4.11 or lower): when writing to a share (Arch Linux server) from macOS, smbd’s RAM usage is going up pretty much exactly the size of the transferred files - and is released *only* when the connection is completely closed. So obviously, when keeping the share mounted, copying files finally leads to Samba eating up all of my server’s memory, finally crashing.

After some fiddling, I discovered this happens whenever either ’smb encrypt’, ’server signing’ or SMB2 are forced/required by the server. Again: this only happens with Samba 4.12.

After git bisecting, I've come up with the comment that introduced this bug:

commit 4a24d9499757dea377b4e3d8beb7f2c10fd5c5d0
Author: Andreas Schneider <asn@samba.org>
Date:   Fri Aug 23 09:12:21 2019 +0200

    libcli:smb: Use gnutls_aead_cipher_decryptv2() for AES GCM or CCM
    This is a new call which has been added with GnuTLS 3.6.10 and will
    recuduce memory allocations and copying of data.
    Signed-off-by: Andreas Schneider <asn@samba.org>
    Reviewed-by: Simo Sorce <idra@samba.org>
    Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
    Autobuild-Date(master): Tue Oct  8 14:12:44 UTC 2019 on sn-devel-184

 libcli/smb/smb2_signing.c | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

Of course, this renders my Samba shares completely unusable - any idea what to do?

Here’s my testparm output:

Load smb config files from /etc/samba/smb.conf
Loaded services file OK.

Press enter to see a dump of your service definitions

# Global parameters
	log file = /var/log/samba/%m.log
	map to guest = Bad User
	security = USER
	server string = marius-arch Samba Server
	idmap config * : backend = tdb
	smb encrypt = required

	path = /home/marius
	read only = No
	valid users = marius

Best regards,
Marius Steffen
Comment 1 Andreas Schneider 2020-06-04 14:48:29 UTC
Which GnuTLS version do you have installed?
Comment 2 Andreas Schneider 2020-06-04 15:15:59 UTC
I'm not able to reproduce this:

-rw-rw-rw- 1 asn asn-group 4.9G Jun  4 15:07 file.txt

bin/smbclient //$SERVER/tmp -U$USERNAME%$PASSWORD -e -m SMB3 -c "put file.txt"
putting file file.txt as \file.txt (317241.2 kb/s) (average 317241.2 kb/s)

Could you please install debuginfo packages for GnuTLS and run smbd with valgrind?
Comment 3 Marius Steffen 2020-06-04 15:44:07 UTC
I've got gnutls 3.6.13 installed, but also tried their git master and this patch (https://gitlab.com/gnutls/gnutls/-/merge_requests/1274) - without any success changes to the problem.

I'll make a gnutls debug build - could you tell me how to valgrind smbd? Not sure if I'd do it the right way :)

Best regards
Comment 4 Andreas Schneider 2020-06-04 16:08:51 UTC
Ok, the issue can be reproduced with AES-CCM.

Which client do you use?
Comment 5 Andreas Schneider 2020-06-04 16:13:27 UTC
There are two ciphers available: AES-GCM and AES-CCM. It works just fine with AES-GCM, but there is an issue with AES-CCM filling up the memory. It is a GnuTLS bug.
Comment 6 Marius Steffen 2020-06-04 16:57:02 UTC
I'm using macOS as client - maybe that's the reason I couldn't reproduce the issue with my Linux client.
Comment 7 Marius Steffen 2020-06-04 19:38:33 UTC
Is there any way to deactivate AES-CCM via smb.conf for the time being?
Comment 8 Marius Steffen 2020-06-05 12:35:23 UTC
Okay, I've filed a bug report with gnutls, and had a friend find the bug and fix it. Hopefully they realize the potential severity and deploy a patch release ASAP.
Comment 9 Andreas Schneider 2020-06-05 13:56:31 UTC
Comment 10 Andreas Schneider 2020-06-08 04:21:21 UTC
This has been addressed by https://gitlab.com/gnutls/gnutls/-/merge_requests/1278
Comment 11 Björn Jacke 2020-06-08 13:59:14 UTC
I filed bug reports for Debian, SUSE and Red Hat:


Let's hope that they will all fix this as soon as possible. I'm closing this bug here as FIXES for us as there's nothing else that samba can do here.