I noticed an absolutely work-breaking behavior in Samba 4.12 (not present in 4.11 or lower): when writing to a share (Arch Linux server) from macOS, smbd’s RAM usage is going up pretty much exactly the size of the transferred files - and is released *only* when the connection is completely closed. So obviously, when keeping the share mounted, copying files finally leads to Samba eating up all of my server’s memory, finally crashing.
After some fiddling, I discovered this happens whenever either ’smb encrypt’, ’server signing’ or SMB2 are forced/required by the server. Again: this only happens with Samba 4.12.
After git bisecting, I've come up with the comment that introduced this bug:
Author: Andreas Schneider <email@example.com>
Date: Fri Aug 23 09:12:21 2019 +0200
libcli:smb: Use gnutls_aead_cipher_decryptv2() for AES GCM or CCM
This is a new call which has been added with GnuTLS 3.6.10 and will
recuduce memory allocations and copying of data.
Signed-off-by: Andreas Schneider <firstname.lastname@example.org>
Reviewed-by: Simo Sorce <email@example.com>
Autobuild-User(master): Andreas Schneider <firstname.lastname@example.org>
Autobuild-Date(master): Tue Oct 8 14:12:44 UTC 2019 on sn-devel-184
libcli/smb/smb2_signing.c | 29 +++++++++++++++++++++++++++--
1 file changed, 27 insertions(+), 2 deletions(-)
Of course, this renders my Samba shares completely unusable - any idea what to do?
Here’s my testparm output:
Load smb config files from /etc/samba/smb.conf
Loaded services file OK.
Server role: ROLE_STANDALONE
Press enter to see a dump of your service definitions
# Global parameters
log file = /var/log/samba/%m.log
map to guest = Bad User
security = USER
server string = marius-arch Samba Server
idmap config * : backend = tdb
smb encrypt = required
path = /home/marius
read only = No
valid users = marius
Which GnuTLS version do you have installed?
I'm not able to reproduce this:
-rw-rw-rw- 1 asn asn-group 4.9G Jun 4 15:07 file.txt
bin/smbclient //$SERVER/tmp -U$USERNAME%$PASSWORD -e -m SMB3 -c "put file.txt"
putting file file.txt as \file.txt (317241.2 kb/s) (average 317241.2 kb/s)
Could you please install debuginfo packages for GnuTLS and run smbd with valgrind?
I've got gnutls 3.6.13 installed, but also tried their git master and this patch (https://gitlab.com/gnutls/gnutls/-/merge_requests/1274) - without any success changes to the problem.
I'll make a gnutls debug build - could you tell me how to valgrind smbd? Not sure if I'd do it the right way :)
Ok, the issue can be reproduced with AES-CCM.
Which client do you use?
There are two ciphers available: AES-GCM and AES-CCM. It works just fine with AES-GCM, but there is an issue with AES-CCM filling up the memory. It is a GnuTLS bug.
I'm using macOS as client - maybe that's the reason I couldn't reproduce the issue with my Linux client.
Is there any way to deactivate AES-CCM via smb.conf for the time being?
Okay, I've filed a bug report with gnutls, and had a friend find the bug and fix it. Hopefully they realize the potential severity and deploy a patch release ASAP.
This has been addressed by https://gitlab.com/gnutls/gnutls/-/merge_requests/1278
I filed bug reports for Debian, SUSE and Red Hat:
Let's hope that they will all fix this as soon as possible. I'm closing this bug here as FIXES for us as there's nothing else that samba can do here.