Hi, I noticed an absolutely work-breaking behavior in Samba 4.12 (not present in 4.11 or lower): when writing to a share (Arch Linux server) from macOS, smbd’s RAM usage is going up pretty much exactly the size of the transferred files - and is released *only* when the connection is completely closed. So obviously, when keeping the share mounted, copying files finally leads to Samba eating up all of my server’s memory, finally crashing. After some fiddling, I discovered this happens whenever either ’smb encrypt’, ’server signing’ or SMB2 are forced/required by the server. Again: this only happens with Samba 4.12. After git bisecting, I've come up with the comment that introduced this bug: commit 4a24d9499757dea377b4e3d8beb7f2c10fd5c5d0 Author: Andreas Schneider <asn@samba.org> Date: Fri Aug 23 09:12:21 2019 +0200 libcli:smb: Use gnutls_aead_cipher_decryptv2() for AES GCM or CCM This is a new call which has been added with GnuTLS 3.6.10 and will recuduce memory allocations and copying of data. Signed-off-by: Andreas Schneider <asn@samba.org> Reviewed-by: Simo Sorce <idra@samba.org> Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org> Autobuild-Date(master): Tue Oct 8 14:12:44 UTC 2019 on sn-devel-184 libcli/smb/smb2_signing.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) Of course, this renders my Samba shares completely unusable - any idea what to do? Here’s my testparm output: Load smb config files from /etc/samba/smb.conf Loaded services file OK. Server role: ROLE_STANDALONE Press enter to see a dump of your service definitions # Global parameters [global] log file = /var/log/samba/%m.log map to guest = Bad User security = USER server string = marius-arch Samba Server idmap config * : backend = tdb smb encrypt = required [marius] path = /home/marius read only = No valid users = marius Best regards, Marius Steffen
Which GnuTLS version do you have installed?
I'm not able to reproduce this: -rw-rw-rw- 1 asn asn-group 4.9G Jun 4 15:07 file.txt bin/smbclient //$SERVER/tmp -U$USERNAME%$PASSWORD -e -m SMB3 -c "put file.txt" putting file file.txt as \file.txt (317241.2 kb/s) (average 317241.2 kb/s) Could you please install debuginfo packages for GnuTLS and run smbd with valgrind?
I've got gnutls 3.6.13 installed, but also tried their git master and this patch (https://gitlab.com/gnutls/gnutls/-/merge_requests/1274) - without any success changes to the problem. I'll make a gnutls debug build - could you tell me how to valgrind smbd? Not sure if I'd do it the right way :) Best regards Marius
Ok, the issue can be reproduced with AES-CCM. Which client do you use?
There are two ciphers available: AES-GCM and AES-CCM. It works just fine with AES-GCM, but there is an issue with AES-CCM filling up the memory. It is a GnuTLS bug.
I'm using macOS as client - maybe that's the reason I couldn't reproduce the issue with my Linux client.
Is there any way to deactivate AES-CCM via smb.conf for the time being?
Okay, I've filed a bug report with gnutls, and had a friend find the bug and fix it. Hopefully they realize the potential severity and deploy a patch release ASAP.
https://gitlab.com/gnutls/gnutls/-/merge_requests/1277
This has been addressed by https://gitlab.com/gnutls/gnutls/-/merge_requests/1278
I filed bug reports for Debian, SUSE and Red Hat: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962467 https://bugzilla.opensuse.org/show_bug.cgi?id=1172663 https://bugzilla.redhat.com/show_bug.cgi?id=1845083 Let's hope that they will all fix this as soon as possible. I'm closing this bug here as FIXES for us as there's nothing else that samba can do here.
I note that both - there is no new GnuTLS release yet and - Ubuntu also needs a bug filed Furthermore, it would be good if we could detect at configure time and avoid using these routines where the system GnuTLS is unpatched, but as Ubuntu 18.04 appears (if I read version numbers correctly) to have the bug we wouldn't be testing the new code anywhere... A difficult issue.
if i remember correct, the problematic code came in with 3.6.10, Ubuntu 18.04 has an older version but Ubuntu 20.04 has 3.6.13 and needs a fix. I was expecting that they monitor Debian bug reports close enough. Seems like they don't. I'm not able to use the Launchpad tool correctly, maybe someone else wants to point them on the problem?
(In reply to Björn Jacke from comment #13) Thanks for re-checking the version numbers on Ubuntu 18.04 for me, I think I got lost in the sea of version triplets. However, I'm wondering if it is worth working around this on the Samba side. The MR at https://gitlab.com/gnutls/gnutls/-/merge_requests/1278 says: > However, there was a bug in the functions used for the serialization, which > causes memory leaks under a certain condition (i.e. the number of input buffers > is 1). Given widespread patching it looking difficult to achieve, should we just include a useless second (perhaps even empty) element in the IOV to avoid this issue Samba-side in the meantime?
Steve, can you please take care that Canonical fixes this in Ubuntu 20.04 and possibly other affected non LTS releases?
This bug was referenced in samba master: 4d63a1a79f372d4be6633bb1053a1934629da1df 94808cc50e4350a8c3bc250a886e8d4e7802dd12
Created attachment 16224 [details] patch for 4.12
Created attachment 16225 [details] patch for 4.13
Comment on attachment 16224 [details] patch for 4.12 We should also regenerate the bootstrap image so GitLab CI on 4.12 can continue to test this. But this is a start.
Comment on attachment 16225 [details] patch for 4.13 Likewise here the bootstrap change should be brought over, but this is a start. Just backport the README.md fix and metze's GitLab note.
Assigning to Karolin, but we should do the bootstrap work.
Be aware that you probably need to backport Kerberos 1.18 support to 4.12 then ...
(In reply to Andrew Bartlett from comment #21) Pushed to autobuild-v4-{12,13}-test. Re-assigning to Andrew.
This bug was referenced in samba v4-12-test: 02ee82f6e4da19c801b7b4691804249b62b92166
This bug was referenced in samba v4-13-test: 47e446f3f7ccdd65f60dbbab72a1e3b81f650959
This bug was referenced in samba v4-13-stable: 47e446f3f7ccdd65f60dbbab72a1e3b81f650959
(In reply to Andreas Schneider from comment #22) I'm not sure I understand your comment here.
This bug was referenced in samba v4-12-stable (Release samba-4.12.8): 02ee82f6e4da19c801b7b4691804249b62b92166