Bug 14399 - Server RAM filling up when writing to share from macOS
Summary: Server RAM filling up when writing to share from macOS
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.12.3
Hardware: All All
: P5 major (vote)
Target Milestone: ---
Assignee: Andrew Bartlett
QA Contact: Samba QA Contact
Depends on:
Reported: 2020-06-04 14:31 UTC by Marius Steffen
Modified: 2021-02-25 19:05 UTC (History)
6 users (show)

See Also:

patch for 4.12 (1.30 KB, patch)
2020-09-11 14:43 UTC, Andreas Schneider
abartlet: review+
patch for 4.13 (1.26 KB, patch)
2020-09-11 14:44 UTC, Andreas Schneider
abartlet: review+

Note You need to log in before you can comment on or make changes to this bug.
Description Marius Steffen 2020-06-04 14:31:18 UTC

I noticed an absolutely work-breaking behavior in Samba 4.12 (not present in 4.11 or lower): when writing to a share (Arch Linux server) from macOS, smbd’s RAM usage is going up pretty much exactly the size of the transferred files - and is released *only* when the connection is completely closed. So obviously, when keeping the share mounted, copying files finally leads to Samba eating up all of my server’s memory, finally crashing.

After some fiddling, I discovered this happens whenever either ’smb encrypt’, ’server signing’ or SMB2 are forced/required by the server. Again: this only happens with Samba 4.12.

After git bisecting, I've come up with the comment that introduced this bug:

commit 4a24d9499757dea377b4e3d8beb7f2c10fd5c5d0
Author: Andreas Schneider <asn@samba.org>
Date:   Fri Aug 23 09:12:21 2019 +0200

    libcli:smb: Use gnutls_aead_cipher_decryptv2() for AES GCM or CCM
    This is a new call which has been added with GnuTLS 3.6.10 and will
    recuduce memory allocations and copying of data.
    Signed-off-by: Andreas Schneider <asn@samba.org>
    Reviewed-by: Simo Sorce <idra@samba.org>
    Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
    Autobuild-Date(master): Tue Oct  8 14:12:44 UTC 2019 on sn-devel-184

 libcli/smb/smb2_signing.c | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

Of course, this renders my Samba shares completely unusable - any idea what to do?

Here’s my testparm output:

Load smb config files from /etc/samba/smb.conf
Loaded services file OK.

Press enter to see a dump of your service definitions

# Global parameters
	log file = /var/log/samba/%m.log
	map to guest = Bad User
	security = USER
	server string = marius-arch Samba Server
	idmap config * : backend = tdb
	smb encrypt = required

	path = /home/marius
	read only = No
	valid users = marius

Best regards,
Marius Steffen
Comment 1 Andreas Schneider 2020-06-04 14:48:29 UTC
Which GnuTLS version do you have installed?
Comment 2 Andreas Schneider 2020-06-04 15:15:59 UTC
I'm not able to reproduce this:

-rw-rw-rw- 1 asn asn-group 4.9G Jun  4 15:07 file.txt

bin/smbclient //$SERVER/tmp -U$USERNAME%$PASSWORD -e -m SMB3 -c "put file.txt"
putting file file.txt as \file.txt (317241.2 kb/s) (average 317241.2 kb/s)

Could you please install debuginfo packages for GnuTLS and run smbd with valgrind?
Comment 3 Marius Steffen 2020-06-04 15:44:07 UTC
I've got gnutls 3.6.13 installed, but also tried their git master and this patch (https://gitlab.com/gnutls/gnutls/-/merge_requests/1274) - without any success changes to the problem.

I'll make a gnutls debug build - could you tell me how to valgrind smbd? Not sure if I'd do it the right way :)

Best regards
Comment 4 Andreas Schneider 2020-06-04 16:08:51 UTC
Ok, the issue can be reproduced with AES-CCM.

Which client do you use?
Comment 5 Andreas Schneider 2020-06-04 16:13:27 UTC
There are two ciphers available: AES-GCM and AES-CCM. It works just fine with AES-GCM, but there is an issue with AES-CCM filling up the memory. It is a GnuTLS bug.
Comment 6 Marius Steffen 2020-06-04 16:57:02 UTC
I'm using macOS as client - maybe that's the reason I couldn't reproduce the issue with my Linux client.
Comment 7 Marius Steffen 2020-06-04 19:38:33 UTC
Is there any way to deactivate AES-CCM via smb.conf for the time being?
Comment 8 Marius Steffen 2020-06-05 12:35:23 UTC
Okay, I've filed a bug report with gnutls, and had a friend find the bug and fix it. Hopefully they realize the potential severity and deploy a patch release ASAP.
Comment 9 Andreas Schneider 2020-06-05 13:56:31 UTC
Comment 10 Andreas Schneider 2020-06-08 04:21:21 UTC
This has been addressed by https://gitlab.com/gnutls/gnutls/-/merge_requests/1278
Comment 11 Björn Jacke 2020-06-08 13:59:14 UTC
I filed bug reports for Debian, SUSE and Red Hat:


Let's hope that they will all fix this as soon as possible. I'm closing this bug here as FIXES for us as there's nothing else that samba can do here.
Comment 12 Andrew Bartlett 2020-08-20 08:58:05 UTC
I note that both
 - there is no new GnuTLS release yet
 - Ubuntu also needs a bug filed

Furthermore, it would be good if we could detect at configure time and avoid using these routines where the system GnuTLS is unpatched, but as Ubuntu 18.04 appears (if I read version numbers correctly) to have the bug we wouldn't be testing the new code anywhere...

A difficult issue.
Comment 13 Björn Jacke 2020-08-20 13:53:31 UTC
if i remember correct, the problematic code came in with 3.6.10, Ubuntu 18.04 has an older version but Ubuntu 20.04 has 3.6.13 and needs a fix. I was expecting that they monitor Debian bug reports close enough. Seems like they don't. I'm not able to use the Launchpad tool correctly, maybe someone else wants to point them on the problem?
Comment 14 Andrew Bartlett 2020-08-23 22:49:17 UTC
(In reply to Björn Jacke from comment #13)
Thanks for re-checking the version numbers on Ubuntu 18.04 for me, I think I got lost in the sea of version triplets. 

However, I'm wondering if it is worth working around this on the Samba side. 

The MR at https://gitlab.com/gnutls/gnutls/-/merge_requests/1278 says:
> However, there was a bug in the functions used for the serialization, which
> causes memory leaks under a certain condition (i.e. the number of input buffers
> is 1).

Given widespread patching it looking difficult to achieve, should we just include a useless second (perhaps even empty) element in the IOV to avoid this issue Samba-side in the meantime?
Comment 15 Björn Jacke 2020-08-25 11:16:05 UTC
Steve, can you please take care that Canonical fixes this in Ubuntu 20.04 and possibly other affected non LTS releases?
Comment 16 Samba QA Contact 2020-09-11 08:28:05 UTC
This bug was referenced in samba master:

Comment 17 Andreas Schneider 2020-09-11 14:43:30 UTC
Created attachment 16224 [details]
patch for 4.12
Comment 18 Andreas Schneider 2020-09-11 14:44:53 UTC
Created attachment 16225 [details]
patch for 4.13
Comment 19 Andrew Bartlett 2020-09-11 18:46:20 UTC
Comment on attachment 16224 [details]
patch for 4.12

We should also regenerate the bootstrap image so GitLab CI on 4.12 can continue to test this.

But this is a start.
Comment 20 Andrew Bartlett 2020-09-11 18:50:57 UTC
Comment on attachment 16225 [details]
patch for 4.13

Likewise here the bootstrap change should be brought over, but this is a start.

Just backport the README.md fix and metze's GitLab note.
Comment 21 Andrew Bartlett 2020-09-11 18:51:42 UTC
Assigning to Karolin, but we should do the bootstrap work.
Comment 22 Andreas Schneider 2020-09-12 15:07:59 UTC
Be aware that you probably need to backport Kerberos 1.18 support to 4.12 then ...
Comment 23 Karolin Seeger 2020-09-14 10:04:17 UTC
(In reply to Andrew Bartlett from comment #21)
Pushed to autobuild-v4-{12,13}-test.
Re-assigning to Andrew.
Comment 24 Samba QA Contact 2020-09-14 12:09:05 UTC
This bug was referenced in samba v4-12-test:

Comment 25 Samba QA Contact 2020-09-16 08:30:22 UTC
This bug was referenced in samba v4-13-test:

Comment 26 Samba QA Contact 2020-09-18 08:33:53 UTC
This bug was referenced in samba v4-13-stable:

Comment 27 Andrew Bartlett 2020-09-18 19:01:09 UTC
(In reply to Andreas Schneider from comment #22)
I'm not sure I understand your comment here.
Comment 28 Samba QA Contact 2020-10-07 08:17:31 UTC
This bug was referenced in samba v4-12-stable (Release samba-4.12.8):