Bug 13625 - sys_get_nfs4_quota() crippling slow when sharing large NFS e.g. Amazon EFS
Summary: sys_get_nfs4_quota() crippling slow when sharing large NFS e.g. Amazon EFS
Status: CLOSED INVALID
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.6.2
Hardware: x86 Linux
: P5 normal (vote)
Target Milestone: ---
Assignee: Jeremy Allison
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-09-20 10:46 UTC by Tom Chiverton
Modified: 2019-09-19 19:03 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tom Chiverton 2018-09-20 10:46:31 UTC
Test case
1) Create an Amazon Linux AMI
2) Create an Amazon EFS 
3) Mount EFS and create a simple set of folders and files (e.g. ~10 top level folders, with a few folders in each, with a few small (<1k) text files scattered in them
4) df on the AMI will report 100% free space (EFS is ~8 exabytes in size, we only used a tiny tiny tiny fraction)
4) Share EFS mount via Samba to external - you'll need to tweak the Amazon network rules to allow of course.
5) Mount Samba from outside the Amazon VPC.

Windows and Linux will mount, but a simple `ls` of the top level folder will take minutes.

MacOS High Sierra will timeout and refuse to mount.

Workaround : Add a dfree command that reports all the space free immediately.

Ack : https://discussions.apple.com/message/33815554#33815554
Confirm : https://serverfault.com/questions/811957/any-way-to-speed-up-mapped-efs-share-in-windows/917680#917680
Comment 1 Jeremy Allison 2018-09-20 17:40:34 UTC
There is no call to sys_get_nfs4_quota() in the Samba code.

Why is this call needed ? Are you using Samba to re-export an NFS mount ?

Can you give more context please on how this is configured or setup ?
Comment 2 Björn Jacke 2018-09-20 20:49:44 UTC
> There is no call to sys_get_nfs4_quota() in the Samba code.
we have sys_get_nfs_quota in sysquota, I assume we talk about that.
> Why is this call needed ? Are you using Samba to re-export an NFS mount ?
As far as I understand that could provider offers a part of a huge pool
of NFS mounted storage and this is exported by the cloud customer via
SMB over the internet to local Windows boxes.

But if I understand this right, there is nothing that Samba does wrong
here. The NFS server is just a bit dumb that it takes so long to answer
that call. Did you already file a bug there for that performance issue
of the quota command?

You mentioned already the workaround with the dfree command for your
case to prevent samba from asking the slow NFS server for the quota
information. What do you expect what Samba sould do?
Comment 3 Tom Chiverton 2018-09-21 08:59:22 UTC
Sorry I wasn't clear.

I've not confirmed it's sys_get_ns4_quota or not - that's just on the linked comment.

What I *do* know is 'df -h' on the EC2 machine running Samba is super fast e.g.

$ time df -h|grep www
fs-XXXXX.efs.XX-XXXX-XXX.amazonaws.com:/wwwroot          8.0E   14G  8.0E   1% /wwwroot

real    0m0.009s
user    0m0.002s
sys     0m0.000s

But client's of the re-exported NFS share (over Samba) have terrible performance without the dfree workaround.

So I conclude Samba is doing something different to 'df' when it does whatever the dfree command replaces.
Comment 4 Björn Jacke 2019-09-18 10:31:22 UTC
free disk space and quota query are different things. If the NFS server is slow, we can't do much about this. This is not a bug.
Comment 5 Tom Chiverton 2019-09-19 11:04:29 UTC
I think you may be misunderstanding, the NFS server is not slow. 

`df -h` and any other operation on the volume returns *instantly*. It's an AWS EFS (which scales to terrabytes of storage) with a few hundred bytes in.

Samba, however, is super slow to mount. The work around of adding a dfree command to smb.conf confirms it's slow somewhere inside the gubbins of Samba. It must be figuring our the free space on the NFS volumne in a different way to what df is doing, and it's much less efficient.
Comment 6 Björn Jacke 2019-09-19 12:20:12 UTC
df is not querying quota but free disk space, this is what I wrote before.

Please do not reopen this bug report.
Comment 7 Tom Chiverton 2019-09-19 12:26:44 UTC
I'm aware df reports space, not quota. I learnton SunOS back in the .com boom days.

The issue certainly is not closed-fixed (closed invalid if anything) and I'm unsure why you keep mentioning quota. 

I said in a previous comment I wasn't sure it was quota related. 
I did say df is fast, but Samba is slow, and the man page for smb.conf says dfree (which people are using as the workaround) is for "where a problem occurs with the internal disk space calculations", which also isn't about quotas either.

In short, it appears to be a Samba disk space calculation that is at fault. 

If Samba's quota code was at fault, then changing from it's internal disk usage routines to an external command would make no difference, wouldn't it ?