It looks like there is a maximum limit of 65k open files per session. That limit seems to come from https://github.com/samba-team/samba/blob/master/source3/smbd/smbXsrv_open.c#L885 which gets its limit from https://github.com/samba-team/samba/blob/7e91ad4710898d6f14bcbf99955923ea305d2b78/source3/smbd/files.c#L1953. As for recent SMB protocols there is no reason to have this limit, and as when windows does the offline sync stuff that limit can get hit in production, I suggest to remove this limit for recent SMB versions.