We have been trying to understand why Samba versions before Nov 4th 2020 handle file transfers over 16 TB without error, but any later version after March 24th of 2021 of Samba releases in the various branches return an “NT_STATUS_INVALID_PARAMETER” error when crossing the 16 TB boundary. There have been a few random reports by users when searching Google for this issue, specifically TrueNAS, but no resolution or valid answer. We have two customers who cannot use any newer version of Samba in our FreeBSD image. It is not the version of FreeBSD (we are on 13.2) that matters, but the version of Samba. It is not specific to Windows or Linux clients, fails the same way. How can we get some verification from Samba on this issue? We came up with a quick test to verify the problem instead of transferring the entire file. We have tried different block sizes as well. It would seem we can’t be the only users running into this problem. How does Samba verify file transfers over 16 TiBs? We would be happy to work with someone to understand if there is a configuration issue or some other condition causing this error. dd if=/dev/zero bs=1024k of=/mnt/Source/copied_sparse_file_1 seek=16777214 count=4 Succeeds – 4.11.15 Sep 11 13:45:01 sm2u-31 smbd[9208]: Sep 11 13:45:01 sm2u-31 smbd[9208]: smb_set_filetime: modtime: Mon Sep 11 13:45:02 2023 Sep 11 13:45:01 sm2u-31 smbd[9208]: Sep 11 13:45:01 sm2u-31 smbd[9208]: smb_set_filetime: ctime: Thu Jan 1 00:00:00 1970 Sep 11 13:45:01 sm2u-31 smbd[9208]: Sep 11 13:45:01 sm2u-31 smbd[9208]: smb_set_file_time: createtime: Thu Jan 1 00:00:00 1970 Sep 11 13:45:01 sm2u-31 smbd[9208]: Sep 11 13:45:01 sm2u-31 smbd[9208]: smb_set_file_time: setting utimes to modified values. Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.565797, 4, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/sec_ctx.c:216(push_sec_ctx) Sep 11 13:45:01 sm2u-31 smbd[9208]: push_sec_ctx(500, 544) : sec_ctx_stack_ndx = 1 Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.565830, 4, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:576(push_conn_ctx) Sep 11 13:45:01 sm2u-31 smbd[9208]: push_conn_ctx(3461152069) : conn_ctx_stack_ndx = 0 Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.565864, 4, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/sec_ctx.c:320(set_sec_ctx_internal) Sep 11 13:45:01 sm2u-31 smbd[9208]: setting sec ctx (0, 0) - sec_ctx_stack_ndx = 1 Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.565898, 5, pid=9208, effective(500, 544), real(0, 0)] ../../libcli/security/security_token.c:53(security_token_debug) Sep 11 13:45:01 sm2u-31 smbd[9208]: Security token: (NULL) Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.565933, 5, pid=9208, effective(500, 544), real(0, 0)] ../../source3/auth/token_util.c:874(debug_unix_user_token) Sep 11 13:45:01 sm2u-31 smbd[9208]: UNIX token of user 0 Sep 11 13:45:01 sm2u-31 smbd[9208]: Primary group is 0 and contains 0 supplementary groups Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.566028, 4, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/sec_ctx.c:438(pop_sec_ctx) Sep 11 13:45:01 sm2u-31 smbd[9208]: pop_sec_ctx (500, 544) - sec_ctx_stack_ndx = 0 Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.566150, 2, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/close.c:813(close_normal_file) Sep 11 13:45:01 sm2u-31 smbd[9208]: Administrator closed file copied_sparse_file_1 (numopen=1) NT_STATUS_OK Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.566186, 5, pid=9208, effective(500, 544), real(0, 0), class=locking] ../../lib/dbwrap/dbwrap.c:133(dbwrap_lock_order_lock) Sep 11 13:45:01 sm2u-31 smbd[9208]: dbwrap_lock_order_lock: check lock order 1 for /var/db/samba4/smbXsrv_open_global.tdb Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.566382, 5, pid=9208, effective(500, 544), real(0, 0), class=locking] ../../lib/dbwrap/dbwrap.c:162(dbwrap_lock_order_unlock) Sep 11 13:45:01 sm2u-31 smbd[9208]: dbwrap_lock_order_unlock: release lock order 1 for /var/db/samba4/smbXsrv_open_global.tdb Sep 11 13:45:01 sm2u-31 smbd[9208]: [2023/09/11 13:45:01.566417, 5, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/files.c:556(file_free) Sep 11 13:45:01 sm2u-31 smbd[9208]: freed files structure 597332429 (1 used) Sep 11 13:45:02 sm2u-31 smbd[9208]: [2023/09/11 13:45:02.929664, 5, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:326(change_to_user_impersonate) Sep 11 13:45:02 sm2u-31 smbd[9208]: change_to_user_impersonate: Skipping user change - already user Sep 11 13:45:02 sm2u-31 smbd[9208]: [2023/09/11 13:45:02.929804, 5, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:300(print_impersonation_info) Sep 11 13:45:02 sm2u-31 smbd[9208]: print_impersonation_info: Impersonated user: uid=(0,500), gid=(0,544), cwd=[/export/Source] Sep 11 13:45:02 sm2u-31 smbd[9208]: [2023/09/11 13:45:02.933190, 5, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/dosmode.c:72(dos_mode_debug_print) Sep 11 13:45:02 sm2u-31 smbd[9208]: dos_mode_debug_print: dos_mode returning (0x220): "a[sparse]" Sep 11 13:45:02 sm2u-31 smbd[9208]: [2023/09/11 13:45:02.933260, 3, pid=9208, effective(500, 544), real(0, 0), class=smb2] ../../source3/smbd/smb2_write.c:215(smb2_write_complete_internal) Sep 11 13:45:02 sm2u-31 smbd[9208]: smb2: fnum 607832874, file copied_sparse_file_1, length=4194304 offset=17592183947264 wrote=4194304 Sep 11 13:45:02 sm2u-31 smbd[9208]: [2023/09/11 13:45:02.935962, 5, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:326(change_to_user_impersonate) Sep 11 13:45:02 sm2u-31 smbd[9208]: change_to_user_impersonate: Skipping user change - already user Sep 11 13:45:02 sm2u-31 smbd[9208]: [2023/09/11 13:45:02.936198, 5, pid=9208, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:300(print_impersonation_info) Sep 11 13:45:02 sm2u-31 smbd[9208]: print_impersonation_info: Impersonated user: uid=(0,500), gid=(0,544), cwd=[/export/Source] Sep 11 13:45:02 sm2u-31 smbd[9208]: [2023/09/11 13:45:02.936328, 1, pid=9208, effective(500, 544), real(0, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:422(ndr_print_debug) Sep 11 13:45:02 sm2u-31 smbd[9208]: state->lease_ptr: struct smb2_lease Sep 11 13:45:02 sm2u-31 smbd[9208]: lease_key: struct smb2_lease_key Sep 11 13:45:02 sm2u-31 smbd[9208]: data: ARRAY(2) Sep 11 13:45:02 sm2u-31 smbd[9208]: data : 0xfc490170199a4eb9 (-267681121874325831) Sep 11 13:45:02 sm2u-31 smbd[9208]: data : 0x62a1da799fb731a3 (7107201902872834467) Fails – 4.13.7 Sep 11 15:27:36 sm2u-20 smbd[39901]: time : Sun Sep 10 04:45:49 2023 UTC.979054 Sep 11 15:27:36 sm2u-20 smbd[39901]: share_file_id : 0x000000000000149c (5276) Sep 11 15:27:36 sm2u-20 smbd[39901]: uid : 0x000001f4 (500) Sep 11 15:27:36 sm2u-20 smbd[39901]: flags : 0x0000 (0) Sep 11 15:27:36 sm2u-20 smbd[39901]: name_hash : 0x6eddea6d (1860037229) Sep 11 15:27:36 sm2u-20 smbd[39901]: stale : 0x01 (1) Sep 11 15:27:36 sm2u-20 smbd[39901]: [2023/09/11 15:27:36.321907, 5, pid=39901, effective(500, 544), real(0, 0), class=locking] ../../lib/dbwrap/dbwrap.c:180(dbwrap_lock_order_unlock) Sep 11 15:27:36 sm2u-20 smbd[39901]: dbwrap_lock_order_unlock: release lock order 1 for /var/db/samba4/locking.tdb Sep 11 15:27:36 sm2u-20 smbd[39901]: [2023/09/11 15:27:36.321953, 5, pid=39901, effective(500, 544), real(0, 0), class=locking] ../../lib/dbwrap/dbwrap.c:148(dbwrap_lock_order_lock) Sep 11 15:27:36 sm2u-20 smbd[39901]: dbwrap_lock_order_lock: check lock order 1 for /var/db/samba4/smbXsrv_open_global.tdb Sep 11 15:27:36 sm2u-20 smbd[39901]: [2023/09/11 15:27:36.322063, 5, pid=39901, effective(500, 544), real(0, 0), class=locking] ../../lib/dbwrap/dbwrap.c:180(dbwrap_lock_order_unlock) Sep 11 15:27:36 sm2u-20 smbd[39901]: dbwrap_lock_order_unlock: release lock order 1 for /var/db/samba4/smbXsrv_open_global.tdb Sep 11 15:27:36 sm2u-20 smbd[39901]: [2023/09/11 15:27:36.322085, 5, pid=39901, effective(500, 544), real(0, 0)] ../../source3/smbd/files.c:1284(file_free) Sep 11 15:27:36 sm2u-20 smbd[39901]: file_free: freed files structure 2778309654 (3 used) Sep 11 15:27:36 sm2u-20 smbd[39901]: [2023/09/11 15:27:36.322123, 5, pid=39901, effective(500, 544), real(0, 0)] ../../source3/smbd/files.c:76(fsp_new) Sep 11 15:27:36 sm2u-20 smbd[39901]: fsp_new: allocated files structure (4 used) Sep 11 15:27:36 sm2u-20 smbd[39901]: [2023/09/11 15:27:36.322209, 5, pid=39901, effective(500, 544), real(0, 0), class=vfs] ../../source3/smbd/vfs.c:1334(check_reduced_name) Sep 11 15:27:36 sm2u-20 smbd[39901]: check_reduced_name: . reduced to /export/Dest Sep 11 15:27:36 sm2u-20 smbd[39901]: [2023/09/11 15:27:36.322293, 5, pid=39901, effective(500, 544), real(0, 0)] ../../source3/smbd/dosmode.c:69(dos_mode_debug_print) Sep 11 15:27:36 sm2u-20 smbd[39901]: dos_mode_debug_print: fdos_mode returning (0x30): "da" Sep 11 15:27:36 sm2u-20 smbd[39901]: [2023/09/11 15:27:36.322403, 5, pid=39901, effective(500, 544), real(0, 0)] ../../source3/smbd/files.c:1284(file_free) Sep 11 15:27:36 sm2u-20 smbd[39901]: file_free: freed files structure 0 (3 used) Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.391855, 5, pid=72454, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:327(change_to_user_impersonate) Sep 11 15:27:37 sm2u-20 smbd[72454]: change_to_user_impersonate: Skipping user change - already user Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.391882, 4, pid=72454, effective(500, 544), real(0, 0), class=vfs] ../../source3/smbd/vfs.c:938(vfs_ChDir) Sep 11 15:27:37 sm2u-20 smbd[72454]: vfs_ChDir to /export/Dest Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.391977, 5, pid=72454, effective(500, 544), real(0, 0), class=vfs] ../../source3/smbd/vfs.c:1000(vfs_ChDir) Sep 11 15:27:37 sm2u-20 smbd[72454]: vfs_ChDir: vfs_ChDir got /export/Dest Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.392044, 5, pid=72454, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:299(print_impersonation_info) Sep 11 15:27:37 sm2u-20 smbd[72454]: print_impersonation_info: Impersonated user: uid=(0,500), gid=(0,544), cwd=[/export/Dest] Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.392548, 5, pid=72454, effective(500, 544), real(0, 0)] ../../source3/smbd/dosmode.c:69(dos_mode_debug_print) Sep 11 15:27:37 sm2u-20 smbd[72454]: dos_mode_debug_print: fdos_mode returning (0x220): "a[sparse]" Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.392570, 2, pid=72454, effective(500, 544), real(0, 0), class=smb2] ../../source3/smbd/smb2_write.c:209(smb2_write_complete_internal) Sep 11 15:27:37 sm2u-20 smbd[72454]: smb2_write failed: fnum 4083718908, file copied_sparse_file_1, length=4194304 offset=17592183947264 nwritten=-1: NT_STATUS_INVALID_PARAMETER Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.392628, 3, pid=72454, effective(500, 544), real(0, 0), class=smb2] ../../source3/smbd/smb2_server.c:3963(smbd_smb2_request_error_ex) Sep 11 15:27:37 sm2u-20 smbd[72454]: smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_INVALID_PARAMETER] || at ../../source3/smbd/smb2_write.c:138 Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.395417, 5, pid=72454, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:327(change_to_user_impersonate) Sep 11 15:27:37 sm2u-20 smbd[72454]: change_to_user_impersonate: Skipping user change - already user Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.395558, 5, pid=72454, effective(500, 544), real(0, 0)] ../../source3/smbd/uid.c:299(print_impersonation_info) Sep 11 15:27:37 sm2u-20 smbd[72454]: print_impersonation_info: Impersonated user: uid=(0,500), gid=(0,544), cwd=[/export/Dest] Sep 11 15:27:37 sm2u-20 smbd[72454]: [2023/09/11 15:27:37.395707, 1, pid=72454, effective(500, 544), real(0, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:435(ndr_print_debug) Sep 11 15:27:37 sm2u-20 smbd[72454]: state->lease_ptr: struct smb2_lease Sep 11 15:27:37 sm2u-20 smbd[72454]: lease_key: struct smb2_lease_key Sep 11 15:27:37 sm2u-20 smbd[72454]: data: ARRAY(2) Sep 11 15:27:37 sm2u-20 smbd[72454]: data : 0x844687645a59822b (-8915289547251023317) Sep 11 15:27:37 sm2u-20 smbd[72454]: data : 0x508ffe1bc34bd69e (5805137839897958046) Sep 11 15:27:37 sm2u-20 smbd[72454]: lease_state : 0x00000007 (7) Sep 11 15:27:37 sm2u-20 smbd[72454]: 1: SMB2_LEASE_READ Sep 11 15:27:37 sm2u-20 smbd[72454]: 1: SMB2_LEASE_HANDLE :
Instrument this or the following functions with DEBUG messages: bool vfs_valid_pwrite_range(off_t offset, size_t length) { /* * See MAXFILESIZE in [MS-FSA] 2.1.5.3 Server Requests a Write */ static const uint64_t maxfilesize = 0xfffffff0000; uint64_t last_byte_ofs; bool ok; ok = sys_valid_io_range(offset, length); if (!ok) { return false; } if (length == 0) { return true; } last_byte_ofs = offset + length; if (last_byte_ofs > maxfilesize) { return false; } return true; } bool sys_valid_io_range(off_t offset, size_t length) { uint64_t last_byte_ofs; if (offset < 0) { return false; } if (offset > INT64_MAX) { return false; } if (length > UINT32_MAX) { return false; } last_byte_ofs = (uint64_t)offset + (uint64_t)length; if (last_byte_ofs > INT64_MAX) { return false; } return true; } ---------------------------------------------------- vfs_pwrite_data() has the following code: ssize_t vfs_pwrite_data(struct smb_request *req, files_struct *fsp, const char *buffer, size_t N, off_t offset) { size_t total=0; ssize_t ret; bool ok; ok = vfs_valid_pwrite_range(offset, N); if (!ok) { errno = EINVAL; return -1; } so if vfs_valid_pwrite_range() returns -1 you'll get NT_STATUS_INVALID_PARAMETER returned here.
(In reply to Jeremy Allison from comment #1) Yes, the MAXFILESIZE hardcoded as (0xfffffff0000) in [MS-FSA] 2.1.5.3 Server Requests a Write is the problem: https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/fbf656c3-b897-4b9c-abfd-7c8d876d77a1 MS-FSA 2.1.1.1 Per Volume also has a MaxFileSize: A 64-bit unsigned integer that denotes the maximum file size, in bytes, supported by the object store. https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ab3924f-c703-44a7-ac6f-3a75a7a849d8 On Windows it's set like this: For Windows 2000, Windows XP, Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2, the maximum file size of a file on an NTFS volume is the smaller of (232 – 1) * cluster size, and 16 terabytes (TB). For Windows 8 and Windows Server 2012, the maximum file size of a file on an NTFS volume is (232 – 1) * cluster size. For Windows 8.1 and subsequent the maximum file size of a file on an NTFS volume is (((232 * cluster size) – 64K). For example, if the cluster size is 512 bytes, the maximum file size is 2 TB. Maybe the documentation is wrong and we can use a more dynamic Volume.MaxfileSize instead of the hardcoded 0xfffffff0000
(In reply to Stefan Metzmacher from comment #2) I started this in order to find out the max size of the server, but it's not returned directly in MS-FSCC 2.3.22 FSCTL_GET_NTFS_VOLUME_DATA Reply https://git.samba.org/?p=metze/samba/wip.git;a=commitdiff;h=df75e031813b85ba0bfe7489a389fac6f184823c
We were in the midst of adding some debug statements as recommended and getting a build up and running. Based on the most recent comments, this looks like an issue that needs to be fixed in Samba. Is there a chance on getting a quick patch of Samba 4.13.7 so we can confirm with our test? Do you need anything further from us? Thanks
(In reply to Stefan Metzmacher from comment #3) Is there a reason (other than the MS-FSA doc) that we limit file size at all ? Couldn't we just check for 64-bit wrap with offset + size and if that passes then allow the write to go through to the system ?
(In reply to Barry Litt from comment #4) Just remove the maxfilesize check in vfs_valid_pwrite_range
(In reply to Jeremy Allison from comment #5) At least we need to keep the checks in sys_valid_io_range() in order to stay below INT64_MAX. That's also the typical limit in linux, unless the filesystem sets FMODE_UNSIGNED_OFFSET, but that's currently only used for /proc/self/mem. I think we should clarify if MS-FSA doesn't use Volume.MaxFileSize instead of the hardcoded value, as it does in 2.1.5.9.21 FSCTL_OFFLOAD_WRITE. Then we can just set our Volume.MaxFileSize to INT64_MAX, or find/get a way to ask the kernel for s_maxbytes of the superblock, but it seems fstatfs() doesn't return it currently. We could bisect the offsets passed to lseek(SEEK_SET) and check for EINVAL.
(In reply to Stefan Metzmacher from comment #7) Oh I agree we need the range checks, and the wrap checks. But not the maxfilesize restriction I think.
Ok. Just wanted to make sure there were no other ramifications for making that change. Will look at that tonight as was in meetings all day today. Thanks for the quick look.
We built a new image with the lines commented out for the maxfilesize check and all working as expected for large file and our general automated CIFS regression testing. Thanks for the help.
any update on when this will be in a patch? Thx