Bug 14464 - cannot create a directory in home over SMB2, mkdirat returns EBADF
Summary: cannot create a directory in home over SMB2, mkdirat returns EBADF
Status: RESOLVED DUPLICATE of bug 14427
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: 4.12.3
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Ralph Böhme
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-08-11 14:20 UTC by Alexander Bokovoy
Modified: 2020-08-13 06:32 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Bokovoy 2020-08-11 14:20:46 UTC
FreeIPA 4.8 supports running Samba file server on its domain member. We have an automated test that creates such setup, adds a simple share and attempts to operate on it via cifs.ko mount with kerberos authentication and multiuser.

In general, the setup works. However, there are few use cases which started to fail with Samba 4.12. Unfortunately, I did not record the specific version where the regression started to happen.

In the setup, there are three machines: a DC, a file server, and a client.

The file server setup allows to access home shares.

A share was mounted as root on the client:

kinit user1
mkdir -p /mnt/smb
mount -t cifs //fs.ipa.test/homes  /mnt/smb -o multiuser,sec=krb5i

All commands below were executed as usual user, not root

Step 1.
$ kinit user1

Step 2.
On samba client inside shared dir:
$ mkdir test
mkdir: cannot create directory ‘test’: Bad file descriptor

Step 3.
On samba server inside shared dir:
$ mkdir test_serv

On samba client inside shared dir:
$ mkdir test_serv/test
-> Success

Step4
On samba client inside shared dir:
$ mkdir test
-> Success


Looking into the server logs for the step 2, we can see:

2020/08/10 17:51:06.490508,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/uid.c:298(print_impersonation_info)
  print_impersonation_info: Impersonated user: uid=(1692800004,1692800004), gid=(0,1692800004), cwd=[/home/user1]
[2020/08/10 17:51:06.490627,  3, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2] ../../source3/smbd/smb2_server.c:3266(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[9] status[NT_STATUS_FILE_CLOSED] || at ../../source3/smbd/smb2_server.c:2627
[2020/08/10 17:51:06.490711, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2] ../../source3/smbd/smb2_server.c:3158(smbd_smb2_request_done_ex)
  smbd_smb2_request_done_ex: mid [34] idx[9] status[NT_STATUS_FILE_CLOSED] body[8] dyn[yes:1] at ../../source3/smbd/smb2_server.c:3314
[2020/08/10 17:51:06.490795,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../libcli/smb/smb2_signing.c:174(smb2_signing_sign_pdu)
  signed SMB2 message
[2020/08/10 17:51:06.490848, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2_credits] ../../source3/smbd/smb2_server.c:959(smb2_set_operation_credit)
  smb2_set_operation_credit: smb2_set_operation_credit: requested 10, charge 1, granted 10, current possible/max 7459/8192, total granted/max/low/range 743/8192/35/743
[2020/08/10 17:51:06.490946, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2_credits] ../../source3/smbd/smb2_server.c:959(smb2_set_operation_credit)
  smb2_set_operation_credit: smb2_set_operation_credit: requested 10, charge 1, granted 10, current possible/max 7449/8192, total granted/max/low/range 753/8192/35/753
[2020/08/10 17:51:06.490994, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2_credits] ../../source3/smbd/smb2_server.c:959(smb2_set_operation_credit)
  smb2_set_operation_credit: smb2_set_operation_credit: requested 10, charge 1, granted 10, current possible/max 7439/8192, total granted/max/low/range 763/8192/35/763
[2020/08/10 17:51:06.491039,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../libcli/smb/smb2_signing.c:174(smb2_signing_sign_pdu)
  signed SMB2 message
[2020/08/10 17:51:06.492660, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2] ../../source3/smbd/smb2_server.c:4034(smbd_smb2_io_handler)
  smbd_smb2_request idx[1] of 9 vectors
[2020/08/10 17:51:06.492782, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2_credits] ../../source3/smbd/smb2_server.c:694(smb2_validate_sequence_number)
  smb2_validate_sequence_number: smb2_validate_sequence_number: clearing id 35 (position 35) from bitmap
[2020/08/10 17:51:06.492856, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2_credits] ../../source3/smbd/smb2_server.c:694(smb2_validate_sequence_number)
  smb2_validate_sequence_number: smb2_validate_sequence_number: clearing id 36 (position 36) from bitmap
[2020/08/10 17:51:06.492972, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2] ../../source3/smbd/smb2_server.c:2351(smbd_smb2_request_dispatch)
  smbd_smb2_request_dispatch: opcode[SMB2_OP_CREATE] mid = 35
[2020/08/10 17:51:06.493049,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/uid.c:326(change_to_user_impersonate)
  change_to_user_impersonate: Skipping user change - already user
[2020/08/10 17:51:06.493125,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/uid.c:298(print_impersonation_info)
  print_impersonation_info: Impersonated user: uid=(1692800004,1692800004), gid=(0,1692800004), cwd=[/home/user1]
[2020/08/10 17:51:06.493197, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2] ../../source3/smbd/smb2_create.c:706(smbd_smb2_create_send)
  smbd_smb2_create_send: name [test_dir]
[2020/08/10 17:51:06.493248, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2] ../../source3/smbd/smb2_create.c:811(smbd_smb2_create_send)
  smbd_smb2_create_send: open execution phase
[2020/08/10 17:51:06.493295,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/filename.c:481(unix_convert)
  unix_convert called on file "test_dir"
[2020/08/10 17:51:06.493346, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/statcache.c:245(stat_cache_lookup)
  stat_cache_lookup: lookup failed for name [TEST_DIR]
[2020/08/10 17:51:06.493393,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/filename.c:682(unix_convert)
  unix_convert begin: name = test_dir, dirpath = ., start = test_dir
[2020/08/10 17:51:06.493491, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/mangle_hash2.c:413(is_mangled)
  is_mangled test_dir ?
[2020/08/10 17:51:06.493550, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/mangle_hash2.c:352(is_mangled_component)
  is_mangled_component test_dir (len 8) ?
[2020/08/10 17:51:06.493597, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/mangle_hash2.c:413(is_mangled)
  is_mangled test_dir ?
[2020/08/10 17:51:06.493674, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/mangle_hash2.c:352(is_mangled_component)
  is_mangled_component test_dir (len 8) ?
[2020/08/10 17:51:06.493738, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=vfs] ../../source3/smbd/vfs.c:1282(check_reduced_name)
  check_reduced_name: check_reduced_name [.] [/home/user1]
[2020/08/10 17:51:06.493795, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=vfs] ../../source3/smbd/vfs.c:1346(check_reduced_name)
  check_reduced_name realpath [.] -> [/home/user1]
[2020/08/10 17:51:06.493839,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=vfs] ../../source3/smbd/vfs.c:1457(check_reduced_name)
  check_reduced_name: . reduced to /home/user1
[2020/08/10 17:51:06.493979, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/mangle_hash2.c:413(is_mangled)
  is_mangled test_dir ?
[2020/08/10 17:51:06.494030, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/mangle_hash2.c:352(is_mangled_component)
  is_mangled_component test_dir (len 8) ?
[2020/08/10 17:51:06.494070,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/filename.c:1123(unix_convert)
  New file test_dir
[2020/08/10 17:51:06.494112, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=vfs] ../../source3/smbd/vfs.c:1282(check_reduced_name)
  check_reduced_name: check_reduced_name [test_dir] [/home/user1]
[2020/08/10 17:51:06.494172, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=vfs] ../../source3/smbd/vfs.c:1346(check_reduced_name)
  check_reduced_name realpath [test_dir] -> [/home/user1/test_dir]
[2020/08/10 17:51:06.494215,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=vfs] ../../source3/smbd/vfs.c:1457(check_reduced_name)
  check_reduced_name: test_dir reduced to /home/user1/test_dir
[2020/08/10 17:51:06.494260, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/open.c:5925(create_file_default)
  create_file_default: create_file: access_mask = 0x100 file_attributes = 0x0, share_access = 0x7, create_disposition = 0x2 create_options = 0x1 oplock_request = 0x0 private_flags = 0x0 root_dir_fid = 0x0, ea_list = (nil), sd = (nil), fname = test_dir
[2020/08/10 17:51:06.494314, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/open.c:5372(create_file_unixpath)
  create_file_unixpath: create_file_unixpath: access_mask = 0x100 file_attributes = 0x0, share_access = 0x7, create_disposition = 0x2 create_options = 0x1 oplock_request = 0x0 private_flags = 0x0 ea_list = (nil), sd = (nil), fname = test_dir
[2020/08/10 17:51:06.494364,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/open.c:4255(open_directory)
  open_directory: opening directory test_dir, access_mask = 0x100, share_access = 0x7 create_options = 0x1, create_disposition = 0x2, file_attributes = 0x10
[2020/08/10 17:51:06.494415,  5, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/dosmode.c:208(unix_mode)
  unix_mode: unix_mode(test_dir) returning 0755
[2020/08/10 17:51:06.494518, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:3520(posix_get_nt_acl)
  posix_get_nt_acl: called for file .
[2020/08/10 17:51:06.494652, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/passdb/lookup_sid.c:1174(xid_to_sid)
  xid_to_sid: UID 1692800004 -> S-1-5-21-3722864543-743103260-3007237568-1004 from cache
[2020/08/10 17:51:06.494692, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/passdb/lookup_sid.c:1174(xid_to_sid)
  xid_to_sid: GID 1692800004 -> S-0-0 from cache
[2020/08/10 17:51:06.494742, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/passdb/lookup_sid.c:1229(xid_to_sid)
  xid_to_sid: GID 1692800004 -> S-1-22-2-1692800004 fallback
[2020/08/10 17:51:06.494776, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:2763(canonicalise_acl)
  canonicalise_acl: Access ace entries before arrange :
[2020/08/10 17:51:06.494804, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:2776(canonicalise_acl)
  canon_ace index 0. Type = allow SID = S-1-1-0 other SMB_ACL_OTHER ace_flags = 0x0 perms ---
[2020/08/10 17:51:06.494838, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:2776(canonicalise_acl)
  canon_ace index 1. Type = allow SID = S-1-22-2-1692800004 gid 1692800004 SMB_ACL_GROUP_OBJ ace_flags = 0x0 perms ---
[2020/08/10 17:51:06.494869, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:2776(canonicalise_acl)
  canon_ace index 2. Type = allow SID = S-1-5-21-3722864543-743103260-3007237568-1004 uid 1692800004 SMB_ACL_USER_OBJ ace_flags = 0x0 perms rwx
[2020/08/10 17:51:06.494923, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:848(print_canon_ace_list)
  print_canon_ace_list: canonicalise_acl: ace entries after arrange
  canon_ace index 0. Type = allow SID = S-1-5-21-3722864543-743103260-3007237568-1004 uid 1692800004 SMB_ACL_USER_OBJ ace_flags = 0x0 perms rwx
  canon_ace index 1. Type = allow SID = S-1-22-2-1692800004 gid 1692800004 SMB_ACL_GROUP_OBJ ace_flags = 0x0 perms ---
  canon_ace index 2. Type = allow SID = S-1-1-0 other SMB_ACL_OTHER ace_flags = 0x0 perms ---
[2020/08/10 17:51:06.494982, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:1112(map_canon_ace_perms)
  map_canon_ace_perms: Mapped (UNIX) 1c0 to (NT) 1f01ff
[2020/08/10 17:51:06.495012, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:1112(map_canon_ace_perms)
  map_canon_ace_perms: Mapped (UNIX) 0 to (NT) 0
[2020/08/10 17:51:06.495039, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=acls] ../../source3/smbd/posix_acls.c:1112(map_canon_ace_perms)
  map_canon_ace_perms: Mapped (UNIX) 0 to (NT) 0
[2020/08/10 17:51:06.495089,  2, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/open.c:4307(open_directory)
  open_directory: unable to create test_dir. Error was NT_STATUS_INVALID_HANDLE
[2020/08/10 17:51:06.495123, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/open.c:5744(create_file_unixpath)
  create_file_unixpath: NT_STATUS_INVALID_HANDLE
[2020/08/10 17:51:06.495152, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0)] ../../source3/smbd/open.c:6031(create_file_default)
  create_file: NT_STATUS_INVALID_HANDLE
[2020/08/10 17:51:06.495187,  3, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2] ../../source3/smbd/smb2_server.c:3266(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_INVALID_HANDLE] || at ../../source3/smbd/smb2_create.c:334
[2020/08/10 17:51:06.495219, 10, pid=22111, effective(1692800004, 1692800004), real(1692800004, 0), class=smb2] ../../source3/smbd/smb2_server.c:3158(smbd_smb2_request_done_ex)
  smbd_smb2_request_done_ex: mid [35] idx[1] status[NT_STATUS_INVALID_HANDLE] body[8] dyn[yes:1] at ../../source3/smbd/smb2_server.c:3314

It seems the actual failure happened in mkdir_internal() which was called from the open_directory() and called SMB_VFS_MKDIRAT(). Default implementation of mkdirat() VFS wrapper calls mkdirat() and then mkdir_internal() translates returned code to NT status error code. EBADF is translated to NT_STATUS_INVALID_HANDLE and means that a directory file descriptor passed to mkdirat() was invalid.

If the setup in question is left without any additional activity, some time later 'mkdir test_dir' will succeed. Looks like there is either a race for the directory FSP in that path or use of -1 as a FSP fd?
Comment 1 Alexander Bokovoy 2020-08-11 14:25:03 UTC
The issue happens routinely in FreeIPA upstream CI tests against Fedora 32 and Rawhide.

For example, http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/69e99d40-db61-11ea-a417-fa163e8ac44c/report.html

Sadly, Samba logs aren't collected in those runs so we don't know what exactly happened. But with manual runs we get those issues easily reproducible.
Comment 2 Alexander Bokovoy 2020-08-11 14:31:01 UTC
FreeIPA upstream ticket https://pagure.io/freeipa/issue/8184 shows that the problem is seen on f32 but not f31.

I thought it was a bug in cifs.ko because at some point cifs.ko crashed but this was a mistake -- the crash was on the client side in unrelated operation and we are still seeing EBADF propagation from the server side even without cifs.ko crash on the client.

Fedora 31 has Samba 4.11, Fedora 32 has Samba 4.12.
Comment 3 Alexander Bokovoy 2020-08-11 14:31:33 UTC
Jeremy, could you please help with investigation?
Comment 4 Ralph Böhme 2020-08-11 15:35:02 UTC
With which version do you see the failure?
Comment 5 Alexander Bokovoy 2020-08-11 15:43:48 UTC
It was reproducible with at least samba-4.12.0-0.0.rc1.fc32.x86_64 in Fedora 32 and also with current samba-4.12.3-11.el8.3.x86_64.rpm in RHEL 8.3 (beta).

I suspect it is 4.12 vs 4.11.
Comment 6 Ralph Böhme 2020-08-11 15:47:24 UTC
(In reply to Alexander Bokovoy from comment #5)
This could be bug 14427.
Comment 7 Alexander Bokovoy 2020-08-11 15:59:00 UTC
Thanks Ralph, we'll do a test build tomorrow.
Comment 8 Alexander Bokovoy 2020-08-12 11:19:21 UTC
Manual tests do show it working with the patch from bug 14427. My colleague is going to run a formal test run in automation to validate our manual testing.

Once that is done, I'll close this bug as a duplicate of bug 14427.
Comment 9 Alexander Bokovoy 2020-08-13 06:32:01 UTC
A re-run of the test suite against a patched version of Samba with a fix from bug 14427 helped. The bug is no more reproducible.

Thank you!

*** This bug has been marked as a duplicate of bug 14427 ***