Can you get a wireshark trace on port 445 of the failed interaction. That should help track things down.

Also, try using command-line smbclient to connect with encryption and debug level 10. That uses the same internal code paths as libsmbclient and should be easier to reproduce.

Thanks !

Jeremy.

Created attachment 16623 [details]
tcpdump of port 445 on Solaris server

Perhaps I'm doing something wrong but I cannot reproduce the issue with:

smbclient //nonsuch/myshare --debuglevel=10 --encrypt --user=myuser

I am prompted for a password and it appears to work correctly. In the debug output I can see "negotiated dialect[SMB3_11] against server[10.45.1.72]" and can successfully browse the share using the command line.


The mount in fstab however still does not work.

With the following in fstab
//nonsuch/myshare/folder /mnt/nonsuch/folder cifs noauto,vers=3.11,seal,credentials=/root/creds_smb_library_core2 0 0

 I get the following in dmesg:
 [73279.063581] CIFS: Attempting to mount //nonsuch/myshare/folder
 [73279.064792] CIFS: VFS: \\nonsuch Dialect not supported by server. Consider  specifying vers=1.0 or vers=2.0 on mount for accessing older servers
 [73279.066514] CIFS: VFS: cifs_mount failed w/return code = -95

With the following in fstab (which was what I was using when originally reporting this issue)
//nonsuch/myshare/folder /mnt/nonsuch/folder cifs noauto,vers=3.0,seal,credentials=/root/creds_smb_library_core2 0 0

 I get the following in dmesg:
  [73322.319649] CIFS: Attempting to mount //nonsuch/myshare/folder
  [73322.483736] CIFS: VFS: \\nonsuch failed to connect to IPC (rc=-11)
  [73322.582204] CIFS: VFS: session 00000000f5e01434 has no tcon available for a dfs referral request
  [73322.583829] CIFS: VFS: cifs_mount failed w/return code = -2

 and the attached interaction with port 445 in wireshark as captured by the Solaris server.

Oh, well in this case it's a problem with the Linux kernel SMB3 client, which isn't developed as part of Samba, it's separate. Looks like the Samba code is working fine. When you mentioned libsmbclient I assumed you were doing it via our libraries, but doing fstab and mount.cifs is the kernel client.

I'll re-assign to Steve French.

Apologies for my ignorance, my knowledge is a little sketchy on the behind the scenes and I mis-understood this fact. Does this mean that this bugzilla is not the appropriate one for logging this issue?

Well hopefully Steve will pick it up from here, but you can also email him directly at Steve French <smfrench@gmail.com>.

There isn't much we can see in the traces you provide except the following:
1) for the "445broken.pcap" wireshark trace we can see that the server hung up the session (probably server bug but hard to prove since the request they hung up on (SMB3 tree connect) is encrypted when you mount with "seal").  If you have access to that system and can reproduce it, it might be helpful to dump the decryption keys so we can see the failing request and make sure it is not malformed.  See https://wiki.samba.org/index.php/Wireshark_Decryption for details on how to dump decryption keys

2) for the debug trace you attached all we can see is the "EINVAL" being returned

   [  205.645184] CIFS: fs/cifs/connect.c: Received no data or error: 0
   [  205.645187] CIFS: fs/cifs/connect.c: cifs_reconnect: will not do DFS 
                  failover: rc = -22

presumably the same issue (the server hung up due to a bug processing the tree connect request (the first encrypted request) when encryption is specified on mount).

Could you retry without specifying "seal" on mount and include (or send to my gmail) the wireshark traces?  In addition the dynamic trace info may be useful:

1) In one process type "trace-cmd record -e cifs"
2) Run the test in another window
3) dump the dynamic trace info "trace-cmd show"
4) kill the trace from step 1

If it can not be tried without encryption ("seal" on mount) then can you dump the decryption keys for the trace you run (as described in the link above)

Two additional questions:
1) As long as you are running a reasonably recent kernel or distro update (5.0 or later kernel should be fine, and current RHEL/CentOS/Oracle client ie RHEL/CentOS/Oracle 8.4 or 8.3 should be fine as well since Redhat backports many fixes to their older kernel) can you try not specifying "vers=" at all and see what it negotiates with the server?  (should be 3.1.1 - you can see in /proc/fs/cifs/DebugData)

2) Does mounting with "vers=2.1" work?

1) As long as you are running a reasonably recent kernel or distro update (5.0 or later kernel should be fine, and current RHEL/CentOS/Oracle client ie RHEL/CentOS/Oracle 8.4 or 8.3 should be fine as well since Redhat backports many fixes to their older kernel) can you try not specifying "vers=" at all and see what it negotiates with the server?  (should be 3.1.1 - you can see in /proc/fs/cifs/DebugData)

If I fail to specify the vers= option. Mounting fails completely. Dmesg says our usual:

[162979.482987] CIFS: VFS: \\nonsuch failed to connect to IPC (rc=-11)
[162979.484449] CIFS: VFS: session 0000000083b7840c has no tcon available for a dfs referral request
[162979.485899] CIFS: VFS: cifs_mount failed w/return code = -2

and the Solaris server says:

May 24 23:09:46 nonsuch smbcmn: [ID 997540 kern.warning] WARNING: ../../common/fs/smbsrv/smb2_dispatch.c:smb2_dispatch_message:134:Decryption failure (71)!
May 25 19:44:21 nonsuch smbcmn: [ID 997540 kern.warning] WARNING: ../../common/fs/smbsrv/smb2_dispatch.c:smb2_dispatch_message:134:Decryption failure (72)!

I have also seen the above Solaris server side messages intermittently and think they may be related to the issue I am experiencing. But if SMBv3 is broken on Solaris, I do not understand why it works fine with MacOS.

2) Does mounting with "vers=2.1" work?

Yes, it works fine. In fact, it is required to make anything work as omitting it results in the above behaviour.


Regarding your other comments about traces and encryption keys, I will need to time to get these, but will indeed do so.

Since the Solaris server logs:


May 24 23:09:46 nonsuch smbcmn: [ID 997540 kern.warning] WARNING: ../../common/fs/smbsrv/smb2_dispatch.c:smb2_dispatch_message:134:Decryption failure (71)!

are you sure that you mounted without encryption (ie did not specify "seal=" on mount)?

If this connection is defaulting to encryption whether or not the client specifies it on mount, that implies that the server is configured with encryption as required ... which is odd - because the server allowed vers=2.1 (which is not encrypted, encryption was added in the SMB3 and later versions of the protocol and not supported with SMB2.1) but fails with vers=3.0 or 3.1.1 (smb3.1.1 is the typical default) which presumably means the server is negotiating with encryption required (but only for a subset of dialects).  Strange server configuration.

Can you send or attach the vers=3.1.1 (or default with no vers= specified) wireshark trace so we can see what crypto algorithm the server is defaulting to (even if we can't see the keys - we can see how it is trying to encrypt/decrypt if smb3.1.1 is used instead of smb3.0)

When I omitted vers= as you requested, I definitely did not specify seal. I have attached the server SMB configuration for completeness but the most relevant settings are below:

server_lmauth_level=5
server_minprotocol=2.1
server_maxprotocol=3.1
server_encrypt_data=true
server_reject_unencrypt=false
server_signing_enabled=true
server_signing_required=false
restrict_anonymous=true

I will get a wireshark trace with vers=3.1.1 as you requested shortly.

Created attachment 16628 [details]
Solaris server SMB configuration

Will be interesting to see which crypto algorithm they negotiate - but at least one way around this is to mount with vers=2.1 or change the config line:
        server_encrypt_data=true
so that it doesn't end up causing SMB3.0 and later to encrypt until we figure out where the bug is in encryption (presumably is on the server side since encrypted mounts work to every other server from Linux).

Unfortunately, I spent quite a bit of time trying various combination of options and I have to say I have never managed to get SMBv3 to work between Linux and Solaris with or without encryption. E.g. setting vers=3.1.1 without seal and with server_encrypt_data=false on Solaris does not result in successful mounting:

[164849.013736] CIFS: VFS: \\seraphix-3.achelon.net failed to connect to IPC (rc=-13)
[164849.015760] CIFS: VFS: cifs_mount failed w/return code = -13


But let's try and debug one test case at a time so things don't get confusing. I have a wireshark dump with vers=3.1.1 and seal options. I will email to your gmail.

If there is a bug in Solaris SMBv3 encryption handling, I'm perplexed as to why it works fine when doing this from smbclient. E.g. the below appears to work:

smbclient //nonsuch/myshare --debuglevel=10 --encrypt --user=myuser

Based on the trace you sent - can see that when mounting with 3.1.1 (or default which ends up the same thing), the server responds with SMB2_ENCRYPTION_CAPABILITIES set to CipherId: AES-128-GCM which is interesting because that is the 'normal' case we see (Windows, Azure, current Samba server etc.) so this is less likely to be a bug in the client due to falling back to something different than the more common GCM.

Do you have the equivalent trace from smbclient (the Samba userspace tool) to the same share (trying to negotiate SMB3.1.1) for comparison?

In both trace you sent (seal and not specifying seal on the mount) you can see the server is requiring encryption (unless you mount with smb2.1 or earlier).  See attached screenshot

Created attachment 16629 [details]
session-setup-response-when-no-seal-on-mount

Understood. Please see your gmail for a pcap and a debug level 10 trace of smbclient successfully negotiating an encrypted SMB 3.1.1 connection with the same Solaris share, whereas the kernel mount for the same failed.

Comparing with smbclient, there are a few interesting things which differ:
a) smbclient sets a default domain name ("SAMBA").  To make this identical for the kernel mount ("mount -t cifs ...") case you could try setting domain= parameter to the same.  I doubt this will make a difference because in neither case does the server indicate in its SessionSetup response that authentication ended up as 'guest' so presumably Solaris server thinks the user authenticated properly in both cases (albeit it could be a very unlikely case where "SAMBA/username" is different than "username")
b) there are some NegotiateFlags (NTLMSSP flags) set differently during negotiation:
     1) smbclient sets "Negotiate Version"
     2) cifs.ko sets "Negotiate Seal" and "Negotiate Target Info" and "Negotiate 56"
but otherwise the flags look the same.  
c) smbclient sends both an old Lanman (Lanmanv2) and NTLM (NTLMv2) response in the NTLMSSP_AUTH SessionSetup request, but zeroes the Lanman field, while cifs.ko doesn't send Lanman.  This is unlikely to be related

Key next steps:
1) seeing if it is possible to decrypt the wireshark trace (see link provided earlier for instructions) although this may require rebuilding the cifs.ko to dump keys (does Solaris have a way to view encrypted traces taken on the server side?)
2) looking in more detail at the server.  It doesn't indicate why it was rejected:
./../common/fs/smbsrv/smb2_dispatch.c:smb2_dispatch_message:134:Decryption failure (71)!
May 25 19:44:21 nonsuch smbcmn: [ID 997540 kern.warning] WARNING: ../../common/fs/smbsrv/smb2_dispatch.c:smb2_dispatch_message:134:Decryption failure (72)!

Do you have any contacts with Solaris support to see if they can see if they can provide more information?  My guess is that they don't like the format or the flags of something specified in the tree connect (since this works to every other server type) - but they may also have some subtle rounding error with decryption on their server side.

Googling for the error I do see one other report of similar sounding problem to Solaris so it has been around a long time (see https://bugs.centos.org/view.php?id=16531)

I'm sure it won't shock you, but adding domain=SAMBA to the mount options hasn't miraculously fixed this, but at least it's another data point.

I am going to see if I can rebuild cifs.ko with the right options to dump the keys. But this is a lot of work - is it likely to lead to anything useful?

The reason that it is of some value is that if wireshark can decrypt it and shows no errors (ie decrypt the first encrypted frame, the SMB3.1.1 tree connect request) then it is even more likely a server bug ... (perhaps some strange case where they expect a padded response that has a length divisible by 8 or some such bug)

if you have access to the source RPM then rebuilding it might only take a few minutes (you could e.g. just remove the 2 ifdef CONFIG_CIFS_DEBUG_DUMP_KEYS in fs/cifs/smb2transport.c

e.g. remove the ifdef and endif here (and the one before that in the same file)

#ifdef CONFIG_CIFS_DEBUG_DUMP_KEYS
        cifs_dbg(VFS, "%s: dumping generated AES session keys\n", __func__);
        /*
         * The session id is opaque in terms of endianness, so we can't
         * print it as a long long. we dump it as we got it on the wire
         */
        cifs_dbg(VFS, "Session Id    %*ph\n", (int)sizeof(ses->Suid),
                        &ses->Suid);
        cifs_dbg(VFS, "Cipher type   %d\n", server->cipher_type);
        cifs_dbg(VFS, "Session Key   %*ph\n",
                 SMB2_NTLMV2_SESSKEY_SIZE, ses->auth_key.response);
        cifs_dbg(VFS, "Signing Key   %*ph\n",
                 SMB3_SIGN_KEY_SIZE, ses->smb3signingkey);
        if ((server->cipher_type == SMB2_ENCRYPTION_AES256_CCM) ||
                (server->cipher_type == SMB2_ENCRYPTION_AES256_GCM)) {
                cifs_dbg(VFS, "ServerIn Key  %*ph\n",
                                SMB3_GCM256_CRYPTKEY_SIZE, ses->smb3encryptionkey);
                cifs_dbg(VFS, "ServerOut Key %*ph\n",
                                SMB3_GCM256_CRYPTKEY_SIZE, ses->smb3decryptionkey);
        } else {
                cifs_dbg(VFS, "ServerIn Key  %*ph\n",
                                SMB3_GCM128_CRYPTKEY_SIZE, ses->smb3encryptionkey);
                cifs_dbg(VFS, "ServerOut Key %*ph\n",
                                SMB3_GCM128_CRYPTKEY_SIZE, ses->smb3decryptionkey);
        }
#endif

You might not need to recompile your kernel.

I have a GDB script (made for 4.4) you might be able to run as root

https://lists.samba.org/archive/samba-technical/attachments/20170524/2950140e/cifs_dump_keys.py

It reads the keys from kernel memory. The offset (OFF var) might have to be updated for your kernel (or not) but it's worth a try.

(this script just reads from memory, worst case it prints garbage)

Ah sorry, this script assumes you have a successful mount point. If mount fails it won't be of any use.

I will look into getting the decryption keys. I thought maybe I could dump the TreeConnect on the Solaris side using the Dtrace capability but if it's possible I haven't figured it out yet. 

I'm baffled as to how MacOS and smbclient work fine but Linux kernel mounts don't.
I guess my fear is that some flag is set wrong, maybe during negotiation, and there is no simple knob we can turn to set it just to test.

> fear is that some flag is set wrong, maybe during negotiation

There isn't an obvious reason why any of the flag differences would matter (unless server bug), but it should be possible to test mount with smb3.1.1 (without encryption) by changing the server config line
      server_encrypt_data=true
and make sure server doesn't hang up on tree connect (as it does with encryption)

If we verify that wireshark can decrypt it, then the only strange guesses I can think of that would cause the server to give up on the tree connect are:
1) difference in tree connect flags with smb3.1.1
2) differences in padding of the tree connect request that confuse the server

Unfortunately, as I mentioned by email. The connection does indeed also fail with server_encrypt_data=false.

After setting server_encrypt_data=false I try to mount with vers=3.1.1 and without the seal option (since encryption is off on the server). This results in the following mysterious error:

mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

[249955.014343] CIFS: Attempting to mount //nonsuch/myshare
[249955.020762] CIFS: VFS: \\nonsuch failed to connect to IPC (rc=-13)
[249955.023383] CIFS: VFS: cifs_mount failed w/return code = -13

see wireshark trace in your email "3.1.1_encryptionoff.pcap".

Trying with smbclient on the same share with the same user and same password with server_encrypt_data=false on the server and no --encrypt option results in success:

smbclient //nonsuch/myshare --debuglevel=10 --user=myuser

see wireshark trace in your email "3.1.1_smbclient.pcap".

Presumably Solaris server doesn't like the signature (which is odd).  Let's compare with SMB3.0 as well if possible.  Was the SMB2.1 mounts (which succeeded) signed?  That is weird if signed works with 2.1 and fails with 3.0 and 3.1.1

The server says:

May 26 19:53:58 nonsuch smbsrv: [ID 211007 kern.warning] WARNING: bad signature, cmd=0x3

so you are right, it doesn't like the signature.

I will obtain wireshark traces with vers=3.0 (and equivalent smb.conf
option for smbclient) set for comparison.

Note that in vers=3.1.1  the TreeConnect calls are always signed
which means that client and server must agree on what the session key is.
Just like when encryption is used.

Try to see what happens if you use vers=3.0 in this scenario as the TreeConnect will not be signed.


It does sound like there is something wrong with the session key and client and server disagree on it.

with vers=3.0 (and without seal) specified on the client and server_encrypt_data=false specified on the server, the following new fun happens:

mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

and dmesg says: 

[261829.875860] CIFS: Attempting to mount //nonsuch/myshare
[261829.891575] CIFS: VFS: \\nonsuch\IPC$ validate protocol negotiate failed: -13
[261829.893269] CIFS: VFS: \\nonsuch failed to connect to IPC (rc=-5)
[261829.895940] CIFS: VFS: \\nonsuch\myshare validate protocol negotiate failed: -13
[261829.897752] CIFS: VFS: session 00000000bfe08581 has no tcon available for a dfs referral request
[261829.900102] CIFS: VFS: cifs_mount failed w/return code = -2

I also notice a new message on the server server. Note that cmd=0x3 is now cmd=0xb - whatever that signifies.

May 26 23:11:53 nonsuch smbsrv: [ID 211007 kern.warning] WARNING: bad signature, cmd=0xb

A wireshark trace for the above has been emailed to you "3.0_mount.pcap"

As usual, for comparison smbclient works flawless (hopefully -m SMB3 was the right way to force SMB 3.0 dialect...)
smbclient //nonsuch/myshare -m SMB3 --user=myuser

A wireshark trace for the above smbclient interaction has been emailed to you: "3.0_smbclient.pcap"

Throughout all this dialogue with you, signing on the server has been 'enabled', but not 'required'.

Signing.  It is not required on the server config,

BUT in 3.1.1 signing is required for all TreeConnect calls (0x03) always, by the protocol.

In 3.0 and 3.0.2 signing is required for all Ioctl:fsctl_validate_negotiate_info calls. Also by protocol requirements.

The command code for Ioctl is 0x0b, which you saw in the log.


At least it got the mount process a bit further by using vers=3.0 since we got past the TreeConnect.   I forgot that in 3.0 we have that Ioctl call.
It could have worked, but I forgot about the Ioctl :-(


Anyway, this is another datapoint that tells us that the issue is that it is related to the session key imo.

Thanks for your help it - does seem that you've narrowed down the cause a little. Is it still required for me to dump those keys?

As requested I am emailing you details of the successful SMB2.1 negotiation using both smbclient:

smbclient //nonsuch/myshare -m SMB2 --user=myuser

and the CIFS mount command (with vers=2.1 and without seal).

(In reply to Richard Flint from comment #33)

From 'man smb.conf'

...
SMB2: Re-implementation of the SMB protocol. Used by Windows Vista and later versions of Windows. SMB2 has sub protocols available.

    SMB2_02: The earliest SMB2 version.

    SMB2_10: Windows 7 SMB2 version.

    SMB2_22: Early Windows 8 SMB2 version.

    SMB2_24: Windows 8 beta SMB2 version.

By default SMB2 selects the SMB2_10 variant.

SMB3: The same as SMB2. Used by Windows 8. SMB3 has sub protocols available.

    SMB3_00: Windows 8 SMB3 version. (mostly the same as SMB2_24)

    SMB3_02: Windows 8.1 SMB3 version.

    SMB3_10: early Windows 10 technical preview SMB3 version.

    SMB3_11: Windows 10 technical preview SMB3 version (maybe final).

By default SMB3 selects the SMB3_11 variant.
...

So '-m SMB3' is the same as '-m SMB3_11', I guess to want
'-m SMB3_00' instead in order to force 3.0.0

Thanks for the correction, you are quite right, the comparison file should have been done with -m SMB3_00 not -m SMB3.

I have created two new files:

SMB3_00_smbclient.pcap created with command:

smbclient //nonsuch/myshare -m SMB3_00 --user=myuser

and

SMB3_00_encrypt_smbclient.pcap created with command:

smbclient //nonsuch/myshare -m SMB3_00 --encrypt --user=myuser

I am emailing the traces to Steve French's gmail.

Appreciate it has been sometime since this was updated but I just wanted to update this for completeness. I have tested this on Fedora 35 (5.16.9-200.fc35.x86_64) and can confirm successful negotiation of 3.0 3.0.2 and 3.1.1 with Solaris 11.4 servers both with and without encryption (as enabled by the seal parameter). 

E.g. the following is successful: 
//myserver/myshare/myfolder /mnt/myserver/myfolder cifs noauto,nounix,vers=3.1.1,seal,noserverino,ro,_netdev,noexec,nosuid,perm,nodev,iocharset=utf8,cache=strict,sec=ntlmv2,credentials=/root/password,port=445,context="system_u:object_r:myapp_content_t:s0",forceuid,forcegid,file_mode=0440,dir_mode=0550,uid=1000,gid=1001 0 0

Though noisy. E.g.:

[Sat Feb 19 08:34:30 2022] CIFS: decode_ntlmssp_challenge: authentication has been weakened as server does not support key exchange
[Sat Feb 19 08:34:30 2022] CIFS: VFS: \\myserver\myshare error -9 on ioctl to get interface list
[Sat Feb 19 08:34:30 2022] CIFS: VFS: \\myserver\IPC$ smb2_get_dfs_refer: ioctl error: rc=-19

Intriguingly, despite specifying nounix in the mount, Wireshark shows we are still sending SMB2_POSIX_EXTENSIONS_CAPABILITY in the Negotiate Protocol Request - I'm not clear if that is the desired behaviour.

The issue is still reproducible on the latest CentOS 8 Stream release, but that it works on Fedora 35 makes we wonder if an issue was fixed in the meantime that never got back-ported to CentOS 8. If that's true, then this isn't really a fault in the CIFSVFS product itself I think - or maybe it isn't anymore.

Richard, If CentOS stream is missing backporting a fix for the past year to their 4.18 kernel (which is fairly old), would be less confusing to follow up with them. Do you remember if we narrowed down on the email thread what the change was that fixed this?