Bug 8046 - Authentication in mount.cifs (NTLMv2 in NTLMSSP in SPNEGO in GSS-API)
Summary: Authentication in mount.cifs (NTLMv2 in NTLMSSP in SPNEGO in GSS-API)
Status: RESOLVED FIXED
Alias: None
Product: CifsVFS
Classification: Unclassified
Component: user space tools (show other bugs)
Version: 2.6
Hardware: All All
: P5 regression
Target Milestone: ---
Assignee: shirishpargaonkar@gmail.com
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-03-29 09:09 UTC by Bug Reporter
Modified: 2012-04-06 11:39 UTC (History)
6 users (show)

See Also:


Attachments
Wireshark capture (18.74 KB, text/plain)
2011-05-10 15:17 UTC, ncl.bourbaki
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bug Reporter 2011-03-29 09:09:42 UTC
mount.cifs cannot do recent NLTM authentication, more specifically NTLMv2 in NTLMSSP in SPNEGO in GSS-API.

I've asked Jeff Layton for confirmation last year (cfr. infra) and he referred me to the mailinglist and here.  The problem seems to exist still.

Kind regards
Bart Van den Broeck


On Fri, 30 Apr 2010 08:26:21 +0200 Bart Van den Broeck wrote:

> > Hello mr. Layton
> >
> > [...] It concerns authentication in mount.cifs.  Am I correct in stating
> > that authentication with NTLMv2 (and probably NTLM and LM) is always done
> > directly with hashes in the SMB header (I think you call it "raw NTLMSSP")
> > and there is no way to make mount.cifs use NTLMSSP in SPNEGO in GSS-API (I
> > think you call it "Extended Security" or "SPNEGO-wrapped NTLMSSP")?  Both
> > smbclient and Mac OS X's mount_smbfs do seem to be able to do the latter.

Actually "raw NTLMSSP" is NTLMSSP that's embedded in GSSAPI but without
SPNEGO. The hashes in the header are just plain old NTLMv2. You're
correct that CIFS doesn't support NTLMv2 in NTLMSSP at the moment.


> > Our problem is that we can connect with both smbclient and MacOSX's
> > mount_smbfs to our NetApp ONTAP filer, but not with mount.cifs.  The first
> > two use NTLMSSP in SPNEGO in GSS-API, the latter uses NTLMv2 hashes in the
> > SMB header and that fails.  [...]

Yeah, [...]
Comment 1 Bug Reporter 2011-03-29 09:11:53 UTC
This problem blocks us from being able to mount SMB shares of a NetApp ONTAP filer on a GNU/Linux system.
Comment 2 shirishpargaonkar@gmail.com 2011-04-04 13:39:18 UTC
Hi, would you be able to provide a wireshark trace of the failure?
Comment 3 shirishpargaonkar@gmail.com 2011-04-04 13:43:24 UTC
Can you try with mount option sec=ntlm or sec=ntlmi or sec=ntlmv2 or sec=ntlmv2i
or sec=ntlmssp or sec=ntlmsspi
Comment 4 werner maes 2011-04-04 13:59:19 UTC
I've tried all these options, no success however.
I'm not familiar with wireshark, I have to ask someone.

: kstestlinux64 / 15:54#; mount.cifs //server.DOMAIN.kuleuven.be/user /cifs -o username="DOMAIN\user"
Password: 
mount error(95): Operation not supported
: kstestlinux64 / 15:55#; mount.cifs //server.DOMAIN.kuleuven.be/user /cifs -o username="DOMAIN\user",sec=ntlm
Password: 
mount error(13): Permission denied
: kstestlinux64 / 15:55#; mount.cifs //server.DOMAIN.kuleuven.be/user /cifs -o username="DOMAIN\user",sec=ntlmi
Password: 
mount error(95): Operation not supported
: kstestlinux64 / 15:55#; mount.cifs //server.DOMAIN.kuleuven.be/user /cifs -o username="DOMAIN\user",sec=ntlmv2
Password: 
mount error(13): Permission denied
: kstestlinux64 / 15:55#; mount.cifs //server.DOMAIN.kuleuven.be/user /cifs -o username="DOMAIN\user",sec=ntlmv2i
Password: 
mount error(95): Operation not supported
: kstestlinux64 / 15:55#; mount.cifs //server.DOMAIN.kuleuven.be/user /cifs -o username="DOMAIN\user",sec=ntlmssp
Password: 
mount error(5): Input/output error
: kstestlinux64 / 15:55#; mount.cifs //server.DOMAIN.kuleuven.be/user /cifs -o username="DOMAIN\user",sec=ntlmsspi
Password: 
mount error(5): Input/output error
Comment 5 shirishpargaonkar@gmail.com 2011-04-04 14:06:21 UTC
Hi, do you have any user(name) which does not involve characters like 
\ (i.e. Domain\user) that you can try with various sec= options?
Looks like signing is disabled on the Netapp cifs server!
Comment 6 werner maes 2011-04-04 14:25:47 UTC
Hello

Would something like 
mount.cifs //server.domain.kuleuven.be/user /cifs -o username=user,domain=DOMAIN,sec=ntlmssp

do the job that you asked for (no \ in username)?

The results stay the same with the various options !

My netapp admin has enabled cifs signing but it does not solve the problem. The netapp filer generates this error message:

"login rejected due to invalid client challenge/response parameter length"
Comment 7 shirishpargaonkar@gmail.com 2011-04-04 14:36:50 UTC
Yes, you can try that command except...

mount.cifs //server.domain.kuleuven.be/user /cifs -o
username=user,domain=DOMAIN,sec=ntlmssp


Should //server.domain.kuleuven.be/user be 
 //server.domain.kuleuven.be/share 

You can try without specifying domain=DOMAIN name

If signing on the server is enabled, you will need
mount option sec=ntlmsspi
Comment 8 werner maes 2011-04-04 14:48:53 UTC
hello again

no, there is a share that is called user, my apologies for the confusion.

: kstestlinux64 / 16:39#; smbclient //server.domain.kuleuven.be/user -U "DOMAIN\user" 
Enter DOMAIN\user's password: 
Domain=[DOMAIN] OS=[Windows 5.0] Server=[Windows 2000 LAN Manager]
smb: \> ls
  OK.txt                              A        0  Wed Mar 30 14:22:42 2011

                40960 blocks of size 2097152. 40957 blocks available
smb: \> q

All of our users are domain users, so I always have to add the domain.

: kstestlinux64 / 16:41#; mount.cifs //server.domain.kuleuven.be/user /cifs -o username=user,domain=DOMAIN,sec=ntlmsspi
Password: 
mount error(5): Input/output error

dmesg shows

fs/cifs/cifsfs.c: Devname: //server.domain.kuleuven.be/user flags: 64 
fs/cifs/connect.c: CIFS VFS: in cifs_mount as Xid: 2417 with uid: 0
fs/cifs/connect.c: Domain name set
fs/cifs/connect.c: Username: user
fs/cifs/connect.c: UNC: \\server.domain.kuleuven.be\user ip: x.x.x.x
fs/cifs/netmisc.c: address conversion returned 1 for  x.x.x.x
fs/cifs/connect.c: Socket created
fs/cifs/connect.c: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x1b58
fs/cifs/connect.c: CIFS VFS: in cifs_get_smb_ses as Xid: 2418 with uid: 0
fs/cifs/connect.c: Existing smb sess not found
fs/cifs/cifssmb.c: secFlags 0x1081
fs/cifs/cifssmb.c: NTLMSSP only mechanism, enable extended security
fs/cifs/transport.c: For smb_command 114
fs/cifs/transport.c: Sending smb:  total_len 82
fs/cifs/connect.c: Demultiplex PID: 18794
fs/cifs/connect.c: rfc1002 length 0xb4
fs/cifs/transport.c: sync_mid_result: cmd=114 mid=1 state=4
fs/cifs/cifssmb.c: Dialect: 2
fs/cifs/cifssmb.c: Max buf = 16472
fs/cifs/cifssmb.c: negprot rc -5
fs/cifs/connect.c: CIFS VFS: leaving cifs_get_smb_ses (xid = 2418) rc = -5
fs/cifs/connect.c: CIFS VFS: leaving cifs_mount (xid = 2417) rc = -5
CIFS VFS: cifs_mount failed w/return code = -5
Comment 9 werner maes 2011-04-04 14:55:57 UTC
hello

My netapp admin has made a local account and I can now confirm that the problem is not related to the domain parameter :(
Comment 10 shirishpargaonkar@gmail.com 2011-04-04 15:00:34 UTC
OK. I think wireshark trace would be really useful.

If you do not specify domain=DOMAIN parameter, ntlmssp authentication
packet exchange will discover domain name.
Can you do 
 dmesg -c
 echo 7 > /proc/fs/cifs/cifsFYI
 issue mount command
 send the syslog buffer
Comment 11 shirishpargaonkar@gmail.com 2011-04-04 15:03:41 UTC
Looks like we are failing during negotiation protocol itself?

fs/cifs/cifssmb.c: NTLMSSP only mechanism, enable extended security
fs/cifs/cifssmb.c: negprot rc -5


session setup is not even happening
Comment 12 werner maes 2011-04-04 15:06:56 UTC
I ask someone tomorrow for help with wireshark

: kstestlinux64 / 17:02#; cat /proc/fs/cifs/cifsFYI 
7
: kstestlinux64 / 17:01#; dmesg -c
: kstestlinux64 / 17:01#; mount.cifs //server.domain.kuleuven.be/user /cifs -o username=user,sec=ntlmsspi
Password: 
mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
: kstestlinux64 / 17:02#; dmesg
fs/cifs/cifsfs.c: Devname: //server.domain.kuleuven.be/user flags: 64 
fs/cifs/connect.c: CIFS VFS: in cifs_mount as Xid: 2451 with uid: 0
fs/cifs/connect.c: Username: user
fs/cifs/connect.c: UNC: \\server.domain.kuleuven.be\u0023628 ip: x.x.x.x
fs/cifs/netmisc.c: address conversion returned 1 for x.x.x.x
fs/cifs/connect.c: Socket created
fs/cifs/connect.c: sndbuf 16384 rcvbuf 87380 rcvtimeo 0x1b58
fs/cifs/connect.c: CIFS VFS: in cifs_get_smb_ses as Xid: 2452 with uid: 0
fs/cifs/connect.c: Existing smb sess not found
fs/cifs/cifssmb.c: secFlags 0x1081
fs/cifs/cifssmb.c: NTLMSSP only mechanism, enable extended security
fs/cifs/transport.c: For smb_command 114
fs/cifs/transport.c: Sending smb:  total_len 82
fs/cifs/connect.c: Demultiplex PID: 19153
fs/cifs/connect.c: rfc1002 length 0xb4
fs/cifs/transport.c: sync_mid_result: cmd=114 mid=1 state=4
fs/cifs/cifssmb.c: Dialect: 2
fs/cifs/cifssmb.c: Max buf = 16472
fs/cifs/cifssmb.c: negprot rc -5
fs/cifs/connect.c: CIFS VFS: leaving cifs_get_smb_ses (xid = 2452) rc = -5
fs/cifs/connect.c: CIFS VFS: leaving cifs_mount (xid = 2451) rc = -5CIFS VFS: cifs_mount failed w/return code = -5
Comment 13 werner maes 2011-04-04 15:33:07 UTC
Would this wireshark command be sufficient:

tethereal  -V -x host x.x.x.x and port 445 ?

Seem to give a lot of output
Comment 14 shirishpargaonkar@gmail.com 2011-04-04 15:59:00 UTC
Not sure. Perhaps. I have never used tethereal command.
Comment 15 werner maes 2011-04-05 07:40:12 UTC
Ok
I've used 

    tcpdump -i eth0 -e -n -p -s 0 -w dump.pcap

I'll send you the trace via mail
Comment 16 shirishpargaonkar@gmail.com 2011-04-05 08:15:10 UTC
Need to debug cifs output.   As shown the syslog, we are not going
beyong negprot, cifs client should understand NTLMSSP as one of the
sec mech from MechType list.  Not sure why it is not happening.
Comment 17 werner maes 2011-04-05 08:19:25 UTC
How can I help?
Comment 18 shirishpargaonkar@gmail.com 2011-04-05 08:23:17 UTC
Not sure. Will have to look in the code and syslog debug message.
Will update the bug.
Comment 19 shirishpargaonkar@gmail.com 2011-04-05 12:36:26 UTC
We need to clarify from Netapp server folks why
 Extended Security Negotiation is not supported   in the Flags2 field 
of smb header in the Negotiation Protocol response sent by the server
as seen in the network trace.
Perhaps there is some setting on the server that is not turned ON!
Comment 20 werner maes 2011-04-05 12:44:20 UTC
hello
ok, but why does it work with smbclient? or with macosx?
Comment 21 shirishpargaonkar@gmail.com 2011-04-05 12:51:03 UTC
Can you please send a wireshark trace when smbclient from the same machine
is able to connect?
Perhaps smbclient and/or macosx clien do not insist on extended security!
What authentication mechanism is being used by those two clients?
network trace will tell for sure.
Comment 22 werner maes 2011-04-05 13:16:05 UTC
I'll send a wireshark trace (via mail) from a succesful smbclient connection.

From what I can see it uses NTLMSSP.
Comment 23 shirishpargaonkar@gmail.com 2011-04-05 13:23:07 UTC
Yes it does. smbclient, inspite of similar negotiate protocol response from
the same server (extended security negotiation is not supported in flags2
in smb header), sends ntlmssp type 1 request and session setup proceeds
followed by rest of the exchange.

Let me look into cifs code.  I will probably add more debug code in a patch.
Comment 24 shirishpargaonkar@gmail.com 2011-04-05 14:05:15 UTC
NetApp server does indicate in Capability field in negprot response
that it supports extended security exchanges.
 Extended security exchanges are supported

Perhaps cifs vfs client ought to take that into consideration
as it does consider Extended Security Negotiation bit in Flags2 field
of SMB header.
Comment 25 shirishpargaonkar@gmail.com 2011-04-06 14:04:51 UTC
posted patch for the cifs module in mainline kernel on cifs mailing list.
It has been tested and does resolve the problem reported here.
Comment 26 ncl.bourbaki 2011-05-10 11:40:06 UTC
Hi,

I'd like to report that the given patch on does not allow us to mount NetApp shares (7.3.x)using Kerberos only as authentication method. 
Without the patch a TGS is not requested. So at leat with the patch the ticket is obtained, but the exchange still fails with the following message: "Input/output error". 

Tracing shows: 
  SPNEGO reply blob:: dump ... XXX ...
  fs/cifs/transport.c: For smb_command 115
  fs/cifs/transport.c: Sending smb:  total_len 1916
...
  fs/cifs/connect.c: rfc1002 length 0xf8
...
  fs/cifs/transport.c: sync_mid_result: cmd=115 mid=2 state=4
...
  CIFS VFS: Unexpected SMB signature
  Status code returned 0xc0000016 NT_STATUS_MORE_PROCESSING_REQUIRED
...

Mounting other shares with Kerberos (Windows shares for instance) does work.

Any help is appreciated.

Nicolas.
Comment 27 shirishpargaonkar@gmail.com 2011-05-10 11:46:51 UTC
What is TGS?
Can you list your mount command?
A wireshark trace obtained during the exercise would help.
Comment 28 ncl.bourbaki 2011-05-10 15:17:01 UTC
Created attachment 6441 [details]
Wireshark capture
Comment 29 ncl.bourbaki 2011-05-10 15:19:18 UTC
(In reply to comment #27)
> What is TGS?
> Can you list your mount command?
> A wireshark trace obtained during the exercise would help.

A TGS is a Kerberos service ticket used to authenticate to the CIFS service.

I use the following commands:

sudo mount //filer02.my.realm/test /mnt -o sec=krb5i,user=myuser,uid=myuid

I tried with sec=krb5, no diff.

I provided with a text version of a wireshark capture of the mount.

Thanks for your help.

Nicolas.
Comment 30 shirishpargaonkar@gmail.com 2011-05-11 11:57:04 UTC
(In reply to comment #29)

Not sure what is the cause for server to return KRB5KRB_AP_ERR_MODIFIED.
Comment 31 ncl.bourbaki 2011-05-11 13:05:05 UTC
(In reply to comment #30)
> (In reply to comment #29)
> 
> Not sure what is the cause for server to return KRB5KRB_AP_ERR_MODIFIED.

Well it works for Windows and Mac OS ...
Comment 32 ncl.bourbaki 2011-05-18 09:05:32 UTC
(In reply to comment #31)
> (In reply to comment #30)
> > (In reply to comment #29)
> > 
> > Not sure what is the cause for server to return KRB5KRB_AP_ERR_MODIFIED.
> 
> Well it works for Windows and Mac OS ...

I can confirm that your patch works in conjunction with the following patch:

https://attachments.samba.org/attachment.cgi?id=6171

Without it cifs.upcall will not work with our filer. I managed to track it down because natty's libsmb was able to mount the share using gvfs-mount or konqueror. I compared the two version of source3/libsmb/clikrb5.c and found a reference to bug 7883 which pointed to a solution.

Test configuration: 
  Ubuntu Maverick 
  2.6.38 cifs.ko + your patch
  cifs-utils 4.5 + patch added above
  

Do you know if you're patch is planned to be included in the upstream kernel ?

Cheers.

Nicolas.
Comment 33 Richard Smits 2011-11-14 19:52:58 UTC
We are also experiencing this problem. smbclient works with krb5 security, but a mount.cifs with krb5i security to our netapp filer fails.
----
mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
----
Looks like the same problem. A colleague of me patched samba with the following patch :
https://bugzilla.samba.org/show_bug.cgi?id=6682 - (Netapp and krb5 failed)
(Workaround patch)

That works.

+++ cifssmb.c

Will there be a patch in upstream kernel ?

Greetings .. Richard
Comment 34 shirishpargaonkar@gmail.com 2011-11-15 12:33:23 UTC
I think this bug should be closed if the issue of ntlmssp auth failure
has been fixed.  That is what the bug was opened for.
The failure to authenticate using krb5 should be tracked using bug
6682.

If this is alright, I would like this bug closed in a week's time
since issue of ntlmssp auth failure is now fixed.
Comment 35 shirishpargaonkar@gmail.com 2011-11-21 16:54:07 UTC
Steve, can you close this bug since the problem this bug was opened
for has been fixed?  Or you can assign it to me and I can close it.
Comment 36 Jeff Layton 2012-04-06 11:39:28 UTC
Reassigning to Shirish and closing bug. Nice work, Shirish!