Bug 7257 - mount.cifs cannot mount a DFS share when using Kerberos authentication
mount.cifs cannot mount a DFS share when using Kerberos authentication
Status: RESOLVED INVALID
Product: CifsVFS
Classification: Unclassified
Component: user space tools
2.6
Other Linux
: P5 minor
: ---
Assigned To: Jeff Layton
Samba QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-03-16 14:32 UTC by Chuck Short
Modified: 2012-04-06 12:00 UTC (History)
2 users (show)

See Also:


Attachments
tcpdump capture when running "smbclient -k -c showconnect //warthogs.biz/testnamespace/firstshare" (21.59 KB, application/cap)
2010-03-17 16:19 UTC, Etienne Goyer
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Chuck Short 2010-03-16 14:32:42 UTC
Hi,

This was recently reported in launchpad:

https://bugs.launchpad.net/ubuntu/+source/samba/+bug/539791

If you have any questions please let me know.

Regards
chuck
Comment 1 Jelmer Vernooij 2010-03-17 09:40:30 UTC
The bug description, from the Launchpad page:

In Karmic, it is possible to mount a CIFS filesystem from a DFS referral, but it would fail when using Kerberos authentication. Witness:

WARTHOGS\carol@karmic-desktop:~$ mount.cifs //warthogs.biz/namespace1/firstshare share --verbose -o sec=krb5

mount.cifs kernel mount options: unc=//warthogs.biz\namespace1,user=carol,,,,,,,,,,domain=WARTHOGS,ver=1,sec=krb5,uid=288883801,gid=288883201,prefixpath=firstshare,ip=192.168.122.200
mount error(126): Required key not available
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

In the above example, //warthogs.biz/namespace1/firstshare is a DFS referral that points to //warthogs-adc/share1.

When not specifying "-o sec=krb5" (or when using "-o sec=ntlm"), the mount complete just fine. It also work is I use the direct UNC path, //warthogs-adc/share1, with "-o sec=krb5", so it does really appear that the problem manifest itself only when both conditions are true (the UNC path is a DFS referral *and* we are authenticating using Kerberos).

uname -r: 2.6.31-20-generic

keyutils 1.2-10 (no change to /etc/request-key.conf)

likewise-open5 5.0.3991.1+krb5-0ubuntu2, AD functional level Win2k3 domain joined.
Comment 2 Jeff Layton 2010-03-17 09:48:54 UTC
It looks like the problem is that the KDC has no principal called

    cifs/warthogs.biz

...or...

    host/warthogs.biz

...there are 2 ways to resolve this:

1) fix the DFS referral such that it points to a hostname that matches the host portion of the service tickets (i.e. warthogs-adc instead of warthogs.biz).

2) add cifs/warthogs.biz or host/warthogs.biz to the KDC and teach the server to accept them
Comment 3 Etienne Goyer 2010-03-17 10:00:27 UTC
Jeff,

The DFS referral do not resolve to warthogs.biz. it resolve to the actual UNC path where the service live.  In this case, it is //warthogs-adc/share1.  The principal for cifs/warthogs-adc does actually exist on the KDC (an ADC, actually); I can connect to the service with "smbclient -k //warthogs-adc/share1" no problem.

It really does seem that mount.cifs (or perhaps cifs.upcall) is not even resolving the DFS referral in the first place.  But then, only when doing Kerberos authentication, as it work just fine when using "-o sec=ntlm" option to mount.cifs.



Comment 4 Jeff Layton 2010-03-17 10:13:43 UTC
Ahh ok, I see -- you're mounting the DFS referral itself, but are using a different. Indeed, what you're trying to do won't work.

Why? Because in order to fetch the DFS referral, you need to first authenticate to the server that will give you that referral (warthogs.biz in this case). That initial authentication is failing here

To fix that you'll need to do "option 2" from comment #2. Add principals for warthogs.biz to the kdc, and teach warthogs.biz to accept those principals in a session setup request.
Comment 5 Etienne Goyer 2010-03-17 10:26:50 UTC
Sorry Jeff, you lost me there.  warthogs.biz is the domain; warthogs-adc.warthogs.biz is the ADC for that domain.  In this case, it happen to host the service that is the endpoint for the //warthogs.biz/namespace1/firstshare DFS referral.  My understanding is that mount.cifs (or cifs.upcall, depending on how you look at it) needs to retrieve the key for the endpoint (that is, cifs/warthogs-adc).  At least, AFAICT, that is what smbclient does, and it works.
Comment 6 Jeff Layton 2010-03-17 10:33:37 UTC
I'm sorry, you've lost me too...

The log you posted showed this as your mount command:

   mount.cifs //warthogs.biz/namespace1/firstshare share --verbose -o sec=krb5

Now, cifs will grab the hostname portion of the UNC and then attempt to use that to create a service principal that it should use to authenticate. That is failing here -- there is no cifs/warthogs.biz service principal.

You seem to be saying that the client should be using the principal cifs/warthogs-adc instead. My question to you is -- how should the client be determining this?

Comment 7 Etienne Goyer 2010-03-17 10:40:12 UTC
By resolving the DFS referral.  Or am I missing something?
Comment 8 Jeff Layton 2010-03-17 10:50:22 UTC
...and how exactly will we get the DFS referral w/o authenticating first?

Getting a DFS referral means connecting to the IPC$ share, which means that you need a valid SMB session. To get a valid SMB session, you have to authenticate.
Comment 9 Etienne Goyer 2010-03-17 10:56:00 UTC
That part, unfortunately, I cannot help with: that's where my understanding of the problem stops. :)

Basically, I was kinda expecting from mount.cifs to behave like smbclient.  In the case of smbclient, given a UNC that is a DFS referral, it does whatever needs to be done behind the scene and connect to the service.  If that can be done, great.  If it can't, well ... it's good to know.
Comment 10 Jeff Layton 2010-03-17 11:00:30 UTC
I'm not exactly clear on what smbclient actually does. Perhaps you could attach a network capture of smbclient connecting to the share in your setup?
Comment 11 Etienne Goyer 2010-03-17 16:19:12 UTC
Created attachment 5508 [details]
tcpdump capture when running "smbclient -k -c showconnect //warthogs.biz/testnamespace/firstshare"

If you would prefer a different capture, let me know.  Thanks for looking into that!
Comment 12 Jeff Layton 2010-03-17 16:53:56 UTC
Ok, I think I see what's happening. The server in this case is sending a principal name back to the client in the Negotiate Protocol response. smbclient is using that to get a ticket name.

Note that this is really bad behavior by both the server and client. The client, in particular since you're essentially trusting the server to tell you what service principal to use. This allows an attacker to potentially spoof DNS and redirect the connection to a server that he/she controls.

Not trusting that info was a conscious decision. You can read the thread from a couple of years ago here:

http://lists.samba.org/archive/linux-cifs-client/2008-August/003348.html

The correct solution is to fix it so that your KDC holds service principals for all possible hostnames.

...now, that said, I'm not 100% opposed to patches that turn on this behavior as an option. I'm not interested in doing that work, but if you or someone else wants to take it on, I'd be willing to help review them.
Comment 13 Etienne Goyer 2010-03-18 09:32:19 UTC
"The correct solution is to fix it so that your KDC holds service principals for
all possible hostnames."

AFAICT, that would not address the problem as I experience it.  The problem being that mount.cifs does not even resolve DFS referral in the first place.  The fact that cifs.upcall cannot figure out which principal to get a ticket for is just a side effect of that.

You actually put the finger on it back in August 2008:

    http://lists.samba.org/archive/linux-cifs-client/2008-August/003370.html


I am in exactly the same situation as this other participant in the thread in question:

    http://lists.samba.org/archive/linux-cifs-client/2008-August/003368.html


In the end, I understand the arguments against using the mechListMIC to figure out what server/service to connect to, but the discrepancy in behavior between mounting a DFS referral using Kerberos vs using NTLM authentication remain pretty confusing to the user.  Intuitively, we would expect that authentication mechanisms are interchangeable and should not interfere with the working of mount.cifs itself.  This limitation of mount.cifs wrt to Kerberos authentication and DFS referral break that assumption.  I guess, at the very least, it has to be documented somehow.

Incidentally, if I had a mechanism to resolve DFS referral reliably, I could use that to prepare the target service UNC to pass to mount.cifs.  That would work around this limitation of mount.cifs quite nicely and the problem would be moot, except I cannot find a tool to do that.  I tried using "smbclient -k -c showconnect //dfsroot/namespace/service/subdir", but it would silently discard the subdir part of the UNC.  I will probably file a bug against smbclient about that one.
Comment 14 Jeff Layton 2010-03-18 10:14:29 UTC
(In reply to comment #13)
> "The correct solution is to fix it so that your KDC holds service principals
> for
> all possible hostnames."
> 
> AFAICT, that would not address the problem as I experience it.  The problem
> being that mount.cifs does not even resolve DFS referral in the first place. 
> The fact that cifs.upcall cannot figure out which principal to get a ticket for
> is just a side effect of that.
>

AFAICT, it would. The problem here is that you have a server named "warthogs.biz", and you have no service principal "cifs/warthogs.biz" or "host/warthogs.biz". If you had those service principals in your KDC, and "warthogs.biz" (the fileserver) was set up to accept them, then this problem would not exist for you.

The comment that I made in that thread was wrong. There's no reason that krb5 and DFS can't coexist. krb5 simply requires that you use canonical hostnames everywhere, or have some means to inform programs that need to authenticate what those canonical hostnames are.

> Intuitively, we would expect that authentication
> mechanisms are interchangeable and should not interfere with the working of
> mount.cifs itself.

This intuition is also wrong. There's one rather glaring difference between NTLM and kerberos, and that is that kerberos involves hostnames. NTLM doesn't. If you want to use kerberos, then you have to accept its warts too. In this case, that means using hostnames and DNS properly.

FWIW, using the info in the mechListMIC is *really* a dead end anyway. Recent windows versions (starting with Vista I think) send a bogus name in that field to discourage its use.
Comment 15 Etienne Goyer 2010-03-18 10:32:58 UTC
Ok, I think there is a misunderstanding here: warthogs.biz is *not* an
hostname, it's the Active Directory *domain* name.
"//warthogs.biz/testnamespace" is a domain-based DFS namespace.  Sorry that was
not made clear enough from the start!

It happen that the A record for warthogs.biz actually point to the IP of
warthogs-adc, but that is only because warthogs-adc is the only ADC in this
test Active Directory setup.  It also happen that warthogs-adc host the service
being pointed to by the //warthogs.biz/testnamespace/firstshare, but that is
also just a coincidence.  It may just as well point to any server, and
cifs.upcall would still not know how to get the principal for the target,
simply because it does not know what the target is in the first place.

Either that, or I am utterly confused.
Comment 16 Jeff Layton 2010-03-19 08:55:24 UTC
(In reply to comment #15)
> Ok, I think there is a misunderstanding here: warthogs.biz is *not* an
> hostname, it's the Active Directory *domain* name.
> "//warthogs.biz/testnamespace" is a domain-based DFS namespace.  Sorry that was
> not made clear enough from the start!
> 

CIFS has no way to tell the difference between an AD domain name and a hostname. In practice, it treats either as a hostname.

> It happen that the A record for warthogs.biz actually point to the IP of
> warthogs-adc, but that is only because warthogs-adc is the only ADC in this
> test Active Directory setup.  It also happen that warthogs-adc host the service
> being pointed to by the //warthogs.biz/testnamespace/firstshare, but that is
> also just a coincidence.  It may just as well point to any server, and
> cifs.upcall would still not know how to get the principal for the target,
> simply because it does not know what the target is in the first place.
> 

...and there's the problem. You have a hostname that points to this server and clients that are using that hostname, but you have no krb5 service principals associated with that hostname.

If windows clients don't have the same problem, they're likely getting info about what principal to use via another source -- maybe LDAP?
Comment 17 Etienne Goyer 2010-03-19 09:19:01 UTC
I have no idea.  I do not understand the innards well enough to be able to tell.  All I know is that trying to mount a domain-based DFS referral UNC works when using NTLM authentication, and does not when using Kerberos.
Comment 18 Simo Sorce 2010-03-19 10:07:35 UTC
Just to complete the picture.
Probably what we need is some DNS smarts in the cifs upcall binary, so that it can find out what the right principal name to call in all situations where an alias is used.

Although I agree this has lower priority atm, as you can always provide the correct name and have no issues in the normal case.
Comment 19 Jeff Layton 2010-03-19 10:17:03 UTC
One possible workaround for you (assuming the PTR record for the address points to the right hostname) is to use cifs.upcall with the --trust-dns option. That'll make it reverse-resolve the IP address to get the hostname for the ticket. That's not as secure if DNS is compromised but you may find it helpful.
Comment 20 Jeff Layton 2010-03-19 10:17:58 UTC
(In reply to comment #18)
> Just to complete the picture.
> Probably what we need is some DNS smarts in the cifs upcall binary, so that it
> can find out what the right principal name to call in all situations where an
> alias is used.
> 
> Although I agree this has lower priority atm, as you can always provide the
> correct name and have no issues in the normal case.
> 

The question of course is "how to determine the name"? I'm not opposed to adding smarts to cifs.upcall to do this, I'm just not clear on what those smarts should do.

Comment 21 Simo Sorce 2010-03-19 10:24:47 UTC
(In reply to comment #20)

> The question of course is "how to determine the name"? I'm not opposed to
> adding smarts to cifs.upcall to do this, I'm just not clear on what those
> smarts should do.

In a moment of wisdom I say the smarts are TBD and will move on, unfortunately I have no time to work on it right now either :)

Comment 22 Etienne Goyer 2010-03-19 10:26:07 UTC
"The question of course is "how to determine the name"? I'm not opposed to
adding smarts to cifs.upcall to do this, I'm just not clear on what those
smarts should do."

Perhaps it is not possible for some reason, but whichever mechanism get used
when doing NTLM auth miht be a possibility.


"One possible workaround for you (assuming the PTR record for the address
points to the right hostname) is to use cifs.upcall with the --trust-dns option.
That'll make it reverse-resolve the IP address to get the hostname for the
ticket. That's not as secure if DNS is compromised but you may find it
helpful."

That works wonderfully in my test setup here, but in the real world, I cannot
make the assumption that A and PTR records align.  It is pretty common that
they don't in productio
Comment 23 Jeff Layton 2010-03-19 10:33:39 UTC
(In reply to comment #22)
> "The question of course is "how to determine the name"? I'm not opposed to
> adding smarts to cifs.upcall to do this, I'm just not clear on what those
> smarts should do."
> 
> Perhaps it is not possible for some reason, but whichever mechanism get used
> when doing NTLM auth miht be a possibility.
>

As I said above, NTLM doesn't care one bit about the hostname. It's quite a different authentication scheme from krb5.

Comment 24 Jeff Layton 2012-04-06 12:00:26 UTC
At this point, I don't see this as fixable. Your DNS configuration is just broken, and what smbclient does is just plain unsafe. There have been some fixes recently to cifs DFS and krb5 handling which may help this, but I wouldn't count on it.