Bug 6563 - ntlm_auth returns invalid NT_KEY
ntlm_auth returns invalid NT_KEY
Status: RESOLVED FIXED
Product: Samba 3.3
Classification: Unclassified
Component: Winbind
3.3.4
x86 Linux
: P3 critical
: ---
Assigned To: Kai Blin
Samba QA Contact
:
Depends on: 7945
Blocks:
  Show dependency treegraph
 
Reported: 2009-07-17 15:25 UTC by Mike Loosbrock (550-5.1.1 The email account that you tried to reach does not exist)
Modified: 2014-11-13 02:49 UTC (History)
15 users (show)

See Also:


Attachments
winbind 3.0.31 success ntlm-auth output (194 bytes, text/plain)
2009-11-25 07:18 UTC, Rajesh Kumar G
no flags Details
samba 3.0.31 winbind success log (173.42 KB, text/plain)
2009-11-25 07:19 UTC, Rajesh Kumar G
no flags Details
winbind 3.3.2 failue log (411.56 KB, text/plain)
2009-11-25 07:19 UTC, Rajesh Kumar G
no flags Details
winbind 3.3.2 failue ntlm-auth output (229 bytes, text/plain)
2009-11-25 07:20 UTC, Rajesh Kumar G
no flags Details
NTLMv1 challenge response creation script (1.79 KB, text/plain)
2009-11-26 03:57 UTC, Rajesh Kumar G
no flags Details
Patch to add --force-samlogon option to winbindd (3.34 KB, application/octet-stream)
2010-08-02 12:02 UTC, Rob Colantuoni
no flags Details
Patch to add 'winbind:forcesamlogon' parameter (1.03 KB, patch)
2010-08-03 06:32 UTC, Rob Colantuoni
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Loosbrock (550-5.1.1 The email account that you tried to reach does not exist) 2009-07-17 15:25:09 UTC
I have three FreeRADIUS servers that use ntlm_auth->winbind to authenticate 802.1x supplicants via PEAP/MSCHAPv2.

Server A -> Debian, samba/winbind 3.0.30
Server B -> Debian, samba/winbind 3.0.30
Server C -> Debian, samba/winbind 3.3.4

All three servers have this /etc/samba/smb.conf file:

[global]
  security          = ads
  realm             = BU.AC.BETHEL.EDU
  encrypt passwords = yes
  workgroup = BU
  idmap uid         = 10000-20000
  idmap gid         = 10000-20000
  disable netbios   = yes
  server string     = %h
  dns proxy         = no
  syslog only       = yes

All three servers can successfully authenticate as follows:

$ sudo ntlm_auth --username=testuser --domain=BU
password: 
NT_STATUS_OK: Success (0x0)

All three servers can successfully do NTLMv2 Session authentication: (nt-response and challenge arguments are from FreeRADIUS debug output)

A >$ sudo ntlm_auth --username=testuser --domain=BU --request-nt-key --nt-response=6a35c5807ace4a4df06245523692c6084c632ed1d72044f9 --challenge=f93a04b51ab1c842
NT_KEY: CC5771309789EBBD4E08B19C3D7FF1ED
---
B >$ sudo ntlm_auth --username=testuser --domain=BU --request-nt-key --nt-response=6a35c5807ace4a4df06245523692c6084c632ed1d72044f9 --challenge=f93a04b51ab1c842
NT_KEY: CC5771309789EBBD4E08B19C3D7FF1ED
---
C >$ sudo ntlm_auth --username=testuser --domain=BU --request-nt-key --nt-response=6a35c5807ace4a4df06245523692c6084c632ed1d72044f9 --challenge=f93a04b51ab1c842
NT_KEY: 923D1337481A1F4A6B4723915A860A15

Note that Server C returns a different NT_KEY, which is causing PEAP/MSCHAPv2 clients to abort the authentication attempt.

I'm not a samba or PEAP/MSCHAPv2 expert, but I've observed that given the same --username, --domain, --nt-repsonse and --challenge parameters,

1.) ntlm_auth on both Servers A and B always returns the same NT_KEY.
2.) ntlm_auth on both Servers A and B always returns an NT_KEY that allows PEAP/MSCHAPv2 clients to successfully authenticate.
3.) ntlm_auth on Server C never returns the same NT_KEY as ntlm_auth on Servers A and B.
4.) ntlm_auth on Server C always returns a different NT_KEY after restarting the winbindd daemon.
5.) ntlm_auth on Server C never returns an NT_KEY that allows PEAP/MSCHAPv2 clients to successfully authenticate.

Based on observation 4 above, my uneducated guess is that winbindd is mangling the NT_KEY.

I had this problem sometime last fall with samba/winbind 3.2 on Debian and resolved it by downgrading back to samba/winbind 3.0.30.
Comment 1 Guenther Deschner 2009-07-17 17:03:30 UTC
Kai, I'm away for two weeks. Could you please have a look ?
Comment 2 Kai Blin 2009-07-19 09:08:21 UTC
So 3.2.0 has this problem as well? I'll look into that the coming week.
Comment 3 Mike Loosbrock (550-5.1.1 The email account that you tried to reach does not exist) 2009-07-20 14:26:01 UTC
(In reply to comment #2)
> So 3.2.0 has this problem as well? I'll look into that the coming week.
> 
Looks like it. I'm seeing the same behavior observed in 3.3.4 (Debian testing 2:3.3.4-1) in 3.2.5 (Debian stable 2:3.2.5-4lenny6) as well.

If it helps any, I'm a FreeRADIUS user. There have been multiple reports of this type of behavior on the FreeRADIUS lists, and Alan DeKok from FreeRADIUS posted about this issue on the samba-technical list back in May.

Alan's post:
http://lists.samba.org/archive/samba-technical/2009-May/064892.html
Comment 4 Kai Blin 2009-07-21 16:51:42 UTC
Interesting. I still need to set up freeradius on a test box to play with that. So far I can't reproduce the error, but I didn't try with user-supplied response to the challenge yet.
Comment 5 Kai Blin 2009-07-27 09:55:42 UTC
Is there any way to test radius without having a radius-capable switch around? I've set up a freeradius server, but I seem to have no way to trigger an ntlm_auth call.
Comment 6 Mike Loosbrock (550-5.1.1 The email account that you tried to reach does not exist) 2009-07-27 21:58:07 UTC
(In reply to comment #5)
> Is there any way to test radius without having a radius-capable switch around?
> I've set up a freeradius server, but I seem to have no way to trigger an
> ntlm_auth call.

One way is to use the eapol_test utility that comes with wpa_supplicant.  It basically plays the part of both the 802.1x supplicant and the switch during the EAP exchange with the RADIUS server.

It appears most distributions don't package eapol_test with wpa_supplicant, so you may have to build it yourself (I had to with Debian). Here are detailed instructions on building and using it:

http://deployingradius.com/scripts/eapol_test/

I'm sure this is beginning to stray outside the normal boundaries of samba development, so let me know if there is anything else I can do to help you out with this.
Comment 7 Mike Loosbrock (550-5.1.1 The email account that you tried to reach does not exist) 2009-08-07 14:34:24 UTC
Is there anything more I can do to help?
Comment 8 Kai Blin 2009-08-08 02:07:02 UTC
It seems like my development system doesn't come with a TLS-enabled freeradius server, so I can't turn on peap, so I can't test the MSCHAP auth. I'm planning to add a feature to ntlm_auth to allow me to test this without having to run a radius server. This might still take a bit, sorry about that.
Comment 9 Mike Loosbrock (550-5.1.1 The email account that you tried to reach does not exist) 2009-08-10 20:18:17 UTC
Yeah, I know in Debian FreeRADIUS does not ship with openssl support due to licensing issues. Having NTLMv2 Session authentication testing capabilities built right into ntlm_auth would be awesome.
Comment 10 Kai Blin 2009-09-16 13:29:19 UTC
I finally managed to get a setup where I can specify the session key the DC hands me, and I can't see the current master tree mangling the session key. What DC is this against?
Comment 11 Mike Loosbrock (550-5.1.1 The email account that you tried to reach does not exist) 2009-09-21 15:03:45 UTC
My systems administration team tells me Active Directory 2008 running in 2003 Native mode. Does that answer your question?
Comment 12 Rajesh Kumar G 2009-11-25 07:18:22 UTC
Created attachment 5004 [details]
winbind 3.0.31 success ntlm-auth output
Comment 13 Rajesh Kumar G 2009-11-25 07:19:01 UTC
Created attachment 5005 [details]
samba 3.0.31 winbind success log
Comment 14 Rajesh Kumar G 2009-11-25 07:19:40 UTC
Created attachment 5006 [details]
winbind 3.3.2 failue log
Comment 15 Rajesh Kumar G 2009-11-25 07:20:03 UTC
Created attachment 5007 [details]
winbind 3.3.2 failue ntlm-auth output
Comment 16 Rajesh Kumar G 2009-11-25 07:20:51 UTC
This issue is consistently reproducible with the following setup.

Windows 2008 AD parent domain controller in Windows 2003 mode.
Windows 2003 child domain controller in Windows 2003 mode.
client machine joined to Windows 2003 DC using samba 3.3.2+.

executing ntm_auth with challenge/response from freeradius returns invalid NT_KEY


The issue does not occur if machine is joined to Windows 2003 DC using samba 3.0.31. executing ntlm_auth returns correct NT_KEY.


I'm attaching winbind -d12 level logs with the following test setup.


Windows 2008 parent domain:

Domain - EIGHTAD6.TESTING.COM
DC     - EIGHTAD-DC.EIGHTAD6.TESTING.COM (in windows 2003 mode)

Windows 2003 child domain:

Domain - CHILD03.EIGHTAD6.TESTING.COM
DC     - NORMA.CHILD03.EIGHTAD6.TESTING.COM (in windows 2003 mode)


Two client machines one running samba 3.3.2 and another with samba 3.0.31 both joined to  NORMA.CHILD03.EIGHTAD6.TESTING.COM, the child domain.

They try to authenticate a user 'test' in the child domain using ntlm_auth. In the client machine with samba 3.0.31 the NT_KEY returned is proper while with samba 3.3.2 the NT_KEY looks different and wrong.

Note that if it is a normal setup, i.e

a normal stand alone windows 2003 domain (in any mode) or
a normal stand alone windows 2008 domain (in any mode)

then samba 3.3.2 works correctly without issues.
Comment 17 Kai Blin 2009-11-25 15:28:35 UTC
Are you also using Radius? I'm still looking for a good way to test this without having to set up a Radius server.
Comment 18 Rajesh Kumar G 2009-11-26 03:57:02 UTC
(In reply to comment #17)
> Are you also using Radius? I'm still looking for a good way to test this
> without having to set up a Radius server.
> 

you can use the attached python script to generate challange and nt-response strings given a username and password.

Usage: ntlm_gen.py <username> <password>

like,

[rajkg@localhost ntlm]$ ./ntlm_gen.py test test123#
Challenge :  cdf59226328c2cb3
Response  :  a91ac7059628da6fb07566035f619855b1857b36255f16f0
NT_KEY    :  870135ABD69B8F916F728794CE42D64E

The script works with python 2.4. You will need the following extra python modules also

http://barryp.org/software/py-md4/

http://twhiteman.netfirms.com/des.html


Comment 19 Rajesh Kumar G 2009-11-26 03:57:52 UTC
Created attachment 5015 [details]
NTLMv1 challenge response creation script
Comment 20 Kai Blin 2009-12-08 09:57:53 UTC
Ok, I just got around to set up the following network to confirm this:

WIN2K8.HOME.KBLIN.ORG (Win2k8 R2 DC in 2003 mode as forest root)
^
|
WIN2K3-FOREST.WIN2K8.HOME.KBLIN.ORG (Win2k3 DC subforest, child of WIN2K8 forest)
^                           ^
|                           |
S3.0 member server   and    S3.3 member server

I can wbinfo -K/-a authenticate users against both the WIN2K8 and the WIN2K3-FOREST domains. using ntlm_auth, I can authenticate from both samba servers. However, as soon as I add a challenge, authentication fails on both systems.

What's your smb.conf settings? Did you set up anything special on the DC?
Comment 21 Rajesh Kumar G 2009-12-09 04:36:54 UTC
(In reply to comment #20)
> Ok, I just got around to set up the following network to confirm this:
> 

There's nothing special in the smb.conf

[global]
        realm = CHILD03.EIGHTAD6.TESTING.COM
        workgroup = CHILD03

        server string = Samba Server Version %v

        security = ads
        passdb backend = tdbsam


Check if your domain controller's security policy is set to reject NTLM authentications, In the Domain Controller Security Policy

Security Settings --> Local Policies --> Security Options --> Network Security: LAN Manager authentication level

should NOT be set to 

Send NTLMv2 response only\refuse LM & NTLM

Did you use the script that I attached before to generate the challenge/response? I have reconfirmed that the script generates correct challenge/response.


Comment 22 Kai Blin 2009-12-16 05:57:01 UTC
I don't have any policies set that refuse NTLM authentication. Also, if I don't specify a challenge, authentication via both wbinfo and ntlm_auth seems to work fine. ntlm_auth using the squid helper mode negotiates the flags 0x60088205 for both 3.0 and 3.3.
Comment 23 Mike Loosbrock (550-5.1.1 The email account that you tried to reach does not exist) 2010-02-03 09:30:36 UTC
Kia, this issue appears to have cleared up at or around 3.4.3.

On my Debian Testing machine, the samba packages (version 2:3.4.3-2) no longer exhibit the invalid NT_KEY behavior.  And to my knowledge, our Windows Domain environment has not changed since I last tested.
Comment 24 Norbert Wegener (550 5.1.1; User unknown in relay recipient table) 2010-07-02 10:52:48 UTC
On Ubuntu 10.04 with all (samba) updates applied the problem still exists. What Freeradius and eapol_test show can be seen at: http://tinyurl.com/36wn5lz
If required I will provide more information.
Comment 25 Norbert Wegener (550 5.1.1; User unknown in relay recipient table) 2010-07-02 11:51:08 UTC
Forgot to mention: Samba is Version 3.4.7
Comment 26 Alan DeKok 2010-07-04 02:11:22 UTC
There have been a number more reports of this on the freeradius-users list.  The only common thread
seems to be that 3.0.x works, and later versions work (or not) sporadically.
Comment 27 Aiko Barz 2010-07-19 05:21:27 UTC
This is my first freeradius installation. It took me days, to figure out, that ntlm_auth is the villain...
Comment 28 Jean-Paul Chapalain 2010-07-20 02:52:39 UTC
Where are we to resolve this bug. I'm stuck in the implementation of a infrastruture authentication EAP-PEAP with FreeRADIUS.

Regards.
Comment 29 Volker Lendecke 2010-07-20 03:10:50 UTC
Wild guess: Can this be a duplicate of bug 7568 (or the other way round)?

Volker
Comment 30 Alan DeKok 2010-07-20 03:17:15 UTC
It's quite possibly the same as bug #7568.  The ntlm_auth problem has been reported sporadically on the freeradius list for about 2 years, IIRC.
Comment 31 Kai Blin 2010-07-20 03:25:59 UTC
(In reply to comment #24)
> On Ubuntu 10.04 with all (samba) updates applied the problem still exists. What
> Freeradius and eapol_test show can be seen at: http://tinyurl.com/36wn5lz

The real URL is: http://www.wegener-net.de/freeradius/ just in case tinyurl decides to stop forwarding the URL. 

Comment 32 Volker Lendecke 2010-07-23 18:19:55 UTC
diff --git a/source3/winbindd/winbindd_pam.c b/source3/winbindd/winbindd_pam.c
index bf113e8..277316f 100644
--- a/source3/winbindd/winbindd_pam.c
+++ b/source3/winbindd/winbindd_pam.c
@@ -1785,6 +1785,7 @@ enum winbindd_result winbindd_dual_pam_auth_crap(struct winbindd_domain *domain,
                        goto done;
                }
 
+               domain->can_do_samlogon_ex = false;
                logon_fn = domain->can_do_samlogon_ex
                        ? rpccli_netlogon_sam_network_logon_ex
                        : rpccli_netlogon_sam_network_logon;

solves it for me.

This is the wrong fix, but someone with enough patching skills might want to ack/nack it.

Volker
Comment 33 Norbert Wegener (550 5.1.1; User unknown in relay recipient table) 2010-07-26 03:01:42 UTC
I can confirm that the patch works for me with samba-latest.
Thank you. 
Will this fix - although "wrong" make it into future releases?

Norbert
Comment 34 Kai Blin 2010-07-28 16:41:56 UTC
Afraid I still can't reproduce this:
Current git master against win2k3:

root@samba-client:~/test# ./ntlm_gen.py administrator secret
Challenge :  3560f2a07f2f72f8
Response  :  cf49f03eebeeac7ffdb080cca483dd95e9e85e69cd86a08b
NT_KEY    :  25EE06323AC15264CF82397711EF38DF

root@samba-client:~/test# ntlm_auth --username=administrator --domain=win2k3 --request-nt-key --nt-response=bd93da6a2274452289cfb5556efafee9a97e04d5e94718f6 --challenge=618df5728c1b2170
NT_KEY: 25EE06323AC15264CF82397711EF38DF

This is on an Ubuntu 10.04 box against a plain Win2k3 AD DC, haven't tried the win2k3 trusting win2k8 thing yet.
Comment 35 Rob Colantuoni 2010-08-02 11:59:54 UTC
Expanding on Volker's suggestion, I've created a patch that adds a --force-samlogon option to winbindd. If winbindd is run with this option, it will skip the attempt to use SamLogonEx and this results in the proper NT_KEY being returned.

This patch can be used against samba_latest, which is currently 3.5.4.

Rob Colantuoni
Comment 36 Rob Colantuoni 2010-08-02 12:02:17 UTC
Created attachment 5880 [details]
Patch to add --force-samlogon option to winbindd

This will resolve issues resulting in winbind using SamLogonEx returning an NT_KEY that the client views as invalid. Thanks to Volker for the pointer to the issue!
Comment 37 Volker Lendecke 2010-08-02 14:19:24 UTC
Thanks for looking at this. However, we'd rather make that a parametric option. Look at lp_parm_bool(). Command-line parameters are not the Samba-way to force obscure special behaviour.

Volker
Comment 38 Rob Colantuoni 2010-08-02 14:24:48 UTC
Volker,

Will do... that makes more sense to me as well, I wasn't sure if it would be more/less acceptable than a command-line option.

Rob
Comment 39 Rob Colantuoni 2010-08-03 06:32:59 UTC
Created attachment 5882 [details]
Patch to add 'winbind:forcesamlogon' parameter

Changed functionality from command-line option to parameter
Comment 40 Rob Colantuoni 2010-08-03 06:35:01 UTC
Volker,

I created a parameter version and attached the patch. Is there somewhere these obscure parameters are documented?

Rob
Comment 41 Guenther Deschner 2010-08-10 06:29:16 UTC
Do you have a chance to test the patch provided in 
https://bugzilla.samba.org/show_bug.cgi?id=7568 ?

I am convinced it will resolve this issue as well.
Comment 42 Andrew Bartlett 2010-08-11 20:20:10 UTC
I agree, the patch in bug 7568 is a plausible way to fix this patch.  The failure to sign in that  bug is the exact same thing as an invalid NT Key. 

This shows up because the SamLogonEx call does not do credential chaining, and so we don't notice that we are actually operating with a different implied session key. 
Comment 43 Andrew D 2010-08-17 06:23:17 UTC
Will a patch be released for 3.4.x soon?

This also affects mpd authing against freeradius and using ntlm_auth on samba 3.2, 3.3 and 3.4 on freebsd 8.1.
Comment 44 Guenther Deschner 2010-08-17 06:58:44 UTC
(In reply to comment #43)
> Will a patch be released for 3.4.x soon?
> 
> This also affects mpd authing against freeradius and using ntlm_auth on samba
> 3.2, 3.3 and 3.4 on freebsd 8.1.

A patch for 3.4 is attached to the bug mentioned in the previous comments, you can use this link:
https://bugzilla.samba.org/attachment.cgi?id=5894

The next 3.4 release will have this patch included, though there is no planned release date for 3.4.9 yet. There won't be new releases for 3.3 and 3.2, but you will find a patch for 3.3 in https://bugzilla.samba.org/show_bug.cgi?id=7568 as well. 3.2 should not be affected by this.
Comment 45 Myles Fenton 2010-11-07 20:48:45 UTC
Greetings Guenther,
(In reply to comment #44)
Could you please confirm that the patches in comment #44 are now part of the 3.5.6-43.el5.i386.rpm's located at?:
ftp://ftp.sernet.de/pub/samba/3.5/rhel/5/i386/

At Monash University we are trying to roll out PEAP-MSCHAPv2 Wireless using Radiator Radius against a Win2008R2 AD. Using Samba 3.4.9 the User-Session-key returned from ntlm_auth changes to a bogus key after between 100 and 1000's authentications. Restarting windbind fixes the problem for another couple of minutes but this is not a suitable solution for a production environment. We hope the problem is resolved by your patches in comment #44 above.

Best regards,

Myles
Comment 46 Guenther Deschner 2010-11-08 03:31:00 UTC
(In reply to comment #45)
> Greetings Guenther,
> (In reply to comment #44)
> Could you please confirm that the patches in comment #44 are now part of the
> 3.5.6-43.el5.i386.rpm's located at?:
> ftp://ftp.sernet.de/pub/samba/3.5/rhel/5/i386/
> 
> At Monash University we are trying to roll out PEAP-MSCHAPv2 Wireless using
> Radiator Radius against a Win2008R2 AD. Using Samba 3.4.9 the User-Session-key
> returned from ntlm_auth changes to a bogus key after between 100 and 1000's
> authentications. Restarting windbind fixes the problem for another couple of
> minutes but this is not a suitable solution for a production environment. We
> hope the problem is resolved by your patches in comment #44 above.
> 
> Best regards,
> 
> Myles

Yes, 3.5.6 has all necessary fixes for this issue. Unless the sernet packages do contain other changes, it should just work with those packages.
Comment 47 Greg Bock 2010-11-08 18:25:36 UTC
I'm still seeing the issue after patching samba 3.4.9 with the patch from comment #44. Server is FreeBSD 6 with a current ports tree. I've currently worked around the issue by also adding the patch and parameter from comment #39 but I was under the impression that the patch from bug 7568 was all that was needed.
Comment 48 Petr Karpenko 2010-11-16 13:15:40 UTC
(In reply to comment #39)
> Created an attachment (id=5882) [details]
> Patch to add 'winbind:forcesamlogon' parameter
> 
> Changed functionality from command-line option to parameter
> 
After compiling Samba 3.5.6 with this patch where I should use this parameter?
Comment 49 Rob Colantuoni 2010-11-16 13:41:36 UTC
(In reply to comment #48)
> (In reply to comment #39)
> > Created an attachment (id=5882) [details] [details]
> > Patch to add 'winbind:forcesamlogon' parameter
> > 
> > Changed functionality from command-line option to parameter
> > 
> After compiling Samba 3.5.6 with this patch where I should use this parameter?
> 
Add:
winbind:forcesamlogon = True

to the Global section of smb.conf

Comment 50 Petr Karpenko 2010-11-16 14:11:07 UTC
(In reply to comment #49)
> (In reply to comment #48)
> > (In reply to comment #39)
> > > Created an attachment (id=5882) [details] [details] [details]
> > > Patch to add 'winbind:forcesamlogon' parameter
> > > 
> > > Changed functionality from command-line option to parameter
> > > 
> > After compiling Samba 3.5.6 with this patch where I should use this parameter?
> > 
> Add:
> winbind:forcesamlogon = True
> 
> to the Global section of smb.conf
> 
Thank you! After adding this parameter everything seems to be fine. But immediately after restarting winbindd I've got situation oppisite to mentioned in #45. Here is a Freeradius output:

Found Auth-Type = MSCHAP
+- entering group MS-CHAP {...}
[mschap] Told to do MS-CHAPv2 for tst-tmp with NT-Password
[mschap] 	expand: %{Stripped-User-Name} -> 
[mschap] 	... expanding second conditional
[mschap] WARNING: Deprecated conditional expansion ":-".  See "man unlang" for details
[mschap] 	expand: %{User-Name:-None} -> tst-tmp
[mschap] 	expand: --username=%{%{Stripped-User-Name}:-%{User-Name:-None}} -> --username=tst-tmp
[mschap]  mschap2: 80
[mschap] 	expand: --challenge=%{mschap:Challenge:-00} -> --challenge=263fdcbaa5eab23d
[mschap] 	expand: --nt-response=%{mschap:NT-Response:-00} -> --nt-response=6ac9670c7768f57359f352231a3d2082d89126fcaecc80cd
Exec-Program output: Access denied (0xc0000022) 
Exec-Program-Wait: plaintext: Access denied (0xc0000022) 
Exec-Program: returned: 1
[mschap] External script failed.
[mschap] FAILED: MS-CHAP2-Response is incorrect
++[mschap] returns reject
Failed to authenticate the user.

But after couple minutes without any action from my side we could see following situation:

+- entering group MS-CHAP {...}
[mschap] Told to do MS-CHAPv2 for tst-tmp with NT-Password
[mschap] 	expand: %{Stripped-User-Name} -> 
[mschap] 	... expanding second conditional
[mschap] WARNING: Deprecated conditional expansion ":-".  See "man unlang" for details
[mschap] 	expand: %{User-Name:-None} -> tst-tmp
[mschap] 	expand: --username=%{%{Stripped-User-Name}:-%{User-Name:-None}} -> --username=tst-tmp
[mschap]  mschap2: ab
[mschap] 	expand: --challenge=%{mschap:Challenge:-00} -> --challenge=7a70df2454d1819d
[mschap] 	expand: --nt-response=%{mschap:NT-Response:-00} -> --nt-response=fad0430ecb49808bd70923bdb904eebac0dc0ad1cfcffa40
Exec-Program output: NT_KEY: 6F76CE716038F6AA2B18FDBB203BCF4B 
Exec-Program-Wait: plaintext: NT_KEY: 6F76CE716038F6AA2B18FDBB203BCF4B 
Exec-Program: returned: 0
[mschap] adding MS-CHAPv2 MPPE keys
++[mschap] returns ok
Comment 51 Lawrence Troup 2010-11-29 04:44:18 UTC
I was seeing this issue on my server (runnning Ubuntu LucidLynx) - ntlm_auth was returning invalid NT_Keys, causing MS-CHAP authentication failure.

I rebuilt winbind with the patch attached to this bug, and set the winbind:forcesamlogon option in my Samba config file. This fixed the problem completely - now ntlm_auth is generating valid NT_Keys, and authentication is succeeding.

Comment 52 Stefan Metzmacher 2011-03-08 08:28:18 UTC
Can you test if 3.5.8 fixes your problem too? There we don't need a special option,
Comment 53 Guenther Deschner 2011-05-18 13:05:55 UTC
any new testing results with 3.5.8 ?

note we are not going to add the "winbind:forcesamlogon" patch but instead just close as fixed in current version.
Comment 54 Ryan Steinmetz (550 5.1.1 unknown) 2011-06-01 23:44:20 UTC
Using 3.5.8 and ntlm_auth, it appears as if the NT_KEY returned matches that which is returned on a box with samba 3.0.37.  I have yet to test this with FreeRADIUS, but the fact that the NT_KEY is identical on both machines leads me to believe it will work fine.
Comment 55 Ryan Steinmetz (550 5.1.1 unknown) 2011-11-08 21:56:53 UTC
I have been running 3.5.8-3.5.9 for a handful of months now with FreeRADIUS without issue.  However, recently, one of the instances started consistently returning an invalid NT_KEY.  I was not able to correlate this to an issue with he domain controller that Samba was communicating with.  I'm currently lacking in additional details, however, restarting winbindd caused it to function correctly again.
Comment 56 Russell Jackson 2011-12-19 20:53:24 UTC
(In reply to comment #55)
> I have been running 3.5.8-3.5.9 for a handful of months now with FreeRADIUS
> without issue.  However, recently, one of the instances started consistently
> returning an invalid NT_KEY.  I was not able to correlate this to an issue with
> he domain controller that Samba was communicating with.  I'm currently lacking
> in additional details, however, restarting winbindd caused it to function
> correctly again.

Chiming in with a me too.

We've also suddenly have had problems with this again (sporadically) after having run 3.5.8 problem free since June. Restarting winbind fixed the problem initially, but we had a relapse a day later. A second restart fixed it yet again and has continued to work for a few days now without issues. Nothing has changed on the radius/samba side in terms of software or configuration.
Comment 57 David Zych 2012-02-03 19:35:02 UTC
I'm seeing the same problem after upgrading one of my RADIUS servers to RHEL6.

$ winbindd -V
Version 3.5.10-114.el6

I have a cron task which, every minute, invokes ntlm_auth in as close as possible to precisely the same way my Radiator config does (using a known set of credentials), verifies that the response I get back is correct, and restarts winbind immediately if it's wrong.

COMMAND
ntlm_auth --helper-protocol=ntlm-server-1 --require-membership-of='UOFI\groupxxx'

INPUT
Username: userxxx
NT-Domain: UIUC
LANMAN-Challenge: 0000000000000000
NT-Response: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Request-User-Session-Key: Yes
Request-LanMan-Session-Key: Yes
.

EXPECTED OUTPUT
Authenticated: Yes
User-Session-Key: 94xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.


Most of the time it works, but sporadically winbind will get itself into a state where ntlm_auth returns the wrong key.  In keeping with what others have reported, the automatic restart of winbind triggered by my cron task fixes the issue (at least temporarily).  So far I have had two failures this morning:

at 11:04am, actual output was:
Authenticated: Yes
LANMAN-Session-Key: F3xxxxxxxxxxxxxx
User-Session-Key: 67xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.

and at 12:44pm, actual output was:
Authenticated: Yes
LANMAN-Session-Key: 38xxxxxxxxxxxxxx
User-Session-Key: ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.

Some observations I find interesting:

1. The wrong keys are *different* each time.

2. The invocations that don't work also return a LANMAN-Session-Key, whereas this line is completely absent from the normal invocations.

3. I see the following output in log.wb-UOFI which corresponds in time to just before each instance where my cron task detected the error condition:

[2012/02/03 11:03:35.593559,  1] ../librpc/ndr/ndr.c:421(ndr_push_error)
  ndr_push_error(5): Bad character conversion
[2012/02/03 11:03:35.593702,  1] ../librpc/ndr/ndr.c:421(ndr_push_error)
  ndr_push_error(5): Bad character conversion
[2012/02/03 12:43:11.323491,  1] ../librpc/ndr/ndr.c:421(ndr_push_error)
  ndr_push_error(5): Bad character conversion
[2012/02/03 12:43:11.323601,  1] ../librpc/ndr/ndr.c:421(ndr_push_error)
  ndr_push_error(5): Bad character conversion

I am running with log level = 1.

4. The very first time this happened (not today), my test script was not yet checking for the correct key value, and did not automatically restart winbind.  This prevented a lot of people from logging onto wireless for a bit while I figured out what was going on, but also gave me the chance to observe that once winbind ended up in its broken state, ntlm_auth would consistently produce the *same* wrong key over and over again in response to repeated invocations of ntlm_auth with the same input, until winbind was restarted.

I hope this helps someone figure out what's going on.  This bug is definitely still alive and kicking in 3.5.10, and I don't want to have to go back to an ancient version of Samba if I can help it...
Comment 58 Robert Roll 2012-07-31 13:21:03 UTC
I have observed the same behavior as mentioned above with
version 3.5.6. I too created a check script to re-start the
server when it detected a problem. I'd say I saw this Winbindd
issue a couple of times a week ?.. 

One thing that I did notice early on was that it always
seemed to crop up in a situation when the server was very idle
for a time. For instance, when I was first working with Freeradius
I would bring up a server, do some testing, then go home and come
back in the morning. When I came back in the morning the authentication
would fail. I pretty early on learned it was related to this bug and
simply restarted Winbindd and continued my  Freeradius development.
This is NOT to mean the only situation I would find it to fail was when
it was idle, just that it was very consistent then.

My little check/restart script worked for well over a year. Then last
Friday (July 27,2012 around 4:40 p.m. MDT), something happened in
our environment and Winbindd seemed to start behaving badly even after
a restart. I could not get a good stable Freeradius server due to this
bug. I ended up installing the latest 3.5 version (currently 3.5.16).
This did seem to clear things up. I will say I have not experienced this
bug since, but it is only Tuesday early morning...

Robert
Comment 59 Khapare 2012-11-19 08:43:57 UTC
(In reply to comment #58)
> I have observed the same behavior as mentioned above with
> version 3.5.6. I too created a check script to re-start the
> server when it detected a problem. I'd say I saw this Winbindd
> issue a couple of times a week ?.. 
could you share that check script ? 

in my environment samba and winbind version are :
samba-winbind-3.5.8-68.el6.1.x86_64
samba-common-3.5.8-68.el6.1.x86_64
samba-winbind-clients-3.5.8-68.el6.1.x86_64
samba-3.5.8-68.el6.1.x86_64

This samba/winbind has been giving headache lately

> 
> 
> My little check/restart script worked for well over a year. Then last
> Friday (July 27,2012 around 4:40 p.m. MDT), something happened in
> our environment and Winbindd seemed to start behaving badly even after
> a restart. I could not get a good stable Freeradius server due to this
> bug. I ended up installing the latest 3.5 version (currently 3.5.16).
> This did seem to clear things up. I will say I have not experienced this
> bug since, but it is only Tuesday early morning...
> 
> Robert

okay, then what version of freeradius are you using ?
Comment 60 Stefan Metzmacher 2014-10-01 13:52:23 UTC
The problem should be mostly fixed in the latest 3.6, 4.0 and 4.1 releases.
While a more advanced fix will be in 4.2.0.
Comment 61 tbskyd 2014-11-13 02:49:55 UTC
hi:
   may I ask which samba 3.6 version fix the bug? I use RHEL 6.5 with samba 3.6.9, and I met this bug sometimes.now RHEL 6.6 comes with samba 3.6.23, I wonder if I can finally get rid of the bug. I check samba 3.6 release notes but didn't find anything about this bug.