Kai, I'm away for two weeks. Could you please have a look ?

So 3.2.0 has this problem as well? I'll look into that the coming week.

(In reply to comment #2)
> So 3.2.0 has this problem as well? I'll look into that the coming week.
> 
Looks like it. I'm seeing the same behavior observed in 3.3.4 (Debian testing 2:3.3.4-1) in 3.2.5 (Debian stable 2:3.2.5-4lenny6) as well.

If it helps any, I'm a FreeRADIUS user. There have been multiple reports of this type of behavior on the FreeRADIUS lists, and Alan DeKok from FreeRADIUS posted about this issue on the samba-technical list back in May.

Alan's post:
http://lists.samba.org/archive/samba-technical/2009-May/064892.html

Interesting. I still need to set up freeradius on a test box to play with that. So far I can't reproduce the error, but I didn't try with user-supplied response to the challenge yet.

Is there any way to test radius without having a radius-capable switch around? I've set up a freeradius server, but I seem to have no way to trigger an ntlm_auth call.

(In reply to comment #5)
> Is there any way to test radius without having a radius-capable switch around?
> I've set up a freeradius server, but I seem to have no way to trigger an
> ntlm_auth call.

One way is to use the eapol_test utility that comes with wpa_supplicant.  It basically plays the part of both the 802.1x supplicant and the switch during the EAP exchange with the RADIUS server.

It appears most distributions don't package eapol_test with wpa_supplicant, so you may have to build it yourself (I had to with Debian). Here are detailed instructions on building and using it:

http://deployingradius.com/scripts/eapol_test/

I'm sure this is beginning to stray outside the normal boundaries of samba development, so let me know if there is anything else I can do to help you out with this.

Is there anything more I can do to help?

It seems like my development system doesn't come with a TLS-enabled freeradius server, so I can't turn on peap, so I can't test the MSCHAP auth. I'm planning to add a feature to ntlm_auth to allow me to test this without having to run a radius server. This might still take a bit, sorry about that.

Yeah, I know in Debian FreeRADIUS does not ship with openssl support due to licensing issues. Having NTLMv2 Session authentication testing capabilities built right into ntlm_auth would be awesome.

I finally managed to get a setup where I can specify the session key the DC hands me, and I can't see the current master tree mangling the session key. What DC is this against?

My systems administration team tells me Active Directory 2008 running in 2003 Native mode. Does that answer your question?

Created attachment 5004 [details]
winbind 3.0.31 success ntlm-auth output

Created attachment 5005 [details]
samba 3.0.31 winbind success log

Created attachment 5006 [details]
winbind 3.3.2 failue log

Created attachment 5007 [details]
winbind 3.3.2 failue ntlm-auth output

This issue is consistently reproducible with the following setup.

Windows 2008 AD parent domain controller in Windows 2003 mode.
Windows 2003 child domain controller in Windows 2003 mode.
client machine joined to Windows 2003 DC using samba 3.3.2+.

executing ntm_auth with challenge/response from freeradius returns invalid NT_KEY


The issue does not occur if machine is joined to Windows 2003 DC using samba 3.0.31. executing ntlm_auth returns correct NT_KEY.


I'm attaching winbind -d12 level logs with the following test setup.


Windows 2008 parent domain:

Domain - EIGHTAD6.TESTING.COM
DC     - EIGHTAD-DC.EIGHTAD6.TESTING.COM (in windows 2003 mode)

Windows 2003 child domain:

Domain - CHILD03.EIGHTAD6.TESTING.COM
DC     - NORMA.CHILD03.EIGHTAD6.TESTING.COM (in windows 2003 mode)


Two client machines one running samba 3.3.2 and another with samba 3.0.31 both joined to  NORMA.CHILD03.EIGHTAD6.TESTING.COM, the child domain.

They try to authenticate a user 'test' in the child domain using ntlm_auth. In the client machine with samba 3.0.31 the NT_KEY returned is proper while with samba 3.3.2 the NT_KEY looks different and wrong.

Note that if it is a normal setup, i.e

a normal stand alone windows 2003 domain (in any mode) or
a normal stand alone windows 2008 domain (in any mode)

then samba 3.3.2 works correctly without issues.

Are you also using Radius? I'm still looking for a good way to test this without having to set up a Radius server.

(In reply to comment #17)
> Are you also using Radius? I'm still looking for a good way to test this
> without having to set up a Radius server.
> 

you can use the attached python script to generate challange and nt-response strings given a username and password.

Usage: ntlm_gen.py <username> <password>

like,

[rajkg@localhost ntlm]$ ./ntlm_gen.py test test123#
Challenge :  cdf59226328c2cb3
Response  :  a91ac7059628da6fb07566035f619855b1857b36255f16f0
NT_KEY    :  870135ABD69B8F916F728794CE42D64E

The script works with python 2.4. You will need the following extra python modules also

http://barryp.org/software/py-md4/

http://twhiteman.netfirms.com/des.html

Created attachment 5015 [details]
NTLMv1 challenge response creation script

Ok, I just got around to set up the following network to confirm this:

WIN2K8.HOME.KBLIN.ORG (Win2k8 R2 DC in 2003 mode as forest root)
^
|
WIN2K3-FOREST.WIN2K8.HOME.KBLIN.ORG (Win2k3 DC subforest, child of WIN2K8 forest)
^                           ^
|                           |
S3.0 member server   and    S3.3 member server

I can wbinfo -K/-a authenticate users against both the WIN2K8 and the WIN2K3-FOREST domains. using ntlm_auth, I can authenticate from both samba servers. However, as soon as I add a challenge, authentication fails on both systems.

What's your smb.conf settings? Did you set up anything special on the DC?

(In reply to comment #20)
> Ok, I just got around to set up the following network to confirm this:
> 

There's nothing special in the smb.conf

[global]
        realm = CHILD03.EIGHTAD6.TESTING.COM
        workgroup = CHILD03

        server string = Samba Server Version %v

        security = ads
        passdb backend = tdbsam


Check if your domain controller's security policy is set to reject NTLM authentications, In the Domain Controller Security Policy

Security Settings --> Local Policies --> Security Options --> Network Security: LAN Manager authentication level

should NOT be set to 

Send NTLMv2 response only\refuse LM & NTLM

Did you use the script that I attached before to generate the challenge/response? I have reconfirmed that the script generates correct challenge/response.

I don't have any policies set that refuse NTLM authentication. Also, if I don't specify a challenge, authentication via both wbinfo and ntlm_auth seems to work fine. ntlm_auth using the squid helper mode negotiates the flags 0x60088205 for both 3.0 and 3.3.

Kia, this issue appears to have cleared up at or around 3.4.3.

On my Debian Testing machine, the samba packages (version 2:3.4.3-2) no longer exhibit the invalid NT_KEY behavior.  And to my knowledge, our Windows Domain environment has not changed since I last tested.

On Ubuntu 10.04 with all (samba) updates applied the problem still exists. What Freeradius and eapol_test show can be seen at: http://tinyurl.com/36wn5lz
If required I will provide more information.

Forgot to mention: Samba is Version 3.4.7

There have been a number more reports of this on the freeradius-users list.  The only common thread
seems to be that 3.0.x works, and later versions work (or not) sporadically.

This is my first freeradius installation. It took me days, to figure out, that ntlm_auth is the villain...

Where are we to resolve this bug. I'm stuck in the implementation of a infrastruture authentication EAP-PEAP with FreeRADIUS.

Regards.

Wild guess: Can this be a duplicate of bug 7568 (or the other way round)?

Volker

It's quite possibly the same as bug #7568.  The ntlm_auth problem has been reported sporadically on the freeradius list for about 2 years, IIRC.

(In reply to comment #24)
> On Ubuntu 10.04 with all (samba) updates applied the problem still exists. What
> Freeradius and eapol_test show can be seen at: http://tinyurl.com/36wn5lz

The real URL is: http://www.wegener-net.de/freeradius/ just in case tinyurl decides to stop forwarding the URL. 

diff --git a/source3/winbindd/winbindd_pam.c b/source3/winbindd/winbindd_pam.c
index bf113e8..277316f 100644
--- a/source3/winbindd/winbindd_pam.c
+++ b/source3/winbindd/winbindd_pam.c
@@ -1785,6 +1785,7 @@ enum winbindd_result winbindd_dual_pam_auth_crap(struct winbindd_domain *domain,
                        goto done;
                }
 
+               domain->can_do_samlogon_ex = false;
                logon_fn = domain->can_do_samlogon_ex
                        ? rpccli_netlogon_sam_network_logon_ex
                        : rpccli_netlogon_sam_network_logon;

solves it for me.

This is the wrong fix, but someone with enough patching skills might want to ack/nack it.

Volker

I can confirm that the patch works for me with samba-latest.
Thank you. 
Will this fix - although "wrong" make it into future releases?

Norbert

Afraid I still can't reproduce this:
Current git master against win2k3:

root@samba-client:~/test# ./ntlm_gen.py administrator secret
Challenge :  3560f2a07f2f72f8
Response  :  cf49f03eebeeac7ffdb080cca483dd95e9e85e69cd86a08b
NT_KEY    :  25EE06323AC15264CF82397711EF38DF

root@samba-client:~/test# ntlm_auth --username=administrator --domain=win2k3 --request-nt-key --nt-response=bd93da6a2274452289cfb5556efafee9a97e04d5e94718f6 --challenge=618df5728c1b2170
NT_KEY: 25EE06323AC15264CF82397711EF38DF

This is on an Ubuntu 10.04 box against a plain Win2k3 AD DC, haven't tried the win2k3 trusting win2k8 thing yet.

Expanding on Volker's suggestion, I've created a patch that adds a --force-samlogon option to winbindd. If winbindd is run with this option, it will skip the attempt to use SamLogonEx and this results in the proper NT_KEY being returned.

This patch can be used against samba_latest, which is currently 3.5.4.

Rob Colantuoni

Created attachment 5880 [details]
Patch to add --force-samlogon option to winbindd

This will resolve issues resulting in winbind using SamLogonEx returning an NT_KEY that the client views as invalid. Thanks to Volker for the pointer to the issue!

Thanks for looking at this. However, we'd rather make that a parametric option. Look at lp_parm_bool(). Command-line parameters are not the Samba-way to force obscure special behaviour.

Volker

Volker,

Will do... that makes more sense to me as well, I wasn't sure if it would be more/less acceptable than a command-line option.

Rob

Created attachment 5882 [details]
Patch to add 'winbind:forcesamlogon' parameter

Changed functionality from command-line option to parameter

Volker,

I created a parameter version and attached the patch. Is there somewhere these obscure parameters are documented?

Rob

Do you have a chance to test the patch provided in 
https://bugzilla.samba.org/show_bug.cgi?id=7568 ?

I am convinced it will resolve this issue as well.

I agree, the patch in bug 7568 is a plausible way to fix this patch.  The failure to sign in that  bug is the exact same thing as an invalid NT Key. 

This shows up because the SamLogonEx call does not do credential chaining, and so we don't notice that we are actually operating with a different implied session key. 

Will a patch be released for 3.4.x soon?

This also affects mpd authing against freeradius and using ntlm_auth on samba 3.2, 3.3 and 3.4 on freebsd 8.1.

(In reply to comment #43)
> Will a patch be released for 3.4.x soon?
> 
> This also affects mpd authing against freeradius and using ntlm_auth on samba
> 3.2, 3.3 and 3.4 on freebsd 8.1.

A patch for 3.4 is attached to the bug mentioned in the previous comments, you can use this link:
https://bugzilla.samba.org/attachment.cgi?id=5894

The next 3.4 release will have this patch included, though there is no planned release date for 3.4.9 yet. There won't be new releases for 3.3 and 3.2, but you will find a patch for 3.3 in https://bugzilla.samba.org/show_bug.cgi?id=7568 as well. 3.2 should not be affected by this.

Greetings Guenther,
(In reply to comment #44)
Could you please confirm that the patches in comment #44 are now part of the 3.5.6-43.el5.i386.rpm's located at?:
ftp://ftp.sernet.de/pub/samba/3.5/rhel/5/i386/

At Monash University we are trying to roll out PEAP-MSCHAPv2 Wireless using Radiator Radius against a Win2008R2 AD. Using Samba 3.4.9 the User-Session-key returned from ntlm_auth changes to a bogus key after between 100 and 1000's authentications. Restarting windbind fixes the problem for another couple of minutes but this is not a suitable solution for a production environment. We hope the problem is resolved by your patches in comment #44 above.

Best regards,

Myles

(In reply to comment #45)
> Greetings Guenther,
> (In reply to comment #44)
> Could you please confirm that the patches in comment #44 are now part of the
> 3.5.6-43.el5.i386.rpm's located at?:
> ftp://ftp.sernet.de/pub/samba/3.5/rhel/5/i386/
> 
> At Monash University we are trying to roll out PEAP-MSCHAPv2 Wireless using
> Radiator Radius against a Win2008R2 AD. Using Samba 3.4.9 the User-Session-key
> returned from ntlm_auth changes to a bogus key after between 100 and 1000's
> authentications. Restarting windbind fixes the problem for another couple of
> minutes but this is not a suitable solution for a production environment. We
> hope the problem is resolved by your patches in comment #44 above.
> 
> Best regards,
> 
> Myles

Yes, 3.5.6 has all necessary fixes for this issue. Unless the sernet packages do contain other changes, it should just work with those packages.

I'm still seeing the issue after patching samba 3.4.9 with the patch from comment #44. Server is FreeBSD 6 with a current ports tree. I've currently worked around the issue by also adding the patch and parameter from comment #39 but I was under the impression that the patch from bug 7568 was all that was needed.

(In reply to comment #39)
> Created an attachment (id=5882) [details]
> Patch to add 'winbind:forcesamlogon' parameter
> 
> Changed functionality from command-line option to parameter
> 
After compiling Samba 3.5.6 with this patch where I should use this parameter?

(In reply to comment #48)
> (In reply to comment #39)
> > Created an attachment (id=5882) [details] [details]
> > Patch to add 'winbind:forcesamlogon' parameter
> > 
> > Changed functionality from command-line option to parameter
> > 
> After compiling Samba 3.5.6 with this patch where I should use this parameter?
> 
Add:
winbind:forcesamlogon = True

to the Global section of smb.conf

(In reply to comment #49)
> (In reply to comment #48)
> > (In reply to comment #39)
> > > Created an attachment (id=5882) [details] [details] [details]
> > > Patch to add 'winbind:forcesamlogon' parameter
> > > 
> > > Changed functionality from command-line option to parameter
> > > 
> > After compiling Samba 3.5.6 with this patch where I should use this parameter?
> > 
> Add:
> winbind:forcesamlogon = True
> 
> to the Global section of smb.conf
> 
Thank you! After adding this parameter everything seems to be fine. But immediately after restarting winbindd I've got situation oppisite to mentioned in #45. Here is a Freeradius output:

Found Auth-Type = MSCHAP
+- entering group MS-CHAP {...}
[mschap] Told to do MS-CHAPv2 for tst-tmp with NT-Password
[mschap] 	expand: %{Stripped-User-Name} -> 
[mschap] 	... expanding second conditional
[mschap] WARNING: Deprecated conditional expansion ":-".  See "man unlang" for details
[mschap] 	expand: %{User-Name:-None} -> tst-tmp
[mschap] 	expand: --username=%{%{Stripped-User-Name}:-%{User-Name:-None}} -> --username=tst-tmp
[mschap]  mschap2: 80
[mschap] 	expand: --challenge=%{mschap:Challenge:-00} -> --challenge=263fdcbaa5eab23d
[mschap] 	expand: --nt-response=%{mschap:NT-Response:-00} -> --nt-response=6ac9670c7768f57359f352231a3d2082d89126fcaecc80cd
Exec-Program output: Access denied (0xc0000022) 
Exec-Program-Wait: plaintext: Access denied (0xc0000022) 
Exec-Program: returned: 1
[mschap] External script failed.
[mschap] FAILED: MS-CHAP2-Response is incorrect
++[mschap] returns reject
Failed to authenticate the user.

But after couple minutes without any action from my side we could see following situation:

+- entering group MS-CHAP {...}
[mschap] Told to do MS-CHAPv2 for tst-tmp with NT-Password
[mschap] 	expand: %{Stripped-User-Name} -> 
[mschap] 	... expanding second conditional
[mschap] WARNING: Deprecated conditional expansion ":-".  See "man unlang" for details
[mschap] 	expand: %{User-Name:-None} -> tst-tmp
[mschap] 	expand: --username=%{%{Stripped-User-Name}:-%{User-Name:-None}} -> --username=tst-tmp
[mschap]  mschap2: ab
[mschap] 	expand: --challenge=%{mschap:Challenge:-00} -> --challenge=7a70df2454d1819d
[mschap] 	expand: --nt-response=%{mschap:NT-Response:-00} -> --nt-response=fad0430ecb49808bd70923bdb904eebac0dc0ad1cfcffa40
Exec-Program output: NT_KEY: 6F76CE716038F6AA2B18FDBB203BCF4B 
Exec-Program-Wait: plaintext: NT_KEY: 6F76CE716038F6AA2B18FDBB203BCF4B 
Exec-Program: returned: 0
[mschap] adding MS-CHAPv2 MPPE keys
++[mschap] returns ok

I was seeing this issue on my server (runnning Ubuntu LucidLynx) - ntlm_auth was returning invalid NT_Keys, causing MS-CHAP authentication failure.

I rebuilt winbind with the patch attached to this bug, and set the winbind:forcesamlogon option in my Samba config file. This fixed the problem completely - now ntlm_auth is generating valid NT_Keys, and authentication is succeeding.

Can you test if 3.5.8 fixes your problem too? There we don't need a special option,

any new testing results with 3.5.8 ?

note we are not going to add the "winbind:forcesamlogon" patch but instead just close as fixed in current version.

Using 3.5.8 and ntlm_auth, it appears as if the NT_KEY returned matches that which is returned on a box with samba 3.0.37.  I have yet to test this with FreeRADIUS, but the fact that the NT_KEY is identical on both machines leads me to believe it will work fine.

I have been running 3.5.8-3.5.9 for a handful of months now with FreeRADIUS without issue.  However, recently, one of the instances started consistently returning an invalid NT_KEY.  I was not able to correlate this to an issue with he domain controller that Samba was communicating with.  I'm currently lacking in additional details, however, restarting winbindd caused it to function correctly again.

(In reply to comment #55)
> I have been running 3.5.8-3.5.9 for a handful of months now with FreeRADIUS
> without issue.  However, recently, one of the instances started consistently
> returning an invalid NT_KEY.  I was not able to correlate this to an issue with
> he domain controller that Samba was communicating with.  I'm currently lacking
> in additional details, however, restarting winbindd caused it to function
> correctly again.

Chiming in with a me too.

We've also suddenly have had problems with this again (sporadically) after having run 3.5.8 problem free since June. Restarting winbind fixed the problem initially, but we had a relapse a day later. A second restart fixed it yet again and has continued to work for a few days now without issues. Nothing has changed on the radius/samba side in terms of software or configuration.

I'm seeing the same problem after upgrading one of my RADIUS servers to RHEL6.

$ winbindd -V
Version 3.5.10-114.el6

I have a cron task which, every minute, invokes ntlm_auth in as close as possible to precisely the same way my Radiator config does (using a known set of credentials), verifies that the response I get back is correct, and restarts winbind immediately if it's wrong.

COMMAND
ntlm_auth --helper-protocol=ntlm-server-1 --require-membership-of='UOFI\groupxxx'

INPUT
Username: userxxx
NT-Domain: UIUC
LANMAN-Challenge: 0000000000000000
NT-Response: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Request-User-Session-Key: Yes
Request-LanMan-Session-Key: Yes
.

EXPECTED OUTPUT
Authenticated: Yes
User-Session-Key: 94xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.


Most of the time it works, but sporadically winbind will get itself into a state where ntlm_auth returns the wrong key.  In keeping with what others have reported, the automatic restart of winbind triggered by my cron task fixes the issue (at least temporarily).  So far I have had two failures this morning:

at 11:04am, actual output was:
Authenticated: Yes
LANMAN-Session-Key: F3xxxxxxxxxxxxxx
User-Session-Key: 67xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.

and at 12:44pm, actual output was:
Authenticated: Yes
LANMAN-Session-Key: 38xxxxxxxxxxxxxx
User-Session-Key: ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
.

Some observations I find interesting:

1. The wrong keys are *different* each time.

2. The invocations that don't work also return a LANMAN-Session-Key, whereas this line is completely absent from the normal invocations.

3. I see the following output in log.wb-UOFI which corresponds in time to just before each instance where my cron task detected the error condition:

[2012/02/03 11:03:35.593559,  1] ../librpc/ndr/ndr.c:421(ndr_push_error)
  ndr_push_error(5): Bad character conversion
[2012/02/03 11:03:35.593702,  1] ../librpc/ndr/ndr.c:421(ndr_push_error)
  ndr_push_error(5): Bad character conversion
[2012/02/03 12:43:11.323491,  1] ../librpc/ndr/ndr.c:421(ndr_push_error)
  ndr_push_error(5): Bad character conversion
[2012/02/03 12:43:11.323601,  1] ../librpc/ndr/ndr.c:421(ndr_push_error)
  ndr_push_error(5): Bad character conversion

I am running with log level = 1.

4. The very first time this happened (not today), my test script was not yet checking for the correct key value, and did not automatically restart winbind.  This prevented a lot of people from logging onto wireless for a bit while I figured out what was going on, but also gave me the chance to observe that once winbind ended up in its broken state, ntlm_auth would consistently produce the *same* wrong key over and over again in response to repeated invocations of ntlm_auth with the same input, until winbind was restarted.

I hope this helps someone figure out what's going on.  This bug is definitely still alive and kicking in 3.5.10, and I don't want to have to go back to an ancient version of Samba if I can help it...

I have observed the same behavior as mentioned above with
version 3.5.6. I too created a check script to re-start the
server when it detected a problem. I'd say I saw this Winbindd
issue a couple of times a week ?.. 

One thing that I did notice early on was that it always
seemed to crop up in a situation when the server was very idle
for a time. For instance, when I was first working with Freeradius
I would bring up a server, do some testing, then go home and come
back in the morning. When I came back in the morning the authentication
would fail. I pretty early on learned it was related to this bug and
simply restarted Winbindd and continued my  Freeradius development.
This is NOT to mean the only situation I would find it to fail was when
it was idle, just that it was very consistent then.

My little check/restart script worked for well over a year. Then last
Friday (July 27,2012 around 4:40 p.m. MDT), something happened in
our environment and Winbindd seemed to start behaving badly even after
a restart. I could not get a good stable Freeradius server due to this
bug. I ended up installing the latest 3.5 version (currently 3.5.16).
This did seem to clear things up. I will say I have not experienced this
bug since, but it is only Tuesday early morning...

Robert

(In reply to comment #58)
> I have observed the same behavior as mentioned above with
> version 3.5.6. I too created a check script to re-start the
> server when it detected a problem. I'd say I saw this Winbindd
> issue a couple of times a week ?.. 
could you share that check script ? 

in my environment samba and winbind version are :
samba-winbind-3.5.8-68.el6.1.x86_64
samba-common-3.5.8-68.el6.1.x86_64
samba-winbind-clients-3.5.8-68.el6.1.x86_64
samba-3.5.8-68.el6.1.x86_64

This samba/winbind has been giving headache lately

> 
> 
> My little check/restart script worked for well over a year. Then last
> Friday (July 27,2012 around 4:40 p.m. MDT), something happened in
> our environment and Winbindd seemed to start behaving badly even after
> a restart. I could not get a good stable Freeradius server due to this
> bug. I ended up installing the latest 3.5 version (currently 3.5.16).
> This did seem to clear things up. I will say I have not experienced this
> bug since, but it is only Tuesday early morning...
> 
> Robert

okay, then what version of freeradius are you using ?

The problem should be mostly fixed in the latest 3.6, 4.0 and 4.1 releases.
While a more advanced fix will be in 4.2.0.

hi:
   may I ask which samba 3.6 version fix the bug? I use RHEL 6.5 with samba 3.6.9, and I met this bug sometimes.now RHEL 6.6 comes with samba 3.6.23, I wonder if I can finally get rid of the bug. I check samba 3.6 release notes but didn't find anything about this bug.