It has been proposed by IBM that the problem is in asn1_tag_remaining(). I haven't had a chance to review the specifics yet, but the general location is correct:

"I believe the problem is due to a defect in asn1_tag_remaining(). 
check_spnego_blob_complete() is called to confirm whether or not the security
blob received is complete or not.  It sets the pointers in the data structure
in asn1_start_tag():

        nesting->start = data->ofs;
        nesting->next = data->nesting;
        data->nesting = nesting;

Then in asn1_tag_remaining() those pointers are used:

        remaining = data->nesting->taglen - (data->ofs - data->nesting->start);
        if (remaining > (data->length - data->ofs)) {
                data->has_error = true;
                return -1;
        }
        return remaining;

If I am following the pointers correctly, data->nesting->start = data->ofs.  In
which case, remaining will always evaluate to data->nesting->taglen, which will
always be greater than data->length - data->ofs.  As a result, 
asn1_tag_remaining() will always return an error causing
check_spnego_blob_complete() to return NT_STATUS_OK instead of
NT_STATUS_MORE_PROCESSING_REQUIRED.  This causes the packet to be parsed as a
full security blob, which fails and leads to a login failure error.

In the 3.0.32 version of Samba, the asn1_tag_remaining() function does not
exist, so this problem doesn't occur.

My guess is that the fix would be to repair asn1_tag_remaining() to be something along the lines of:

     remaining = data->nesting->taglen - (data->length - data->ofs);"

(In reply to comment #1)
> My guess is that the fix would be to repair asn1_tag_remaining() to be
> something along the lines of:
> 
>      remaining = data->nesting->taglen - (data->length - data->ofs);"
At first glance, I suspect this is not right, as this function is used in a more general case than just spnego.

More likely we need to not fail the logon with an ASN.1 failure. but rather fail only the offered method, and fall back to NTLMSSP (as this is actually what windows is doing...it's not getting more of the ticket in subsequent packets, it's giving the NTLM challenge in the MORE_PROCESSING_REQUIRED response.

Created attachment 6483 [details]
wireshark capture of session setup of user with 1000 groups.

192.168.213.162 is the windows XP client
192.168.213.10 is the 2k3R2 DC and file server

session setup fully reassembled is in frame 127.  It's kerberos...

Created attachment 6489 [details]
pcap of samba 3.5.8 failling on secblob too big

Created attachment 6490 [details]
pcap of samba 3.5.8 failling on secblob too big

Watchpoint 3: data->has_error

Old value = false
New value = true
asn1_tag_remaining (data=0x7ffff82be660) at ../lib/util/asn1.c:541
541			return -1;
(gdb) where
#0  asn1_tag_remaining (data=0x7ffff82be660) at ../lib/util/asn1.c:541
#1  0x00007ffff7856e64 in asn1_start_tag (data=0x7ffff82be660, tag=<value optimized out>) at ../lib/util/asn1.c:497
#2  0x00007ffff78548b8 in parse_negTokenTarg (blob=..., OIDs=0x7fffffffd470, secblob=0x7fffffffd5f0) at libsmb/clispnego.c:293
#3  0x00007ffff77e6026 in parse_spnego_mechanisms (blob_in=..., pblob_out=<value optimized out>, kerb_mechOID=0x7fffffffd680) at smbd/sesssetup.c:728
#4  0x00007ffff77e7b1d in reply_spnego_negotiate (req=0x7ffff82be340) at smbd/sesssetup.c:797
#5  reply_sesssetup_and_X_spnego (req=0x7ffff82be340) at smbd/sesssetup.c:1303
#6  reply_sesssetup_and_X (req=0x7ffff82be340) at smbd/sesssetup.c:1457
#7  0x00007ffff781efb5 in switch_message (type=115 's', req=0x7ffff82be340, size=16626) at smbd/process.c:1404
#8  0x00007ffff781f357 in construct_reply (conn=0x7ffff82364c0, inbuf=<value optimized out>, nread=16626, unread_bytes=0, seqnum=0, encrypted=false, deferred_pcd=0x0)
    at smbd/process.c:1436
#9  process_smb (conn=0x7ffff82364c0, inbuf=<value optimized out>, nread=16626, unread_bytes=0, seqnum=0, encrypted=false, deferred_pcd=0x0) at smbd/process.c:1509
#10 0x00007ffff781f7d1 in smbd_server_connection_read_handler (ev=<value optimized out>, fde=<value optimized out>, flags=<value optimized out>, private_data=<value optimized out>)
    at smbd/process.c:2126
#11 smbd_server_connection_handler (ev=<value optimized out>, fde=<value optimized out>, flags=<value optimized out>, private_data=<value optimized out>) at smbd/process.c:2141
#12 0x00007ffff7a3f42b in run_events (ev=0x7ffff8236400, selrtn=1, read_fds=0x7fffffffdac0, write_fds=0x7fffffffda40) at lib/events.c:157
#13 0x00007ffff7820e25 in smbd_server_connection_loop_once () at smbd/process.c:844
#14 smbd_process () at smbd/process.c:2455
#15 0x00007ffff7d2ced5 in smbd_accept_connection (ev=<value optimized out>, fde=<value optimized out>, flags=<value optimized out>, private_data=<value optimized out>)
    at smbd/server.c:391
#16 0x00007ffff7a3f42b in run_events (ev=0x7ffff8236400, selrtn=1, read_fds=0x7fffffffdea0, write_fds=0x7fffffffde20) at lib/events.c:157
#17 0x00007ffff7a3f623 in s3_event_loop_once (ev=0x7ffff8236400, location=<value optimized out>) at lib/events.c:220
#18 0x00007ffff7a400c0 in _tevent_loop_once (ev=0x7ffff8236400, location=0x7ffff7f3bc7c "smbd/server.c:743") at ../lib/tevent/tevent.c:497
#19 0x00007ffff7d2db43 in smbd_parent_loop (argc=<value optimized out>, argv=<value optimized out>) at smbd/server.c:743

Created attachment 6491 [details]
git-am fix for 3.5.next

Jim - please check this patch (but I'm 100% convinced it's right :-). I'll create a patch for 3.6.0 as well as this affects all releases.

Jeremy.

Jim, you should be able to reproduce the Windows -> windows spnego split of 4k by setting the parameter:

max xmit = 4356

(which is what Windows uses) in the [global] section of your smb.conf.

Jeremy.

Created attachment 6492 [details]
git-am fix for 3.6.0

Same fix for 3.6.0.

Jeremy.

Comment on attachment 6491 [details]
git-am fix for 3.5.next

Yep, this fixes it!

Re-assigning to Karolin for inclusion in 3.5.x and 3.6.0.

Jeremy.

Pushed to v3-5-test and v3-6-test.
Closing out bug report.

Thanks!