Bug 10289 - Connection to DFS share unreliable.
Connection to DFS share unreliable.
Status: NEEDINFO
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Tools
4.4.3
All All
: P5 normal
: ---
Assigned To: Jeremy Allison
Samba QA Contact
:
Depends on: 8003
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-22 13:02 UTC by David Woodhouse
Modified: 2016-06-18 15:08 UTC (History)
4 users (show)

See Also:


Attachments
debug output (13.38 KB, text/plain)
2013-11-22 13:02 UTC, David Woodhouse
no flags Details
debug output (10.63 KB, text/plain)
2016-06-17 09:19 UTC, David Woodhouse
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Woodhouse 2013-11-22 13:02:43 UTC
Created attachment 9468 [details]
debug output

For contacting a domain-based DFS share such as \\AMR.CORP.INTEL.COM\ISS I believe we're supposed to contact the domain controller to ask for a referral to a specific machine.

But instead, we seem to do a DNS lookup for AMR.CORP.INTEL.COM and go straight to whichever IP address we pick first out of those results. Which sometimes works, and sometimes doesn't. And *never* returns a local machine on the same continent as me, which Windows does.

Often it fails, reporting NT_STATUS_BAD_NETWORK_NAME. Probably because we end up talking to a machine which doesn't have a copy of the share we were after, and to which we *wouldn't* have been referred, if we'd asked properly.
Comment 1 David Woodhouse 2013-11-22 13:53:33 UTC
Vaguely prodding at this in my ignorance, hoping to learn something and/or shed some light on what's happening...

I have discovered 'rpcclient dfsenum', and established that most of the time, it gives me no useful answer:


[dwmw2@shinybook ~]$ wbinfo --getdcname=AMR.CORP.INTEL.COM
IRSAMR201.amr.corp.intel.com
[dwmw2@shinybook ~]$ rpcclient -k IRSAMR201.amr.corp.intel.com -c 'dfsenum 3'
result was WERR_NOT_FOUND


But if I explicitly select one of the servers I know to be *working*, I get more meaningful results:

[dwmw2@shinybook ~]$ rpcclient -k fmsisscorp05.amr.corp.intel.com -c 'dfsenum 3' 
path: \\amr.corp.intel.com\ISS
	comment: Access to Intel Software Supply (ISS) shares. Chen, Darrin J; Green, Michael P
	state: 513
	num_stores: 8
		storage[0] server: azsissrepl01.amr.corp.intel.com
		storage[0] share: ISS
		storage[1] server: FMSISSCORP05.amr.corp.intel.com
		storage[1] share: ISS
...

But I have a chicken and egg problem; how do I *find* which server(s) I'm supposed to be talking to...?
Comment 2 David Woodhouse 2013-11-22 15:33:40 UTC
This is vaguely enlightening...

http://blogs.technet.com/b/josebda/archive/2009/04/15/understanding-windows-server-2008-dfs-n-by-analyzing-network-traces.aspx

And looking through my own captures, I see vaguely similar results. We *are* doing the lookup and getting a sane set of results. I wonder if the issue is just that we are not falling back to another server from the list, when the first one we choose is not responding.

This is the full set of results:

path: \\amr.corp.intel.com\ISS
	state: 513
	num_stores: 8
		storage[0] server: azsissrepl01.amr.corp.intel.com
		storage[0] share: ISS
		storage[1] server: FMSISSCORP05.amr.corp.intel.com
		storage[1] share: ISS
		storage[2] server: FMSISSLAN02.amr.corp.intel.com
		storage[2] share: ISS
		storage[3] server: CRSISSLAN05.AMR.CORP.INTEL.COM
		storage[3] share: ISS
		storage[4] server: AZSISSLAN01
		storage[4] share: ISS
		storage[5] server: ORSISSLAN07.AMR.CORP.INTEL.COM
		storage[5] share: ISS
		storage[6] server: FMSISSLAN03.AMR.CORP.INTEL.COM
		storage[6] share: ISS
		storage[7] server: AZSISSLAN02.AMR.CORP.INTEL.COM
		storage[7] share: ISS


But FMSISSCORP05.amr.corp.intel.com doesn't *have* a share named 'ISS', so that one fails. I suspect 'AZSISSLAN01' will fail too, if it's ever the one we try. I don't have amr.corp.intel.com in my DNS search domains, and I don't think Samba is cunning enough to qualify it for itself.

If we happen to pick one of the *other* referrals, it's fine.
Comment 3 David Woodhouse 2013-11-22 20:56:32 UTC
Correction. FMSISSCORP05 works fine; it's FMSISSLAN02 which is buggered. Which will become more relevant when I start to show packet dumps.

First we connect a domain server and ask for a referral:

NetBIOS Session Service
    Message Type: Session message (0x00)
    Length: 120
SMB (Server Message Block Protocol)
    SMB Header
        Server Component: SMB
        SMB Command: Trans2 (0x32)
        NT Status: STATUS_SUCCESS (0x00000000)
        Flags: 0x18
            0... .... = Request/Response: Message is a request to the server
            .0.. .... = Notify: Notify client only on open
            ..0. .... = Oplocks: OpLock not requested/granted
            ...1 .... = Canonicalized Pathnames: Pathnames are canonicalized
            .... 1... = Case Sensitivity: Path names are caseless
            .... ..0. = Receive Buffer Posted: Receive buffer has not been posted
            .... ...0 = Lock and Read: Lock&Read, Write&Unlock are not supported
        Flags2: 0xc843
            1... .... .... .... = Unicode Strings: Strings are Unicode
            .1.. .... .... .... = Error Code Type: Error codes are NT error codes
            ..0. .... .... .... = Execute-only Reads: Don't permit reads if execute-only
            ...0 .... .... .... = Dfs: Don't resolve pathnames with Dfs
            .... 1... .... .... = Extended Security Negotiation: Extended security negotiation is supported
            .... .0.. .... .... = Reparse Path: The request does not use a @GMT reparse path
            .... .... .1.. .... = Long Names Used: Path names in request are long file names
            .... .... ...0 .... = Security Signatures Required: Security signatures are not required
            .... .... .... 0... = Compressed: Compression is not requested
            .... .... .... .0.. = Security Signatures: Security signatures are not supported
            .... .... .... ..1. = Extended Attributes: Extended attributes are supported
            .... .... .... ...1 = Long Names Allowed: Long file names are allowed in the response
        Process ID High: 0
        Signature: 0000000000000000
        Reserved: 0000
        Tree ID: 26630  (\\AMR.CORP.INTEL.COM\IPC$)
            [Path: \\AMR.CORP.INTEL.COM\IPC$]
            [Mapped in: 176]
        Process ID: 16091
        User ID: 28672
        Multiplex ID: 5
    Trans2 Request (0x32)
        Word Count (WCT): 15
        Total Parameter Count: 50
        Total Data Count: 0
        Max Parameter Count: 2
        Max Data Count: 65535
        Max Setup Count: 0
        Reserved: 00
        Flags: 0x0000
            .... .... .... ..0. = One Way Transaction: Two way transaction
            .... .... .... ...0 = Disconnect TID: Do NOT disconnect TID
        Timeout: Return immediately (0)
        Reserved: 0000
        Parameter Count: 50
        Parameter Offset: 68
        Data Count: 0
        Data Offset: 120
        Setup Count: 1
        Reserved: 00
        Subcommand: GET_DFS_REFERRAL (0x0010)
        Byte Count (BCC): 55
        Padding: 004420
        GET_DFS_REFERRAL Parameters
            Max Referral Level: 3
            File Name: \amr.corp.intel.com\iss
        Padding: 0000


... and get the response...

SMB (Server Message Block Protocol)
    SMB Header
        Server Component: SMB
        [Response to: 177]
        [Time from request: 0.039166000 seconds]
        SMB Command: Trans2 (0x32)
        NT Status: STATUS_SUCCESS (0x00000000)
        Flags: 0x98
            1... .... = Request/Response: Message is a response to the client/redirector
            .0.. .... = Notify: Notify client only on open
            ..0. .... = Oplocks: OpLock not requested/granted
            ...1 .... = Canonicalized Pathnames: Pathnames are canonicalized
            .... 1... = Case Sensitivity: Path names are caseless
            .... ..0. = Receive Buffer Posted: Receive buffer has not been posted
            .... ...0 = Lock and Read: Lock&Read, Write&Unlock are not supported
        Flags2: 0xc843
            1... .... .... .... = Unicode Strings: Strings are Unicode
            .1.. .... .... .... = Error Code Type: Error codes are NT error codes
            ..0. .... .... .... = Execute-only Reads: Don't permit reads if execute-only
            ...0 .... .... .... = Dfs: Don't resolve pathnames with Dfs
            .... 1... .... .... = Extended Security Negotiation: Extended security negotiation is supported
            .... .0.. .... .... = Reparse Path: The request does not use a @GMT reparse path
            .... .... .1.. .... = Long Names Used: Path names in request are long file names
            .... .... ...0 .... = Security Signatures Required: Security signatures are not required
            .... .... .... 0... = Compressed: Compression is not requested
            .... .... .... .0.. = Security Signatures: Security signatures are not supported
            .... .... .... ..1. = Extended Attributes: Extended attributes are supported
            .... .... .... ...1 = Long Names Allowed: Long file names are allowed in the response
        Process ID High: 0
        Signature: 0000000000000000
        Reserved: 0000
        Tree ID: 26630  (\\AMR.CORP.INTEL.COM\IPC$)
            [Path: \\AMR.CORP.INTEL.COM\IPC$]
            [Mapped in: 176]
        Process ID: 16091
        User ID: 28672
        Multiplex ID: 5
    Trans2 Response (0x32)
        Subcommand: GET_DFS_REFERRAL (0x0010)
        Word Count (WCT): 10
        Total Parameter Count: 0
        Total Data Count: 918
        Reserved: 0000
        Parameter Count: 0
        Parameter Offset: 56
        Parameter Displacement: 0
        Data Count: 918
        Data Offset: 56
        Data Displacement: 0
        Setup Count: 0
        Reserved: 00
        Byte Count (BCC): 919
        Padding: 00
        GET_DFS_REFERRAL Data
            Path Consumed: 46
            Num Referrals: 8
            Flags: 0x0003
                .... .... .... ..1. = Hold Storage: Referral SERVER HOLDS STORAGE for the file
                .... .... .... ...1 = Fielding: The server in referral is FIELDING CAPABLE
            Padding: 0000
            Referrals
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 272
                    Alt Path Offset: 320
                    Node Offset: 368
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \amr.corp.intel.com\iss
                    Alt Path: \amr.corp.intel.com\iss
                    Node: \FMSISSLAN02.amr.corp.intel.com\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 238
                    Alt Path Offset: 286
                    Node Offset: 406
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \amr.corp.intel.com\iss
                    Alt Path: \amr.corp.intel.com\iss
                    Node: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 204
                    Alt Path Offset: 252
                    Node Offset: 444
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \amr.corp.intel.com\iss
                    Alt Path: \amr.corp.intel.com\iss
                    Node: \FMSISSCORP05.amr.corp.intel.com\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 170
                    Alt Path Offset: 218
                    Node Offset: 484
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \amr.corp.intel.com\iss
                    Alt Path: \amr.corp.intel.com\iss
                    Node: \AZSISSLAN02.AMR.CORP.INTEL.COM\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 136
                    Alt Path Offset: 184
                    Node Offset: 522
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \amr.corp.intel.com\iss
                    Alt Path: \amr.corp.intel.com\iss
                    Node: \AZSISSLAN01\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 102
                    Alt Path Offset: 150
                    Node Offset: 522
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \amr.corp.intel.com\iss
                    Alt Path: \amr.corp.intel.com\iss
                    Node: \ORSISSLAN07.AMR.CORP.INTEL.COM\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 68
                    Alt Path Offset: 116
                    Node Offset: 560
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \amr.corp.intel.com\iss
                    Alt Path: \amr.corp.intel.com\iss
                    Node: \CRSISSLAN05.AMR.CORP.INTEL.COM\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 34
                    Alt Path Offset: 82
                    Node Offset: 598
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \amr.corp.intel.com\iss
                    Alt Path: \amr.corp.intel.com\iss
                    Node: \azsissrepl01.amr.corp.intel.com\ISS
Comment 4 David Woodhouse 2013-11-22 20:57:49 UTC
And then in the failing case we contact one of the servers (FMSISSLAN02) and ask for a referral again, and get a 'NOT FOUND' response:

SMB (Server Message Block Protocol)
    SMB Header
        Server Component: SMB
        SMB Command: Trans2 (0x32)
        NT Status: STATUS_SUCCESS (0x00000000)
        Flags: 0x18
            0... .... = Request/Response: Message is a request to the server
            .0.. .... = Notify: Notify client only on open
            ..0. .... = Oplocks: OpLock not requested/granted
            ...1 .... = Canonicalized Pathnames: Pathnames are canonicalized
            .... 1... = Case Sensitivity: Path names are caseless
            .... ..0. = Receive Buffer Posted: Receive buffer has not been posted
            .... ...0 = Lock and Read: Lock&Read, Write&Unlock are not supported
        Flags2: 0xc843
            1... .... .... .... = Unicode Strings: Strings are Unicode
            .1.. .... .... .... = Error Code Type: Error codes are NT error codes
            ..0. .... .... .... = Execute-only Reads: Don't permit reads if execute-only
            ...0 .... .... .... = Dfs: Don't resolve pathnames with Dfs
            .... 1... .... .... = Extended Security Negotiation: Extended security negotiation is supported
            .... .0.. .... .... = Reparse Path: The request does not use a @GMT reparse path
            .... .... .1.. .... = Long Names Used: Path names in request are long file names
            .... .... ...0 .... = Security Signatures Required: Security signatures are not required
            .... .... .... 0... = Compressed: Compression is not requested
            .... .... .... .0.. = Security Signatures: Security signatures are not supported
            .... .... .... ..1. = Extended Attributes: Extended attributes are supported
            .... .... .... ...1 = Long Names Allowed: Long file names are allowed in the response
        Process ID High: 0
        Signature: 0000000000000000
        Reserved: 0000
        Tree ID: 2053  (\\FMSISSLAN02.AMR.CORP.INTEL.COM\IPC$)
            [Path: \\FMSISSLAN02.AMR.CORP.INTEL.COM\IPC$]
            [Mapped in: 308]
        Process ID: 16091
        User ID: 18432
        Multiplex ID: 5
    Trans2 Request (0x32)
        Word Count (WCT): 15
        Total Parameter Count: 74
        Total Data Count: 0
        Max Parameter Count: 2
        Max Data Count: 65535
        Max Setup Count: 0
        Reserved: 00
        Flags: 0x0000
            .... .... .... ..0. = One Way Transaction: Two way transaction
            .... .... .... ...0 = Disconnect TID: Do NOT disconnect TID
        Timeout: Return immediately (0)
        Reserved: 0000
        Parameter Count: 74
        Parameter Offset: 68
        Data Count: 0
        Data Offset: 144
        Setup Count: 1
        Reserved: 00
        Subcommand: GET_DFS_REFERRAL (0x0010)
        Byte Count (BCC): 79
        Padding: 004420
        GET_DFS_REFERRAL Parameters
            Max Referral Level: 3
            File Name: \FMSISSLAN02.amr.corp.intel.com\ISS
        Padding: 0000


SMB (Server Message Block Protocol)
    SMB Header
        Server Component: SMB
        [Response to: 309]
        [Time from request: 0.196614000 seconds]
        SMB Command: Trans2 (0x32)
        NT Status: STATUS_NOT_FOUND (0xc0000225)
        Flags: 0x98
            1... .... = Request/Response: Message is a response to the client/redirector
            .0.. .... = Notify: Notify client only on open
            ..0. .... = Oplocks: OpLock not requested/granted
            ...1 .... = Canonicalized Pathnames: Pathnames are canonicalized
            .... 1... = Case Sensitivity: Path names are caseless
            .... ..0. = Receive Buffer Posted: Receive buffer has not been posted
            .... ...0 = Lock and Read: Lock&Read, Write&Unlock are not supported
        Flags2: 0xc843
            1... .... .... .... = Unicode Strings: Strings are Unicode
            .1.. .... .... .... = Error Code Type: Error codes are NT error codes
            ..0. .... .... .... = Execute-only Reads: Don't permit reads if execute-only
            ...0 .... .... .... = Dfs: Don't resolve pathnames with Dfs
            .... 1... .... .... = Extended Security Negotiation: Extended security negotiation is supported
            .... .0.. .... .... = Reparse Path: The request does not use a @GMT reparse path
            .... .... .1.. .... = Long Names Used: Path names in request are long file names
            .... .... ...0 .... = Security Signatures Required: Security signatures are not required
            .... .... .... 0... = Compressed: Compression is not requested
            .... .... .... .0.. = Security Signatures: Security signatures are not supported
            .... .... .... ..1. = Extended Attributes: Extended attributes are supported
            .... .... .... ...1 = Long Names Allowed: Long file names are allowed in the response
        Process ID High: 0
        Signature: 0000000000000000
        Reserved: 0000
        Tree ID: 2053  (\\FMSISSLAN02.AMR.CORP.INTEL.COM\IPC$)
            [Path: \\FMSISSLAN02.AMR.CORP.INTEL.COM\IPC$]
            [Mapped in: 308]
        Process ID: 16091
        User ID: 18432
        Multiplex ID: 5
    Trans2 Response (0x32)
        Subcommand: GET_DFS_REFERRAL (0x0010)
        Word Count (WCT): 0
        Byte Count (BCC): 0
Comment 5 David Woodhouse 2013-11-22 20:59:12 UTC
While in the successful case we contact a different server, and get a more useful response to the referral request:

SMB (Server Message Block Protocol)
    SMB Header
        Server Component: SMB
        SMB Command: Trans2 (0x32)
        NT Status: STATUS_SUCCESS (0x00000000)
        Flags: 0x18
            0... .... = Request/Response: Message is a request to the server
            .0.. .... = Notify: Notify client only on open
            ..0. .... = Oplocks: OpLock not requested/granted
            ...1 .... = Canonicalized Pathnames: Pathnames are canonicalized
            .... 1... = Case Sensitivity: Path names are caseless
            .... ..0. = Receive Buffer Posted: Receive buffer has not been posted
            .... ...0 = Lock and Read: Lock&Read, Write&Unlock are not supported
        Flags2: 0xc843
            1... .... .... .... = Unicode Strings: Strings are Unicode
            .1.. .... .... .... = Error Code Type: Error codes are NT error codes
            ..0. .... .... .... = Execute-only Reads: Don't permit reads if execute-only
            ...0 .... .... .... = Dfs: Don't resolve pathnames with Dfs
            .... 1... .... .... = Extended Security Negotiation: Extended security negotiation is supported
            .... .0.. .... .... = Reparse Path: The request does not use a @GMT reparse path
            .... .... .1.. .... = Long Names Used: Path names in request are long file names
            .... .... ...0 .... = Security Signatures Required: Security signatures are not required
            .... .... .... 0... = Compressed: Compression is not requested
            .... .... .... .0.. = Security Signatures: Security signatures are not supported
            .... .... .... ..1. = Extended Attributes: Extended attributes are supported
            .... .... .... ...1 = Long Names Allowed: Long file names are allowed in the response
        Process ID High: 0
        Signature: 0000000000000000
        Reserved: 0000
        Tree ID: 4101  (\\FMSISSLAN03.AMR.CORP.INTEL.COM\IPC$)
            [Path: \\FMSISSLAN03.AMR.CORP.INTEL.COM\IPC$]
            [Mapped in: 258]
        Process ID: 16653
        User ID: 8194
        Multiplex ID: 5
    Trans2 Request (0x32)
        Word Count (WCT): 15
        Total Parameter Count: 74
        Total Data Count: 0
        Max Parameter Count: 2
        Max Data Count: 65535
        Max Setup Count: 0
        Reserved: 00
        Flags: 0x0000
            .... .... .... ..0. = One Way Transaction: Two way transaction
            .... .... .... ...0 = Disconnect TID: Do NOT disconnect TID
        Timeout: Return immediately (0)
        Reserved: 0000
        Parameter Count: 74
        Parameter Offset: 68
        Data Count: 0
        Data Offset: 144
        Setup Count: 1
        Reserved: 00
        Subcommand: GET_DFS_REFERRAL (0x0010)
        Byte Count (BCC): 79
        Padding: 004420
        GET_DFS_REFERRAL Parameters
            Max Referral Level: 3
            File Name: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
        Padding: 0000


SMB (Server Message Block Protocol)
    SMB Header
        Server Component: SMB
        [Response to: 262]
        [Time from request: 0.192271000 seconds]
        SMB Command: Trans2 (0x32)
        NT Status: STATUS_SUCCESS (0x00000000)
        Flags: 0x98
            1... .... = Request/Response: Message is a response to the client/redirector
            .0.. .... = Notify: Notify client only on open
            ..0. .... = Oplocks: OpLock not requested/granted
            ...1 .... = Canonicalized Pathnames: Pathnames are canonicalized
            .... 1... = Case Sensitivity: Path names are caseless
            .... ..0. = Receive Buffer Posted: Receive buffer has not been posted
            .... ...0 = Lock and Read: Lock&Read, Write&Unlock are not supported
        Flags2: 0xc843
            1... .... .... .... = Unicode Strings: Strings are Unicode
            .1.. .... .... .... = Error Code Type: Error codes are NT error codes
            ..0. .... .... .... = Execute-only Reads: Don't permit reads if execute-only
            ...0 .... .... .... = Dfs: Don't resolve pathnames with Dfs
            .... 1... .... .... = Extended Security Negotiation: Extended security negotiation is supported
            .... .0.. .... .... = Reparse Path: The request does not use a @GMT reparse path
            .... .... .1.. .... = Long Names Used: Path names in request are long file names
            .... .... ...0 .... = Security Signatures Required: Security signatures are not required
            .... .... .... 0... = Compressed: Compression is not requested
            .... .... .... .0.. = Security Signatures: Security signatures are not supported
            .... .... .... ..1. = Extended Attributes: Extended attributes are supported
            .... .... .... ...1 = Long Names Allowed: Long file names are allowed in the response
        Process ID High: 0
        Signature: 0000000000000000
        Reserved: 0000
        Tree ID: 4101
            [Mapped in: 260]
        Process ID: 16653
        User ID: 8194
        Multiplex ID: 5
    Trans2 Response (0x32)
        Subcommand: GET_DFS_REFERRAL (0x0010)
        Word Count (WCT): 10
        Total Parameter Count: 0
        Total Data Count: 966
        Reserved: 0000
        Parameter Count: 0
        Parameter Offset: 56
        Parameter Displacement: 0
        Data Count: 966
        Data Offset: 56
        Data Displacement: 0
        Setup Count: 0
        Reserved: 00
        Byte Count (BCC): 967
        Padding: 00
        GET_DFS_REFERRAL Data
            Path Consumed: 70
            Num Referrals: 8
            Flags: 0x0003
                .... .... .... ..1. = Hold Storage: Referral SERVER HOLDS STORAGE for the file
                .... .... .... ...1 = Fielding: The server in referral is FIELDING CAPABLE
            Padding: 0000
            Referrals
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 272
                    Alt Path Offset: 344
                    Node Offset: 416
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Alt Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Node: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 238
                    Alt Path Offset: 310
                    Node Offset: 454
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Alt Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Node: \FMSISSCORP05.amr.corp.intel.com\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 204
                    Alt Path Offset: 276
                    Node Offset: 494
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Alt Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Node: \FMSISSLAN02.amr.corp.intel.com\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 170
                    Alt Path Offset: 242
                    Node Offset: 532
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Alt Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Node: \AZSISSLAN02.AMR.CORP.INTEL.COM\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 136
                    Alt Path Offset: 208
                    Node Offset: 570
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Alt Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Node: \AZSISSLAN01\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 102
                    Alt Path Offset: 174
                    Node Offset: 570
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Alt Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Node: \ORSISSLAN07.AMR.CORP.INTEL.COM\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 68
                    Alt Path Offset: 140
                    Node Offset: 608
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Alt Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Node: \CRSISSLAN05.AMR.CORP.INTEL.COM\ISS
                Referral
                    Version: 3
                    Size: 34
                    Server Type: Root targets returns (1)
                    Flags: 0x0000
                        .... .... .... ..0. = NameListReferral: NOT a domain/DC referral response
                        .... .... .... .0.. = TargetSetBoundary: NOT the first target in the target set
                    TTL: 300
                    Path Offset: 34
                    Alt Path Offset: 106
                    Node Offset: 646
                    Server GUID: 00000000-0000-0000-0000-000000000000
                    Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Alt Path: \FMSISSLAN03.AMR.CORP.INTEL.COM\ISS
                    Node: \azsissrepl01.amr.corp.intel.com\ISS
Comment 6 Andreas Schneider 2014-10-28 10:42:12 UTC
Could someone look into this?
Comment 7 Karolin Seeger 2014-11-27 10:55:13 UTC
Is this a showstopper for 4.2.0?
Comment 8 Stefan Metzmacher 2014-11-29 10:23:28 UTC
I guess this is fixed with the fix for bug #10378.
Please reopen if it's still a problem with the latest versions.
Comment 9 Michael Adam 2016-01-06 17:58:21 UTC
(In reply to Stefan Metzmacher from comment #8)
> 
> I guess this is fixed with the fix for bug #10378.

I guess this bug still exists:
The above fix was in the server code.
This bug is about the client code.
It also seems to happen when talking exclusively to windows servers.


> Please reopen if it's still a problem with the latest versions.

Done.

I guess this is a duplicate of bug #8003.
Comment 10 Michael Adam 2016-01-06 17:59:31 UTC
this was not a duplicate of 10378
Comment 11 David Woodhouse 2016-06-16 13:50:02 UTC
This still isn't working...


path: \\amr.corp.intel.com\ISS
	comment: Access to Intel Software Supply (ISS) shares.
	state: 513
	num_stores: 6
		storage[0] server: azsissrepl01.amr.corp.intel.com
		storage[0] share: ISS
		storage[1] server: FMSISSCORP05.amr.corp.intel.com
		storage[1] share: ISS
		storage[2] server: CRSISSLAN05.AMR.CORP.INTEL.COM
		storage[2] share: ISS
		storage[3] server: AZSISSLAN02.AMR.CORP.INTEL.COM
		storage[3] share: ISS
		storage[4] server: FMSISSLAN03.AMR.CORP.INTEL.COM
		storage[4] share: ISS
		storage[5] server: ORSISSLAN03
		storage[5] share: ISS


$ smbclient --use-ccache //amr.corp.intel.com/iss 
Domain=[AMR] OS=[Windows Server 2012 R2 Datacenter 9600] Server=[Windows Server 2012 R2 Datacenter 6.3]
Connection to ORSISSLAN03 failed (Error NT_STATUS_UNSUCCESSFUL)


Sure. Pick the *one* non-FQDN in the list, fail to look it up, and don't bother to fall back to trying one of the ones that *would* have worked...
Comment 12 Jeremy Allison 2016-06-16 16:27:57 UTC
(In reply to David Woodhouse from comment #11)

Can you get be a debug level 10 log of this. I need to see where in the source3/libsmb/clidfs.c code we're not looping over the returned list.

Thanks !

Jeremy.
Comment 13 David Woodhouse 2016-06-17 09:19:53 UTC
Created attachment 12185 [details]
debug output
Comment 14 Jeremy Allison 2016-06-17 17:11:12 UTC
The first log doesn't look like a DFS error to me - it's not even getting that far.

smbclient given a hostname internally ends up calling resolve_name_list() synchronously to get a list of IP addresses that belong to a name, then calls smbsock_any_connect_send() to asynchronously connect to every IP in the list, first on port 445, then on port 139. The first one to respond is the one smbclient tries to connect to.

In your first log the machine that responds first is:

10.19.28.248 at port 139

which is screwed up (why is port 139 responding first, when all the initial connect requests are being done on 445 ?). Then when we connect to it the share name 'iss' doesn't exist on that machine:

tree connect failed: NT_STATUS_BAD_NETWORK_NAME

In the second log, you have 26 ip addresses stored in cache for the name amr.corp.intel.com:

namecache_store: storing 26 addresses for amr.corp.intel.com#20: 10.104.192.200,10.104.193.101,10.104.194.205,10.104.192.10,10.104.194.244,10.104.192.230,10.104.193.134,10.104.195.151,10.104.193.133,10.104.193.41,10.104.195.150,10.104.192.101,10.104.192.68,10.104.194.204,10.104.192.100,10.104.195.220,10.104.192.11,10.104.193.68,10.104.192.168,10.104.194.245,10.104.192.6,10.104.192.132,10.104.192.12,10.104.193.100,10.104.192.133,10.104.192.36

and we just pick the first to talk to - which connects on port 445. Then looks like the DFS lookup works, but just to be sure can you instrument 

cli_dfs_get_referral()

with some debug level 10's (I can do it if you don't have time) so I can see what is coming back ?
Comment 15 David Woodhouse 2016-06-18 15:08:18 UTC
Yeah, I had observed the lack of anything about DFS referrals in the output. But then it seemed like an amazing coincidence that it ended up looking for an unqualified "ORSISSLAN03" when that's precisely what I see in the rpcclient 'dfsenum' output.

Will add the debugging and try again.