Created attachment 10749 [details] NetApp rpcclient failure Our corporate enterprise has obtained a shiny new NetApp (FAS2520) that is running release 8.2.2P1 7-Mode. The client is a CentOS 6.5 x86_64 machine using samba-client tools v3.6.9-164 but building/running trunk's source3 yields the exact same behavior. It has been successfully joined to our test Win2003 SP2 maintained AD domain QUEST as verified by the NetApp itself using the 'cifs testdc' and 'cifs domaininfo' commands. We can also look up SIDs directly through the NetApp's interface via the 'cifs lookup <SID>' command as well. The problem though is when we try to issue the following: rpcclient 'netshareenum 2' -U Administrator@QUEST.COM%<password> <NetApp IP> This fails with: could not obtain sid for domain QUEST error: NT_STATUS_ACCESS_DENIED It seems that after debugging this session, the problem is when rpcclient tries to fetch the machine SID by opening up the LSA, we get NT_STATUS_ACCESS_DENIED for the access mask presented. This doesn't make sense to me because we are using the domain administrator account to issue this command which should always work. If we try this command on a Windows server (including the DC itself), we get the list of shares and their real paths as expected. In fact this rpcclient command works everywhere but on this NetApp. I've attached two small PCAP dumps of both sessions (one is the failure to the NetApp and the other is success to the DC which is exporting a few CIFS shares). Debug output (-d10) from rpcclient shows (well at least the pertinent parts): cli_rpc_pipe_open_noauth: opened pipe \lsarpc to machine 192.168.17.247 and bound anonymously. lsa_OpenPolicy: struct lsa_OpenPolicy in: struct lsa_OpenPolicy system_name : * system_name : 0x005c (92) attr : * attr: struct lsa_ObjectAttribute len : 0x00000018 (24) root_dir : NULL object_name : NULL attributes : 0x00000000 (0) sec_desc : NULL sec_qos : * sec_qos: struct lsa_QosInfo len : 0x0000000c (12) impersonation_level : 0x0002 (2) context_mode : 0x01 (1) effective_only : 0x00 (0) access_mask : 0x02000000 (33554432) 0: LSA_POLICY_VIEW_LOCAL_INFORMATION 0: LSA_POLICY_VIEW_AUDIT_INFORMATION 0: LSA_POLICY_GET_PRIVATE_INFORMATION 0: LSA_POLICY_TRUST_ADMIN 0: LSA_POLICY_CREATE_ACCOUNT 0: LSA_POLICY_CREATE_SECRET 0: LSA_POLICY_CREATE_PRIVILEGE 0: LSA_POLICY_SET_DEFAULT_QUOTA_LIMITS 0: LSA_POLICY_SET_AUDIT_REQUIREMENTS 0: LSA_POLICY_AUDIT_LOG_ADMIN 0: LSA_POLICY_SERVER_ADMIN 0: LSA_POLICY_LOOKUP_NAMES 0: LSA_POLICY_NOTIFICATION &r: struct ncacn_packet rpc_vers : 0x05 (5) rpc_vers_minor : 0x00 (0) ptype : DCERPC_PKT_REQUEST (0) pfc_flags : 0x03 (3) drep: ARRAY(4) [0] : 0x10 (16) [1] : 0x00 (0) [2] : 0x00 (0) [3] : 0x00 (0) frag_length : 0x0018 (24) auth_length : 0x0000 (0) call_id : 0x00000002 (2) u : union dcerpc_payload(case 0) request: struct dcerpc_request alloc_hint : 0x0000002c (44) context_id : 0x0000 (0) opnum : 0x0006 (6) object : union dcerpc_object(case 0) empty: struct dcerpc_empty _pad : DATA_BLOB length=0 stub_and_verifier : DATA_BLOB length=0 rpc_api_pipe: host 192.168.17.247 num_setup=2, max_setup=0, param_total=0, this_param=0, max_param=0, data_total=68, this_data=68, max_data=4280, param_offset=84, param_pad=2, param_disp=0, data_offset=84, data_pad=0, data_disp=0 rpc_read_send: data_to_read: 32 r: struct ncacn_packet rpc_vers : 0x05 (5) rpc_vers_minor : 0x00 (0) ptype : DCERPC_PKT_RESPONSE (2) pfc_flags : 0x03 (3) drep: ARRAY(4) [0] : 0x10 (16) [1] : 0x00 (0) [2] : 0x00 (0) [3] : 0x00 (0) frag_length : 0x0030 (48) auth_length : 0x0000 (0) call_id : 0x00000002 (2) u : union dcerpc_payload(case 2) response: struct dcerpc_response alloc_hint : 0x00000018 (24) context_id : 0x0000 (0) cancel_count : 0x00 (0) _pad : DATA_BLOB length=1 [0000] 00 . stub_and_verifier : DATA_BLOB length=24 [0000] 40 E8 57 0B 00 FE FF FF C0 4A F5 00 80 FF FF FF @.W..... .J...... [0010] 40 E8 57 0B 22 00 00 C0 @.W."... Got pdu len 48, data_len 24, ss_len 0 rpc_api_pipe: got frag len of 48 at offset 0: NT_STATUS_OK rpc_api_pipe: host 192.168.17.247 returned 24 bytes. lsa_OpenPolicy: struct lsa_OpenPolicy out: struct lsa_OpenPolicy handle : * handle: struct policy_handle handle_type : 0x0b57e840 (190310464) uuid : fffffe00-4ac0-00f5-80ff-ffff40e8570b result : NT_STATUS_ACCESS_DENIED could not obtain sid for domain QUEST error: NT_STATUS_ACCESS_DENIED Note, that if I remove the fetch_machine_sid() from source3/rpcclient/rpcclient.c which generates the "could not obtain sid for domain" message, the command goes through and everything just works. Also, on the NetApp side, with "options cifs.trace_login" I see the QUEST\Administrator logging in find to the DC and being mapped to "root" as expected: Thu Feb 19 11:02:36 EST [shannon:auth.trace.authenticateUser.loginTraceIP:info]: AUTH: Login attempt by user administrator of domain QUEST from client machine 192.168.192.124 (AGALLOCH). Thu Feb 19 11:02:36 EST [shannon:auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- attempting authentication with domain controller \\MAHANADI. Thu Feb 19 11:02:36 EST [shannon:auth.trace.authenticateUser.loginTraceMsg:info]: AUTH: User from 192.168.192.124 authenticated by DC. Thu Feb 19 11:02:36 EST [shannon:auth.trace.mapNTToUnix:info]: AUTH: Mapping Windows user administrator to Unix user administrator. Thu Feb 19 11:02:36 EST [shannon:auth.trace.authenticateUser.loginTraceMsg:info]: AUTH: User administrator CIFS home directory is set to /vol/vol0/home/administrator. Thu Feb 19 11:02:36 EST [shannon:auth.trace.mapNTToUnix:info]: AUTH: Mapping Windows user administrator to Unix user pcuser. Thu Feb 19 11:02:36 EST [shannon:auth.trace.mapNTToUnix.mapPrivNTToRoot:info]: AUTH: Mapping Windows user administrator to Unix user root due to the filer option wafl.nt_admin_priv_map_to_root: mapping succeeds. Thu Feb 19 11:02:36 EST [shannon:auth.trace.authenticateUser.loginAccepted:info]: AUTH: Login by administrator from 192.168.192.124 accepted. This is probably pilot error on my part, but I'm at a loss why on this NetApp, the LSA open is failing with the domain administrator account!
Created attachment 10750 [details] Same command to our DC where it succeeds
Created attachment 10751 [details] Sorry, this pcap has BOTH sides (my BPF was rusty!).
Created attachment 10752 [details] Again, both sides now rpcclient succeeding.
Can someone from the samba team explain why rpcclient (and a lot of the other code in source4) always use 0x02000000 (SEC_FLAG_MAXIMUM_ALLOWED)) when binding to the \\lsarpc pipe? This seems a bit gratuitous in that not all RPCs require this level of access to the LSA. Note, the NetApp does respond to NetShareEnum as well as other RPC commands if I reduce the access mask to something more inline with the request. Then everything works as expected.
(In reply to pisymbol from comment #4) SEC_FLAG_MAXIMUM_ALLOWED does not say "get me everything". It says "get me everything I am allowed to do". It's the task of the server to calculate what is allowed. If NetApp does not do it properly, it is very likely a NetApp defect. However, there is no reason except laziness on our side to use this. If you tell us what specific flags are needed, we can easily make the change. The code line in question is source3/rpcclient/rpcclient.c:178.
(In reply to Volker Lendecke from comment #5) Volker, fair enough. I should have RTFM about the access mask first before making the "gratuitous" comment. That line is exactly what I modified to get this to go! I still don't understand why rpcclient has to open the LSA for every RPC command as part of the fetch_machine_sid() logic? Can we make this mask tunable (default to SEC_FLAG_MAXIMUM_ALLOWED) but allow a user to change it in smb.conf? "lsa accessmask = <blah>". Something like that.
(In reply to pisymbol from comment #6) A tunable isn't required. We query something from LSA, and for this we need very specific and focused access bits. The reason why we do it is also laziness: Quite a few rpcclient subcommands need the server's sid. It is just easy to query this when rpcclient starts and store it somewhere. It is more coding effort to do it only on demand. Probably for netshareenum we don't need it, but this would be a bit more effort to code away. Putting the right bits into the call is the correct fix here. If you got it resolved, can you tell us what NetApp wants to see here?
(In reply to Volker Lendecke from comment #7) Volker, but that's why I suggested the tunable because from what you just said, you don't know what the right bits are since it depends on what RPC is being requested. (I was sort of reading in between the lines with my tunable suggestion). There seems a few fixes here for this based on what you just said: - NetApp supports the SEC_FLAG_MAX_ALLOWED mask and does the right thing. Like you, I prefer this fix. - rpcclient doesn't willy nilly call fetch_machine_sid() and does it on demand. And when it does it on demand, it *could* use SEC_FLAG_MAX_ALLOWED and/or if that doesn't work, try a reduced ACCESS_MASK instead (one that gets enough LSA privs to perform the action) else fail. How much work is that? Are you willing to entertain the later if I can't get the former fixed from NetApp? I would love to do the patch and have it reviewed by you! :-)
I don't have access to a NetApp box, and NetApp does not allow me to use their simulator. If I had access, it would be a matter of trying a few things. Not more than, say, 15 minutes of playing around with those flags. I guess it will take longer to even get through their support hotline. In particular if you tell them you are running Samba they will IMMEDIATELY hang up. Been there, done that. Is there any legal way that I can get my hands on NetApp? Or is even connecting to it with rpcclient being seen as a license violation?
(In reply to Volker Lendecke from comment #9) No idea Volker. I have been changing the ACCESS_MASK to get it to work. Is the goal to find the maximum number of bits that can be set and still get this to go? If so, that is EXACTLY what I have been doing (twiddling bits and watching it succeed and fail, etc.). I have an outstanding bug with NetApp and I am actively trying to get from engineering on what ACCESS_MASK is supported. I guess let's start here.
(In reply to pisymbol from comment #10) What is the result of your experiments, i.e. what mask gets you in?
(In reply to Volker Lendecke from comment #11) So far I have: 0x1 0x800 0x801 0x20801 0x10000000 0x20000000
Volker, NetApp's official answer is they don't support the SEC_FLAG_MAX_ALLOWED bitmask and probably never will. What would you like to do?
(In reply to pisymbol from comment #13) Patch the right mask into rpcclient. Did NetApp tell you what bits they need here?
Crickets, and I suspect Volker they aren't going to give me an official answer (the support ticket has already been closed). I really don't understand their engineering organization since all of this stuff is already in the public domain. Weird. Why not a two prong approach? Try with the first mask, and if that fails, try with another that is less permissive but will still do the job.