Bug 14310 - Can't use MMC DNS Manager with Samba 4.12
Summary: Can't use MMC DNS Manager with Samba 4.12
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB (show other bugs)
Version: 4.12.0
Hardware: All All
: P5 regression (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL: https://gitlab.com/samba-team/samba/-...
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-03 20:09 UTC by Alex MacCuish
Modified: 2021-02-09 22:22 UTC (History)
5 users (show)

See Also:


Attachments
Error at log-level 10 (1.40 KB, text/plain)
2020-03-03 21:23 UTC, Alex MacCuish
no flags Details
backported patch for 4.12 (only) (16.01 KB, patch)
2020-05-15 08:38 UTC, Andrew Bartlett
abartlet: review? (gary)
gary: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alex MacCuish 2020-03-03 20:09:17 UTC
On Samba 4.12.0, I receive the error "ndr_pull_uint32: ndr_pull_error(Buffer Size Error): Pull bytes 4 (../../librpc/ndr/ndr_basic.c:152) at ../../librpc/ndr/ndr_basic.c:152" when opening the DNS Manager in MMC.
Comment 1 Alex MacCuish 2020-03-03 21:23:44 UTC
Created attachment 15838 [details]
Error at log-level 10

This jumped out at me from the log-level 10 output. I'm happy to send the full logs but would rather do that somewhere more private.
Comment 2 Alex MacCuish 2020-03-03 22:22:53 UTC
I'm thinking the DNS RPC server isn't running:

samba-tool dns serverinfo dc01
Failed to connect host xxx.xxx.xxx.xxx on port 135 - NT_STATUS_CONNECTION_REFUSED
Failed to connect host xxx.xxx.xxx.xxx (dc01) on port 135 - NT_STATUS_CONNECTION_REFUSED.
ERROR: Connecting to DNS RPC server dc01 failed with (3221226038, 'The transport-connection attempt was refused by the remote system.')
Comment 3 Andrew Bartlett 2020-03-15 23:26:33 UTC
I've marked the bug private to allow Alex to upload a log.

I'll mark that attachment private when the log comes in and re-open the bug.
Comment 4 Alex MacCuish 2020-03-15 23:36:51 UTC
Created attachment 15863 [details]
Log Level 10
Comment 6 Andrew Bartlett 2020-03-16 00:16:56 UTC
Re-opening bug now attachment is private.
Comment 7 Alex MacCuish 2020-03-31 13:36:45 UTC
I was unable to reproduce the issue on a freshly provisioned DC, so I thought there may be some corruption in my DB that was causing the issue. To investigate, I exported the DB using samba-tool domain backup online (and offline a 2nd time) and tried to restore into an empty VM, but that also failed:

Removing computer account: CN=XXX,OU=Domain Controllers,DC=XXX,DC=XXX,DC=XXX (and any child objects)
ndr_pull_uint32: ndr_pull_error(Buffer Size Error): Pull bytes 4 (../../librpc/ndr/ndr_basic.c:181) at ../../librpc/ndr/ndr_basic.c:181
ERROR(runtime): uncaught exception - (9002, 'WERR_DNS_ERROR_RCODE_SERVER_FAILURE')
  File "/usr/local/samba/lib/python3.7/site-packages/samba/netcmd/__init__.py", line 186, in _run
    return self.run(*args, **kwargs)
  File "/usr/local/samba/lib/python3.7/site-packages/samba/netcmd/domain_backup.py", line 607, in run
    remove_dc(samdb, logger, cn)
  File "/usr/local/samba/lib/python3.7/site-packages/samba/remove_dc.py", line 449, in remove_dc
    remove_dns_account=True)
  File "/usr/local/samba/lib/python3.7/site-packages/samba/remove_dc.py", line 371, in offline_remove_ntds_dc
    remove_dns_account=remove_dns_account)
  File "/usr/local/samba/lib/python3.7/site-packages/samba/remove_dc.py", line 283, in offline_remove_server
    remove_dns_references(samdb, logger, dnsHostName)
  File "/usr/local/samba/lib/python3.7/site-packages/samba/remove_dc.py", line 114, in remove_dns_references
    samdb.dns_replace(dnsHostName, [])
  File "/usr/local/samba/lib/python3.7/site-packages/samba/samdb.py", line 1256, in dns_replace
    return dsdb_dns.replace(self, dns_name, new_records)
A transaction is still active in ldb context [0x22ac3c0] on /sambarestore/private/sam.ldb

I'm happy to send a copy, minus secrets, of my AD DB for investigation if that would help?
Comment 8 Andrew Bartlett 2020-03-31 19:43:29 UTC
I'm sorry for the delay in looking into this.

I strongly suspect that our recent work to address fuzzing results from Google's ozz-fuzz has caused Samba to no longer parse previously valid DNS records. 

To locate good examples, use ldbsearch with the --show-binary option on the DNS partitions, then show us the attribute values without --show-binary (eg as base64).

These are unlikely to be sensitive beyond IP addresses and names.
Comment 9 Alex MacCuish 2020-03-31 19:50:17 UTC
Created attachment 15883 [details]
ldbsearch data

ldbsearch of the domain and forest partitions, _bin signifies with --show-binary option, compressed into .tar.xz file.
Comment 10 Alex MacCuish 2020-03-31 19:51:22 UTC
(In reply to Andrew Bartlett from comment #8)

No worries Andrew, I hope you are safe and well!

I suspected as much and for good measure removed all dNSTombstoned records just in case it was an older one hanging about but no good. I've submitted what I think is what you asked for :)
Comment 11 Alex MacCuish 2020-04-02 13:45:44 UTC
Is there a way I can filter for the "dodgy" records?
Comment 12 Andrew Bartlett 2020-04-03 04:28:25 UTC
I've set the ldbsearch data to be private as I don't think the world needs to see the main data.

The key data is:
dNSProperty:: AAAAAMgAAAAAAAAAAQAAAJIAAAAAAAAA
dNSProperty:: AAAAAFDogwEAAAAAAQAAAJAAAAB45YMB

$ make -j && bin/ndrdump dnsp dnsp_DnsProperty struct --base64-input --input=AAAAAMgAAAAAAAAAAQAAAJIAAAAAAAAA
PYTHONHASHSEED=1 WAF_MAKE=1  ./buildtools/bin/waf build
Waf: Entering directory `/home/abartlet/samba/bin/default'
	Selected embedded Heimdal build
Waf: Leaving directory `/home/abartlet/samba/bin/default'
Build commands will be stored in bin/default/compile_commands.json
'build' finished successfully (11.502s)
pull returned Success
    dnsp_DnsProperty: struct dnsp_DnsProperty
        wDataLength              : 0x00000000 (0)
        namelength               : 0x000000c8 (200)
        flag                     : 0x00000000 (0)
        version                  : 0x00000001 (1)
        id                       : DSPROPERTY_ZONE_NS_SERVERS_DA (146)
        data                     : union dnsPropertyData(case 0)
        name                     : 0x00000000 (0)
dump OK

$ make -j && bin/ndrdump dnsp dnsp_DnsProperty struct --base64-input --input=AAAAAFDogwEAAAAAAQAAAJAAAAB45YMB
PYTHONHASHSEED=1 WAF_MAKE=1  ./buildtools/bin/waf build
Waf: Entering directory `/home/abartlet/samba/bin/default'
	Selected embedded Heimdal build
Waf: Leaving directory `/home/abartlet/samba/bin/default'
Build commands will be stored in bin/default/compile_commands.json
'build' finished successfully (9.527s)
pull returned Success
    dnsp_DnsProperty: struct dnsp_DnsProperty
        wDataLength              : 0x00000000 (0)
        namelength               : 0x0183e850 (25421904)
        flag                     : 0x00000000 (0)
        version                  : 0x00000001 (1)
        id                       : DSPROPERTY_ZONE_SCAVENGING_SERVERS_DA (144)
        data                     : union dnsPropertyData(case 0)
        name                     : 0x0183e578 (25421176)
dump OK

Reverting ae9a25d1ebe8101ea36f3573cc67261c2d79f4da fixes it, so we need to find a way to still avoid bug 14206.
Comment 13 Andrew Bartlett 2020-04-03 04:57:09 UTC
Alex,

Can you describe the history of your domain?  Was it running Windows at some point?

I'm just trying to understand how these values were created, so I can address this properly.

Thanks!
Comment 14 Andrew Bartlett 2020-04-03 06:08:07 UTC
Just to clarify my plan here, I'm assuming this was a Microsoft domain or there has been a Microsoft DC there at some point.

If so, I'll ask Microsoft what such a dnsProperty is meant to mean.  

The value() property in our IDL is an extension and is the cause of the original fuzzing issue.  We may simply revert the problem patch and instead move that length calculation to the callers.
Comment 15 Alex MacCuish 2020-04-03 10:16:42 UTC
I thought I'd rebuilt the domain after the Windows DC we were running had issues but it seems not. Since samba and bind don't use Trust Anchors I'm going to say yes this must be left over from a Windows DC. Thank you for your help with this issue btw!
Comment 16 Alex MacCuish 2020-04-03 10:54:31 UTC
I've just removed the Trust Anchors container and purged the tombstone from all DCs for good measure, and restarted all services but the DNS manager still isn't working for our DCs that have upgraded to 4.12.0. Are there any other records? Is there a such I can run that just shows records which cause the code to trip up?

Cheers!
Comment 17 Alex MacCuish 2020-04-03 23:13:20 UTC
So git revert fee5c6a4247aeac71318186bbff7708d25de5912 definitely fixes the problem for now :)
Comment 18 Andrew Bartlett 2020-04-05 23:44:33 UTC
I've asked Microsoft about this.  If we don't get time/resources to fix this properly then a revert would be an acceptable option.
Comment 19 Derek Lambert 2020-04-22 16:39:29 UTC
I'm seeing the same error with DNS updates (and also when using DNS Manager).

21-Apr-2020 16:54:00.536 samba_dlz: starting transaction on zone dom.domain.com
21-Apr-2020 16:54:00.538 client @0xffff88a203e0 10.10.50.10#54757/key H-MSN-SMBDC-01\$\@DOM.DOMAIN.COM: updating zone 'dom.domain.com/NONE': prerequisites are OK
21-Apr-2020 16:54:00.538 samba_dlz: Starting GENSEC mechanism spnego
21-Apr-2020 16:54:00.539 samba_dlz: Starting GENSEC submechanism gssapi_krb5
21-Apr-2020 16:54:00.614 samba_dlz: gensec_gssapi: NO credentials were delegated
21-Apr-2020 16:54:00.614 samba_dlz: GSSAPI Connection will be cryptographically sealed
21-Apr-2020 16:54:00.641 samba_dlz: Successful AuthZ: [(null),krb5] user [DOMAIN]\[H-MSN-SMBDC-01$] [S-1-5-21-692937900-3436463876-1478695411-1195] at [Tue, 21 Apr 2020 16:54:00.641117 CDT] Remote host [NULL] local host [NULL]
21-Apr-2020 16:54:00.641 samba_dlz: {"timestamp": "2020-04-21T16:54:00.641407-0500", "type": "Authorization", "Authorization": {"version": {"major": 1, "minor": 1}, "localAddress": null, "remoteAddress": null, "serviceDescription": null, "authType": "krb5", "domain": "DOMAIN", "account": "H-MSN-SMBDC-01$", "sid": "S-1-5-21-692937900-3436463876-1478695411-1195", "sessionId": "4ba455bd-18fb-4ce1-b1b5-0f6a7a619b97", "logonServer": "H-MSN-SMBDC-01", "transportProtection": "SEAL", "accountFlags": "0x00002100"}}
21-Apr-2020 16:54:00.643 samba_dlz: allowing update of signer=H-MSN-SMBDC-01\$\@DOM.DOMAIN.COM name=dom.domain.com tcpaddr=10.10.50.10 type=NS key=2508936627.sig-h-msn-smbdc-01.dom.domain.com/160/0
21-Apr-2020 16:54:00.683 client @0xffff88a203e0 10.10.50.10#54757/key H-MSN-SMBDC-01\$\@DOM.DOMAIN.COM: updating zone 'dom.domain.com/NONE': update section prescan OK
21-Apr-2020 16:54:00.685 client @0xffff88a203e0 10.10.50.10#54757/key H-MSN-SMBDC-01\$\@DOM.DOMAIN.COM: updating zone 'dom.domain.com/NONE': adding an RR at 'dom.domain.com' NS h-msn-smbdc-01.dom.domain.com.
21-Apr-2020 16:54:00.687 samba_dlz: ndr_pull_uint32: ndr_pull_error(Buffer Size Error): Pull bytes 4 (../../librpc/ndr/ndr_basic.c:152) at ../../librpc/ndr/ndr_basic.c:152
21-Apr-2020 16:54:00.687 samba_dlz: failed to modify DC=@,DC=dom.domain.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=dom,DC=domain,DC=com - WERR_DNS_ERROR_RCODE_SERVER_FAILURE
21-Apr-2020 16:54:00.696 client @0xffff88a203e0 10.10.50.10#54757/key H-MSN-SMBDC-01\$\@DOM.DOMAIN.COM: updating zone 'dom.domain.com/NONE': rolling back
21-Apr-2020 16:54:00.696 samba_dlz: DSDB Transaction [rollback] at [Tue, 21 Apr 2020 16:54:00.696638 CDT] duration [159812]
21-Apr-2020 16:54:00.696 samba_dlz: {"timestamp": "2020-04-21T16:54:00.696789-0500", "type": "dsdbTransaction", "dsdbTransaction": {"version": {"major": 1, "minor": 0}, "action": "rollback", "transactionId": "3095c3eb-c0e1-4b63-93bb-bc986fd9c1ea", "duration": 159812}}
21-Apr-2020 16:54:00.697 samba_dlz: cancelling transaction on zone dom.domain.com

Is there a way to determine what records are causing the issue?
Comment 20 Andrew Bartlett 2020-04-22 20:10:23 UTC
(In reply to Derek Lambert from comment #19)

G'Day Derek, 

You can see that
DC=@,DC=dom.domain.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=dom,DC=domain,DC=com
is the record we have a problem with.

Have you ever had a Windows DC in that domain?

Thanks,

Andrew Bartlett
Comment 21 Derek Lambert 2020-04-22 20:46:30 UTC
Yes, I currently do. I'm trying to migrate them to Samba :)
Comment 22 Derek Lambert 2020-04-22 20:50:15 UTC
In ADSI Edit the dNSProperty for DC=@,DC=dom.domain.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=dom,DC=domain,DC=com is <not set>.
Comment 23 Andrew Bartlett 2020-04-22 21:37:18 UTC
(In reply to Derek Lambert from comment #21)
So, we have a query out with Microsoft as to what the correct parsing is for this value.  We tightened up our parsing code for Samba 4.12 in response to some fuzzing work and in this case it was tightened up too much.

I'm thinking that we will end up with two different NDR structures, one with length 0 (and enforced with range(0,0) and one with the data.
Comment 24 Alex MacCuish 2020-04-22 21:55:46 UTC
(In reply to Andrew Bartlett from comment #20)
It looks like one of his DCs is trying to update its A record for the root of the domain. I don't think DC=@,DC=dom.domain.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=dom,DC=domain,DC=com is the actual problem here, it just happens to be the update that has been logged and is failing, like any DNS update would do when there's corrupt data. The actual corrupt data will lie somewhere else (like with me, in CN=TrustAnchors) no?
Comment 25 Andrew Bartlett 2020-05-07 11:07:30 UTC
I got an answer from MS and they say to just ignore the dud attribute value.  So a fix here shouldn't be too hard, I just need some time.
Comment 26 Andrew Bartlett 2020-05-14 02:53:52 UTC
Alex,

Would you like to be credited?  Is so, can you let me know what form?  (Name, name and e-mail, name and affiliation/company)?

Thanks!
Comment 27 Alex MacCuish 2020-05-14 07:38:06 UTC
Andrew,

That's very kind of you. Just Alex MacCuish is fine :)

Keep safe and well!
Comment 28 Andrew Bartlett 2020-05-15 08:38:58 UTC
Created attachment 15984 [details]
backported patch for 4.12 (only)
Comment 29 Andrew Bartlett 2020-05-18 21:51:25 UTC
Comment on attachment 15984 [details]
backported patch for 4.12 (only)

Karolin,

Please merge for 4.12.next
Comment 30 Andrew Bartlett 2020-05-19 10:23:53 UTC
Alex,

I'm very sorry, we didn't get this tagged in time for the 4.12.3 release just made.  It will I hope be in the next 4.12 release currently scheduled for July 2 2020.

Sorry!
Comment 31 Karolin Seeger 2020-05-22 11:56:59 UTC
(In reply to Andrew Bartlett from comment #29)
Pushed to autobuild-v4-12-test.
Comment 32 Karolin Seeger 2020-06-04 10:28:39 UTC
(In reply to Karolin Seeger from comment #31)
Pushed to v4-12-test.
Closing out bug report.

Thanks!