Hi! After creation of a new dns record, the new record is not visible with "nslookup", "host" and "samba-tool dns query". The record is neither visible with a LDAP-browser in the expected container DC=h4.mydom.org,CN=MicrosoftDNS,DC=DomainDnsZones,DC=mydom,DC=org Retrying to add the same record, i get a WERR_DNS_ERROR_RECORD_ALREADY_EXISTS and i can delete the record. see here: root@dctest2:~# samba-tool dns add dc2 h4.mydom.org testc50 CNAME testa50.h4.mydom.org Record added successfully root@dctest2:~# samba-tool dns query dc2 h4.mydom.org testc50 ALL ERROR: Record or zone does not exist. root@dctest2:~# host testc50.h4.mydom.org localhost Using domain server: Name: localhost Address: 127.0.0.1#53 Aliases: Host testc50.h4.mydom.org not found: 3(NXDOMAIN) root@dctest2:~# samba-tool dns add dc2 h4.mydom.org testc50 CNAME testa50.h4.mydom.org ERROR(runtime): uncaught exception - (9711, 'WERR_DNS_ERROR_RECORD_ALREADY_EXISTS') File "/usr/local/samba/lib/python3.6/site-packages/samba/netcmd/__init__.py", line 186, in _run return self.run(*args, **kwargs) File "/usr/local/samba/lib/python3.6/site-packages/samba/netcmd/dns.py", line 945, in run raise e File "/usr/local/samba/lib/python3.6/site-packages/samba/netcmd/dns.py", line 941, in run 0, server, zone, name, add_rec_buf, None) root@dctest2:~# samba-tool dns delete dc2 h4.mydom.org testc50 CNAME testa50.h4.mydom.org Record deleted successfully No matter if the records are added with samba-tool or with the Windows RSAT-tools. On previous versions this works fine.
Sometimes it works on samba 4.11 too , usually not. On samba 4.10 it works allways fine.
To reproduce: 1. create a new domain zone (jupiter.mydom.org) and create 10 A records (testa1-testa10 -> 192.168.1.1-192.168.1.10 ) and 10 CNAME (testc1-testc10 -> testa1-testa10) in the new zone samba-tool dns zonecreate dc2 jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa1 A 192.168.1.1 samba-tool dns add dc2 jupiter.mydom.org testc1 CNAME testa1.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa2 A 192.168.1.2 samba-tool dns add dc2 jupiter.mydom.org testc2 CNAME testa2.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa3 A 192.168.1.3 samba-tool dns add dc2 jupiter.mydom.org testc3 CNAME testa3.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa4 A 192.168.1.4 samba-tool dns add dc2 jupiter.mydom.org testc4 CNAME testa4.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa5 A 192.168.1.5 samba-tool dns add dc2 jupiter.mydom.org testc5 CNAME testa5.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa6 A 192.168.1.6 samba-tool dns add dc2 jupiter.mydom.org testc6 CNAME testa6.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa7 A 192.168.1.7 samba-tool dns add dc2 jupiter.mydom.org testc7 CNAME testa7.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa8 A 192.168.1.8 samba-tool dns add dc2 jupiter.mydom.org testc8 CNAME testa8.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa9 A 192.168.1.9 samba-tool dns add dc2 jupiter.mydom.org testc9 CNAME testa9.jupiter.mydom.org samba-tool dns add dc2 jupiter.mydom.org testa10 A 192.168.1.10 samba-tool dns add dc2 jupiter.mydom.org testc10 CNAME testa10.jupiter.mydom.org The result until here is as expected: samba-tool dns query dc2 jupiter.mydom.org jupiter.mydom.org all Name=, Records=2, Children=0 SOA: serial=21, refresh=900, retry=600, expire=86400, minttl=3600, ns=dc2.mydom.org., email=hostmaster.mydom.org. (flags=600000f0, serial=21, ttl=3600) NS: dc2.mydom.org. (flags=600000f0, serial=1, ttl=3600) Name=testa1, Records=1, Children=0 A: 192.168.1.1 (flags=f0, serial=2, ttl=900) Name=testa10, Records=1, Children=0 A: 192.168.1.10 (flags=f0, serial=20, ttl=900) Name=testa2, Records=1, Children=0 A: 192.168.1.2 (flags=f0, serial=4, ttl=900) Name=testa3, Records=1, Children=0 A: 192.168.1.3 (flags=f0, serial=6, ttl=900) Name=testa4, Records=1, Children=0 A: 192.168.1.4 (flags=f0, serial=8, ttl=900) Name=testa5, Records=1, Children=0 A: 192.168.1.5 (flags=f0, serial=10, ttl=900) Name=testa6, Records=1, Children=0 A: 192.168.1.6 (flags=f0, serial=12, ttl=900) Name=testa7, Records=1, Children=0 A: 192.168.1.7 (flags=f0, serial=14, ttl=900) Name=testa8, Records=1, Children=0 A: 192.168.1.8 (flags=f0, serial=16, ttl=900) Name=testa9, Records=1, Children=0 A: 192.168.1.9 (flags=f0, serial=18, ttl=900) Name=testc1, Records=1, Children=0 CNAME: testa1.jupiter.mydom.org. (flags=f0, serial=3, ttl=900) Name=testc10, Records=1, Children=0 CNAME: testa10.jupiter.mydom.org. (flags=f0, serial=21, ttl=900) Name=testc2, Records=1, Children=0 CNAME: testa2.jupiter.mydom.org. (flags=f0, serial=5, ttl=900) Name=testc3, Records=1, Children=0 CNAME: testa3.jupiter.mydom.org. (flags=f0, serial=7, ttl=900) Name=testc4, Records=1, Children=0 CNAME: testa4.jupiter.mydom.org. (flags=f0, serial=9, ttl=900) Name=testc5, Records=1, Children=0 CNAME: testa5.jupiter.mydom.org. (flags=f0, serial=11, ttl=900) Name=testc6, Records=1, Children=0 CNAME: testa6.jupiter.mydom.org. (flags=f0, serial=13, ttl=900) Name=testc7, Records=1, Children=0 CNAME: testa7.jupiter.mydom.org. (flags=f0, serial=15, ttl=900) Name=testc8, Records=1, Children=0 CNAME: testa8.jupiter.mydom.org. (flags=f0, serial=17, ttl=900) Name=testc9, Records=1, Children=0 CNAME: testa9.jupiter.mydom.org. (flags=f0, serial=19, ttl=900) 2. create a other domain zone (saturn.mydom.org) and create the same 10 A records (testa1-testa10 -> 192.168.2.1-192.168.2.10) and the same 10 CNAME (testc1-testc10 -> testa1-testa10) in the second zone now, the first 5 A and the first 5 CNAME goes tho the second zone (saturn.mydom.org), but the rest is located in the wrong zone (jupiter.mydom.org) samba-tool dns zonecreate dc2 saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa1 A 192.168.2.1 samba-tool dns add dc2 saturn.mydom.org testc1 CNAME testa1.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa2 A 192.168.2.2 samba-tool dns add dc2 saturn.mydom.org testc2 CNAME testa2.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa3 A 192.168.2.3 samba-tool dns add dc2 saturn.mydom.org testc3 CNAME testa3.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa4 A 192.168.2.4 samba-tool dns add dc2 saturn.mydom.org testc4 CNAME testa4.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa5 A 192.168.2.5 samba-tool dns add dc2 saturn.mydom.org testc5 CNAME testa5.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa6 A 192.168.2.6 samba-tool dns add dc2 saturn.mydom.org testc6 CNAME testa6.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa7 A 192.168.2.7 samba-tool dns add dc2 saturn.mydom.org testc7 CNAME testa7.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa8 A 192.168.2.8 samba-tool dns add dc2 saturn.mydom.org testc8 CNAME testa8.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa9 A 192.168.2.9 samba-tool dns add dc2 saturn.mydom.org testc9 CNAME testa9.saturn.mydom.org samba-tool dns add dc2 saturn.mydom.org testa10 A 192.168.2.10 samba-tool dns add dc2 saturn.mydom.org testc10 CNAME testa10.saturn.mydom.org Now the new result is: samba-tool dns query dc2 saturn.mydom.org saturn.mydom.org all Name=, Records=2, Children=0 SOA: serial=21, refresh=900, retry=600, expire=86400, minttl=3600, ns=dc2.mydom.org., email=hostmaster.mydom.org. (flags=600000f0, serial=21, ttl=3600) NS: dc2.mydom.org. (flags=600000f0, serial=1, ttl=3600) Name=testa1, Records=1, Children=0 A: 192.168.2.1 (flags=f0, serial=2, ttl=900) Name=testa2, Records=1, Children=0 A: 192.168.2.2 (flags=f0, serial=4, ttl=900) Name=testa3, Records=1, Children=0 A: 192.168.2.3 (flags=f0, serial=6, ttl=900) Name=testa4, Records=1, Children=0 A: 192.168.2.4 (flags=f0, serial=8, ttl=900) Name=testa5, Records=1, Children=0 A: 192.168.2.5 (flags=f0, serial=10, ttl=900) Name=testc1, Records=1, Children=0 CNAME: testa1.saturn.mydom.org. (flags=f0, serial=3, ttl=900) Name=testc2, Records=1, Children=0 CNAME: testa2.saturn.mydom.org. (flags=f0, serial=5, ttl=900) Name=testc3, Records=1, Children=0 CNAME: testa3.saturn.mydom.org. (flags=f0, serial=7, ttl=900) Name=testc4, Records=1, Children=0 CNAME: testa4.saturn.mydom.org. (flags=f0, serial=9, ttl=900) Name=testc5, Records=1, Children=0 CNAME: testa5.saturn.mydom.org. (flags=f0, serial=11, ttl=900) samba-tool dns query dc2 jupiter.mydom.org jupiter.mydom.org all Name=, Records=2, Children=0 SOA: serial=21, refresh=900, retry=600, expire=86400, minttl=3600, ns=dc2.mydom.org., email=hostmaster.mydom.org. (flags=600000f0, serial=21, ttl=3600) NS: dc2.mydom.org. (flags=600000f0, serial=1, ttl=3600) Name=testa1, Records=1, Children=0 A: 192.168.1.1 (flags=f0, serial=2, ttl=900) Name=testa10, Records=2, Children=0 A: 192.168.1.10 (flags=f0, serial=20, ttl=900) A: 192.168.2.10 (flags=f0, serial=20, ttl=900) Name=testa2, Records=1, Children=0 A: 192.168.1.2 (flags=f0, serial=4, ttl=900) Name=testa3, Records=1, Children=0 A: 192.168.1.3 (flags=f0, serial=6, ttl=900) Name=testa4, Records=1, Children=0 A: 192.168.1.4 (flags=f0, serial=8, ttl=900) Name=testa5, Records=1, Children=0 A: 192.168.1.5 (flags=f0, serial=10, ttl=900) Name=testa6, Records=2, Children=0 A: 192.168.1.6 (flags=f0, serial=12, ttl=900) A: 192.168.2.6 (flags=f0, serial=12, ttl=900) Name=testa7, Records=2, Children=0 A: 192.168.1.7 (flags=f0, serial=14, ttl=900) A: 192.168.2.7 (flags=f0, serial=14, ttl=900) Name=testa8, Records=2, Children=0 A: 192.168.1.8 (flags=f0, serial=16, ttl=900) A: 192.168.2.8 (flags=f0, serial=16, ttl=900) Name=testa9, Records=2, Children=0 A: 192.168.1.9 (flags=f0, serial=18, ttl=900) A: 192.168.2.9 (flags=f0, serial=18, ttl=900) Name=testc1, Records=1, Children=0 CNAME: testa1.jupiter.mydom.org. (flags=f0, serial=3, ttl=900) Name=testc10, Records=2, Children=0 CNAME: testa10.jupiter.mydom.org. (flags=f0, serial=21, ttl=900) CNAME: testa10.saturn.mydom.org. (flags=f0, serial=21, ttl=900) Name=testc2, Records=1, Children=0 CNAME: testa2.jupiter.mydom.org. (flags=f0, serial=5, ttl=900) Name=testc3, Records=1, Children=0 CNAME: testa3.jupiter.mydom.org. (flags=f0, serial=7, ttl=900) Name=testc4, Records=1, Children=0 CNAME: testa4.jupiter.mydom.org. (flags=f0, serial=9, ttl=900) Name=testc5, Records=1, Children=0 CNAME: testa5.jupiter.mydom.org. (flags=f0, serial=11, ttl=900) Name=testc6, Records=2, Children=0 CNAME: testa6.jupiter.mydom.org. (flags=f0, serial=13, ttl=900) CNAME: testa6.saturn.mydom.org. (flags=f0, serial=13, ttl=900) Name=testc7, Records=2, Children=0 CNAME: testa7.jupiter.mydom.org. (flags=f0, serial=15, ttl=900) CNAME: testa7.saturn.mydom.org. (flags=f0, serial=15, ttl=900) Name=testc8, Records=2, Children=0 CNAME: testa8.jupiter.mydom.org. (flags=f0, serial=17, ttl=900) CNAME: testa8.saturn.mydom.org. (flags=f0, serial=17, ttl=900) Name=testc9, Records=2, Children=0 CNAME: testa9.jupiter.mydom.org. (flags=f0, serial=19, ttl=900) CNAME: testa9.saturn.mydom.org. (flags=f0, serial=19, ttl=900)
I don't know if my problem is related. I will try to explain. I have to reverse zones: 0.168.192.in-addr.arpa 2.168.192.in-addr.arpa I create a new entry with samba-tool dns add dc1 0.168.192.in-addr.arpa 157 PTR xxxxxx.domain.de This works only if 157.2.168.192.in-addr.arpa (pointing to any other host) does not exist. If it does the entry is created again. 157.2.168.192.in-addr.arpa ponting to xxxxxx. If there is no entry in the 2.168.192.in-addr.arpa zone then the entry is correctly created in the 0.168.192.in-addr.arpa zone. Can someone reproduce this?
I can reproduce this by adding more then 10 PTR with the same entries in 2 different arpazones. ad example: from 1.0.168.192.in-addr.arpa to 15.0.168.192.in-addr.arpa and from 1.1.168.192.in-addr.arpa to 15.1.168.192.in-addr.arpa All the PTR of the first zone are created in the first zone. The first 10 PTR of the second zone are created in the second zone (as expected), but the rest is created in the (wrong) first zone. the result is: samba-tool dns query dc2 1.168.192.in-addr.arpa 1.168.192.in-addr.arpa all Name=, Records=2, Children=0 SOA: serial=16, refresh=900, retry=600, expire=86400, minttl=3600, ns=dc2.klingons.net., email=hostmaster.klingons.net. (flags=600000f0, serial=16, ttl=3600) NS: dc2.klingons.net. (flags=600000f0, serial=1, ttl=3600) Name=1, Records=1, Children=0 PTR: testa1.venus.klingons.net (flags=f0, serial=2, ttl=900) Name=10, Records=1, Children=0 PTR: testa10.venus.klingons.net (flags=f0, serial=11, ttl=900) Name=2, Records=1, Children=0 PTR: testa2.venus.klingons.net (flags=f0, serial=3, ttl=900) Name=3, Records=1, Children=0 PTR: testa3.venus.klingons.net (flags=f0, serial=4, ttl=900) Name=4, Records=1, Children=0 PTR: testa4.venus.klingons.net (flags=f0, serial=5, ttl=900) Name=5, Records=1, Children=0 PTR: testa5.venus.klingons.net (flags=f0, serial=6, ttl=900) Name=6, Records=1, Children=0 PTR: testa6.venus.klingons.net (flags=f0, serial=7, ttl=900) Name=7, Records=1, Children=0 PTR: testa7.venus.klingons.net (flags=f0, serial=8, ttl=900) Name=8, Records=1, Children=0 PTR: testa8.venus.klingons.net (flags=f0, serial=9, ttl=900) Name=9, Records=1, Children=0 PTR: testa9.venus.klingons.net (flags=f0, serial=10, ttl=900) samba-tool dns query dc2 0.168.192.in-addr.arpa 0.168.192.in-addr.arpa all Name=, Records=2, Children=0 SOA: serial=16, refresh=900, retry=600, expire=86400, minttl=3600, ns=dc2.klingons.net., email=hostmaster.klingons.net. (flags=600000f0, serial=16, ttl=3600) NS: dc2.klingons.net. (flags=600000f0, serial=1, ttl=3600) Name=1, Records=1, Children=0 PTR: testa1.jupiter.klingons.net (flags=f0, serial=2, ttl=900) Name=10, Records=1, Children=0 PTR: testa10.jupiter.klingons.net (flags=f0, serial=11, ttl=900) Name=11, Records=2, Children=0 PTR: testa11.jupiter.klingons.net (flags=f0, serial=12, ttl=900) PTR: testa11.venus.klingons.net (flags=f0, serial=12, ttl=900) Name=12, Records=2, Children=0 PTR: testa12.jupiter.klingons.net (flags=f0, serial=13, ttl=900) PTR: testa12.venus.klingons.net (flags=f0, serial=13, ttl=900) Name=13, Records=2, Children=0 PTR: testa13.jupiter.klingons.net (flags=f0, serial=14, ttl=900) PTR: testa13.venus.klingons.net (flags=f0, serial=14, ttl=900) Name=14, Records=2, Children=0 PTR: testa14.jupiter.klingons.net (flags=f0, serial=15, ttl=900) PTR: testa14.venus.klingons.net (flags=f0, serial=15, ttl=900) Name=15, Records=2, Children=0 PTR: testa15.jupiter.klingons.net (flags=f0, serial=16, ttl=900) PTR: testa15.venus.klingons.net (flags=f0, serial=16, ttl=900) Name=2, Records=1, Children=0 PTR: testa2.jupiter.klingons.net (flags=f0, serial=3, ttl=900) Name=3, Records=1, Children=0 PTR: testa3.jupiter.klingons.net (flags=f0, serial=4, ttl=900) Name=4, Records=1, Children=0 PTR: testa4.jupiter.klingons.net (flags=f0, serial=5, ttl=900) Name=5, Records=1, Children=0 PTR: testa5.jupiter.klingons.net (flags=f0, serial=6, ttl=900) Name=6, Records=1, Children=0 PTR: testa6.jupiter.klingons.net (flags=f0, serial=7, ttl=900) Name=7, Records=1, Children=0 PTR: testa7.jupiter.klingons.net (flags=f0, serial=8, ttl=900) Name=8, Records=1, Children=0 PTR: testa8.jupiter.klingons.net (flags=f0, serial=9, ttl=900) Name=9, Records=1, Children=0 PTR: testa9.jupiter.klingons.net (flags=f0, serial=10, ttl=900)
some extra info : Message from Heinz, > Hi Louis, > > my system: > > Ubuntu 18.04.3 LTS > Kernel 4.15.0-74 > > samba Version 4.11.6 (on 4.12.0.rc2 same issue) > on samba 4.10.5 it works fine. Full thread. https://www.spinics.net/lists/samba/msg161854.html
Here are the steps to reproduce: Add a record in zone "2": samba-tool dns add dc1.domain.de 2.168.192.in-addr.arpa 157 PTR zone0.domain.de -U Administrator Password for [DOMAIN-02\Administrator]: Record added successfully Check record: host 192.168.2.157 157.2.168.192.in-addr.arpa domain name pointer zone0.domain.de. Add the record in zone "0" with d10: samba-tool dns add dc1.domain.de 0.168.192.in-addr.arpa 157 PTR zone0.domain.de -d10 -U Administrator INFO: Current debug levels: all: 10 tdb: 10 printdrivers: 10 lanman: 10 smb: 10 rpc_parse: 10 rpc_srv: 10 rpc_cli: 10 passdb: 10 sam: 10 auth: 10 winbind: 10 vfs: 10 idmap: 10 quota: 10 acls: 10 locking: 10 msdfs: 10 dmapi: 10 registry: 10 scavenger: 10 dns: 10 ldb: 10 tevent: 10 auth_audit: 10 auth_json_audit: 10 kerberos: 10 drs_repl: 10 smb2: 10 smb2_credits: 10 dsdb_audit: 10 dsdb_json_audit: 10 dsdb_password_audit: 10 dsdb_password_json_audit: 10 dsdb_transaction_audit: 10 dsdb_transaction_json_audit: 10 dsdb_group_audit: 10 dsdb_group_json_audit: 10 lpcfg_load: refreshing parameters from /etc/samba/smb.conf Processing section "[global]" Processing section "[netlogon]" Processing section "[sysvol]" pm_process() returned Yes GENSEC backend 'gssapi_spnego' registered GENSEC backend 'gssapi_krb5' registered GENSEC backend 'gssapi_krb5_sasl' registered GENSEC backend 'spnego' registered GENSEC backend 'schannel' registered GENSEC backend 'naclrpc_as_system' registered GENSEC backend 'sasl-EXTERNAL' registered GENSEC backend 'ntlmssp' registered GENSEC backend 'ntlmssp_resume_ccache' registered GENSEC backend 'http_basic' registered GENSEC backend 'http_ntlm' registered GENSEC backend 'http_negotiate' registered GENSEC backend 'krb5' registered GENSEC backend 'fake_gssapi_krb5' registered Using binding ncacn_ip_tcp:dc1.domain.de[,sign] Mapped to DCERPC endpoint 135 resolve_lmhosts: Attempting lmhosts lookup for name dc1.domain.de<0x20> startlmhosts: Can't open lmhosts file /etc/samba/lmhosts. Error was No such file or directory rpc request data: [0000] 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ [0010] 00 00 00 00 02 00 00 00 4B 00 00 00 4B 00 00 00 ........ K...K... [0020] 05 00 13 00 0D A4 C2 AB 50 4D 57 B3 40 9D 66 EE ........ PMW.@.f. [0030] 4F D5 FB A0 76 05 00 02 00 00 00 13 00 0D 04 5D O...v... .......] [0040] 88 8A EB 1C C9 11 9F E8 08 00 2B 10 48 60 02 00 ........ ..+.H`.. [0050] 02 00 00 00 01 00 0B 02 00 00 00 01 00 07 02 00 ........ ........ [0060] 00 00 01 00 09 04 00 00 00 00 00 00 00 00 00 00 ........ ........ [0070] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ [0080] 01 00 00 00 .... rpc reply data: [0000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ [0010] 00 00 00 00 01 00 00 00 01 00 00 00 00 00 00 00 ........ ........ [0020] 01 00 00 00 03 00 00 00 4B 00 00 00 4B 00 00 00 ........ K...K... [0030] 05 00 13 00 0D A4 C2 AB 50 4D 57 B3 40 9D 66 EE ........ PMW.@.f. [0040] 4F D5 FB A0 76 05 00 02 00 00 00 13 00 0D 04 5D O...v... .......] [0050] 88 8A EB 1C C9 11 9F E8 08 00 2B 10 48 60 02 00 ........ ..+.H`.. [0060] 02 00 00 00 01 00 0B 02 00 00 00 01 00 07 02 00 ........ ........ [0070] C0 01 01 00 09 04 00 00 00 00 00 00 00 00 00 00 ........ ........ Mapped to DCERPC endpoint 49153 resolve_lmhosts: Attempting lmhosts lookup for name dc1.domain.de<0x20> startlmhosts: Can't open lmhosts file /etc/samba/lmhosts. Error was No such file or directory Starting GENSEC mechanism spnego Starting GENSEC submechanism gssapi_krb5 Password for [DOMAIN-02\Administrator]: Received smb_krb5 packet of length 313 Received smb_krb5 packet of length 189 kinit for Administrator@DOMAIN.DE succeeded gensec_update_send: gssapi_krb5[0x20a1840]: subreq: 0x209f180 gensec_update_send: spnego[0x20a1450]: subreq: 0x208fe80 gensec_update_done: gssapi_krb5[0x20a1840]: NT_STATUS_MORE_PROCESSING_REQUIRED tevent_req[0x209f180/../../source4/auth/gensec/gensec_gssapi.c:1057]: state[2] error[0 (0x0)] state[struct gensec_gssapi_update_state (0x209f330)] timer[(nil)] finish[../../source4/auth/gensec/gensec_gssapi.c:1067] gensec_update_done: spnego[0x20a1450]: NT_STATUS_MORE_PROCESSING_REQUIRED tevent_req[0x208fe80/../../auth/gensec/spnego.c:1631]: state[2] error[0 (0x0)] state[struct gensec_spnego_update_state (0x2090030)] timer[(nil)] finish[../../auth/gensec/spnego.c:2115] dcerpc_pull_auth_trailer: auth_pad_length 0 gensec_gssapi: NO credentials were delegated GSSAPI Connection will be cryptographically signed gensec_update_send: gssapi_krb5[0x20a1840]: subreq: 0x20a2550 gensec_update_send: spnego[0x20a1450]: subreq: 0x2094480 gensec_update_done: gssapi_krb5[0x20a1840]: NT_STATUS_OK tevent_req[0x20a2550/../../source4/auth/gensec/gensec_gssapi.c:1057]: state[2] error[0 (0x0)] state[struct gensec_gssapi_update_state (0x20a2700)] timer[(nil)] finish[../../source4/auth/gensec/gensec_gssapi.c:1074] gensec_update_done: spnego[0x20a1450]: NT_STATUS_MORE_PROCESSING_REQUIRED tevent_req[0x2094480/../../auth/gensec/spnego.c:1631]: state[2] error[0 (0x0)] state[struct gensec_spnego_update_state (0x2094630)] timer[(nil)] finish[../../auth/gensec/spnego.c:2115] dcerpc_pull_auth_trailer: auth_pad_length 0 gensec_update_send: spnego[0x20a1450]: subreq: 0x2094430 gensec_update_done: spnego[0x20a1450]: NT_STATUS_OK tevent_req[0x2094430/../../auth/gensec/spnego.c:1631]: state[2] error[0 (0x0)] state[struct gensec_spnego_update_state (0x20945e0)] timer[(nil)] finish[../../auth/gensec/spnego.c:2115] rpc request data: [0000] 00 00 07 00 00 00 00 00 00 00 02 00 18 00 00 00 ........ ........ [0010] 00 00 00 00 18 00 00 00 64 00 63 00 31 00 2E 00 ........ d.c.1... [0020] 68 00 71 00 2E 00 62 00 72 00 61 00 69 00 6E 00 x.x...x. x.x.x.x. [0030] 2D 00 62 00 69 00 6F 00 74 00 65 00 63 00 68 00 -.x.x.x. x.x.x.x. [0040] 2E 00 64 00 65 00 00 00 04 00 02 00 17 00 00 00 ..d.e... ........ [0050] 00 00 00 00 17 00 00 00 30 2E 31 36 38 2E 31 39 ........ 0.168.19 [0060] 32 2E 69 6E 2D 61 64 64 72 2E 61 72 70 61 00 00 2.in-add r.arpa.. [0070] 04 00 00 00 00 00 00 00 04 00 00 00 31 35 37 00 ........ ....157. [0080] 08 00 02 00 1A 00 00 00 1A 00 0C 00 F0 00 00 00 ........ ........ [0090] 01 00 00 00 84 03 00 00 00 00 00 00 00 00 00 00 ........ ........ [00A0] 19 7A 6F 6E 65 30 2E 68 71 2E 62 72 61 69 6E 2D .zone0.x x.xxxxx- [00B0] 62 69 6F 74 65 63 68 2E 64 65 00 00 00 00 00 00 xxxxxxx. xx...... t: struct dcerpc_sec_verification_trailer _pad : DATA_BLOB length=0 magic : 0000000000000000 count: struct dcerpc_sec_vt_count count : 0x0002 (2) commands: ARRAY(2) commands: struct dcerpc_sec_vt command : 0x0001 (1) 0x01: DCERPC_SEC_VT_COMMAND_ENUM (1) 0: DCERPC_SEC_VT_COMMAND_END 0: DCERPC_SEC_VT_MUST_PROCESS u : union dcerpc_sec_vt_union(case 0x1) bitmask1 : 0x00000001 (1) 1: DCERPC_SEC_VT_CLIENT_SUPPORTS_HEADER_SIGNING commands: struct dcerpc_sec_vt command : 0x4002 (16386) 0x02: DCERPC_SEC_VT_COMMAND_ENUM (2) 1: DCERPC_SEC_VT_COMMAND_END 0: DCERPC_SEC_VT_MUST_PROCESS u : union dcerpc_sec_vt_union(case 0x2) pcontext: struct dcerpc_sec_vt_pcontext abstract_syntax: struct ndr_syntax_id uuid : 50abc2a4-574d-40b3-9d66-ee4fd5fba076 if_version : 0x00000005 (5) transfer_syntax: struct ndr_syntax_id uuid : 8a885d04-1ceb-11c9-9fe8-08002b104860 if_version : 0x00000002 (2) dcerpc_pull_auth_trailer: auth_pad_length 12 rpc reply data: [0000] EF 25 00 00 .%.. ERROR(runtime): uncaught exception - (9711, 'WERR_DNS_ERROR_RECORD_ALREADY_EXISTS') File "/usr/lib64/python3.6/site-packages/samba/netcmd/__init__.py", line 186, in _run return self.run(*args, **kwargs) File "/usr/lib64/python3.6/site-packages/samba/netcmd/dns.py", line 945, in run raise e File "/usr/lib64/python3.6/site-packages/samba/netcmd/dns.py", line 941, in run 0, server, zone, name, add_rec_buf, None) It says it already exists. But it does not exist in zone "0" only in "2". System is CentOS7 with the Sernet Packages. Samba Version 4.11.6.
#!bin/bash # Run this script 2 times!! DC_FQDN="$(hostname -f)" DC_NAME="$(hostname -s)" # After the first run change the "test1/test2 to test01/test02" # and run it again. ZONE_TEST1="test1.domain.tld" ZONE_TEST2="test2.domain.tld" ZONE_ARPA="0.168.192" ZONE_IP="192.168.0" #kinit Administrator !! AUTH before you run this script. # Create zone # test with FQDN samba-tool dns zonecreate $DC_FQDN $ZONE_TEST1 # test with NAME samba-tool dns zonecreate $DC_NAME $ZONE_TEST2 # Create reverse zone samba-tool dns zonecreate $DC_FQDN $ZONE_ARPA.in-addr.arpa # Create A record samba-tool dns add $DC_FQDN $ZONE_TEST1 testa1 A ZONE_IP.1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc1 CNAME testa1.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa2 A ZONE_IP.2 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc2 CNAME testa2.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa3 A ZONE_IP.3 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc3 CNAME testa3.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa4 A ZONE_IP.4 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc4 CNAME testa4.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa5 A ZONE_IP.5 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc5 CNAME testa5.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa6 A ZONE_IP.6 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc6 CNAME testa6.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa7 A ZONE_IP.7 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc7 CNAME testa7.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa8 A ZONE_IP.8 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc8 CNAME testa8.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa9 A ZONE_IP.9 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc9 CNAME testa9.$ZONE_TEST1 samba-tool dns add $DC_FQDN $ZONE_TEST1 testa10 A ZONE_IP.10 samba-tool dns add $DC_FQDN $ZONE_TEST1 testc10 CNAME testa10.$ZONE_TEST1 samba-tool dns add $DC_NAME $ZONE_TEST2 testa1 A ZONE_IP.1 samba-tool dns add $DC_NAME $ZONE_TEST2 testc1 CNAME testa1.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa2 A ZONE_IP.2 samba-tool dns add $DC_NAME $ZONE_TEST2 testc2 CNAME testa2.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa3 A ZONE_IP.3 samba-tool dns add $DC_NAME $ZONE_TEST2 testc3 CNAME testa3.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa4 A ZONE_IP.4 samba-tool dns add $DC_NAME $ZONE_TEST2 testc4 CNAME testa4.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa5 A ZONE_IP.5 samba-tool dns add $DC_NAME $ZONE_TEST2 testc5 CNAME testa5.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa6 A ZONE_IP.6 samba-tool dns add $DC_NAME $ZONE_TEST2 testc6 CNAME testa6.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa7 A ZONE_IP.7 samba-tool dns add $DC_NAME $ZONE_TEST2 testc7 CNAME testa7.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa8 A ZONE_IP.8 samba-tool dns add $DC_NAME $ZONE_TEST2 testc8 CNAME testa8.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa9 A ZONE_IP.9 samba-tool dns add $DC_NAME $ZONE_TEST2 testc9 CNAME testa9.$ZONE_TEST2 samba-tool dns add $DC_NAME $ZONE_TEST2 testa10 A ZONE_IP.10 samba-tool dns add $DC_NAME $ZONE_TEST2 testc10 CNAME testa10.$ZONE_TEST2 # Query samba-tool dns query $DC_FQDN $ZONE_TEST1 $ZONE_TEST1 all samba-tool dns query $DC_NAME $ZONE_TEST2 $ZONE_TEST2 all ## My results Zone test1 : shows 10 A records and 25 CNAMES Zone test2 : shows 5 A records and 5 CNAMES Zone test01 : shows 5 A records and 5 CNAMES Zone test02 : shows 5 A records and 5 CNAMES After 3 runs i have over 40 CNAMES in zone test1 and all other only 5. And the more times you run it the worse it gets.
Hi, is there anything new? With this bug i can not upgrade my DC's and i have to stay on version 4.10 regards, heinz Am Donnerstag, den 27.02.2020, 10:33 +0000 schrieb samba- bugs@samba.org: > https://bugzilla.samba.org/show_bug.cgi?id=14268 > > --- Comment #7 from Louis <belle@samba.org> --- > #!bin/bash > # Run this script 2 times!! > > DC_FQDN="$(hostname -f)" > DC_NAME="$(hostname -s)" > > # After the first run change the "test1/test2 to test01/test02" > # and run it again. > ZONE_TEST1="test1.domain.tld" > ZONE_TEST2="test2.domain.tld" > > ZONE_ARPA="0.168.192" > ZONE_IP="192.168.0" > > #kinit Administrator !! AUTH before you run this script. > > # Create zone > # test with FQDN > samba-tool dns zonecreate $DC_FQDN $ZONE_TEST1 > # test with NAME > samba-tool dns zonecreate $DC_NAME $ZONE_TEST2 > > # Create reverse zone > samba-tool dns zonecreate $DC_FQDN $ZONE_ARPA.in-addr.arpa > > # Create A record > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa1 A ZONE_IP.1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc1 CNAME > testa1.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa2 A ZONE_IP.2 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc2 CNAME > testa2.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa3 A ZONE_IP.3 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc3 CNAME > testa3.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa4 A ZONE_IP.4 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc4 CNAME > testa4.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa5 A ZONE_IP.5 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc5 CNAME > testa5.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa6 A ZONE_IP.6 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc6 CNAME > testa6.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa7 A ZONE_IP.7 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc7 CNAME > testa7.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa8 A ZONE_IP.8 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc8 CNAME > testa8.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa9 A ZONE_IP.9 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc9 CNAME > testa9.$ZONE_TEST1 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testa10 A ZONE_IP.10 > samba-tool dns add $DC_FQDN $ZONE_TEST1 testc10 CNAME > testa10.$ZONE_TEST1 > > samba-tool dns add $DC_NAME $ZONE_TEST2 testa1 A ZONE_IP.1 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc1 CNAME > testa1.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa2 A ZONE_IP.2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc2 CNAME > testa2.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa3 A ZONE_IP.3 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc3 CNAME > testa3.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa4 A ZONE_IP.4 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc4 CNAME > testa4.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa5 A ZONE_IP.5 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc5 CNAME > testa5.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa6 A ZONE_IP.6 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc6 CNAME > testa6.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa7 A ZONE_IP.7 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc7 CNAME > testa7.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa8 A ZONE_IP.8 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc8 CNAME > testa8.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa9 A ZONE_IP.9 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc9 CNAME > testa9.$ZONE_TEST2 > samba-tool dns add $DC_NAME $ZONE_TEST2 testa10 A ZONE_IP.10 > samba-tool dns add $DC_NAME $ZONE_TEST2 testc10 CNAME > testa10.$ZONE_TEST2 > > # Query > samba-tool dns query $DC_FQDN $ZONE_TEST1 $ZONE_TEST1 all > samba-tool dns query $DC_NAME $ZONE_TEST2 $ZONE_TEST2 all > > > ## My results > Zone test1 : shows 10 A records and 25 CNAMES > Zone test2 : shows 5 A records and 5 CNAMES > Zone test01 : shows 5 A records and 5 CNAMES > Zone test02 : shows 5 A records and 5 CNAMES > > After 3 runs i have over 40 CNAMES in zone test1 > and all other only 5. > > And the more times you run it the worse it gets. >
I tested the script from comment #7 with 4.11.9 on a debian buster and the bug is not reproducable there. I also wonder about the comment #7 how there can be 40 CNAMES in a zone if the test script is only adding 20 different CNAMES. One CNAME can only exist once in a zone. Are you maybe not using the internal dns server and the described bugs come out of te unsupported bind plugin?
(In reply to Björn Jacke from comment #9) To be clear, while having two different approaches to DNS is not ideal, the BIND9_DLZ plugin is still a supported part of Samba. It has a testsuite (we mock bind in a torture test) and if there are bugs there, we do need to fix them.
Hi Björn, It is indifferent if you use the bind plugin or the internal dns. The problem occurs on inserting the records - even with "samba-tool dns add ...." I reproduced the problem on samba 4.11.6 and 4.12.0
as mentioned before, this is working for me, the reproducer script does not reproduce the problem with my samba build and also not with sernet-samba on debian buster. You might have a broken build there, some wrong ldb libraries or something like that.
1. i had the problem on my productive system. Then i downgradet to samba 4.10 and this solved the problem. 2. after this, i created a NEW DOMAIN on 2 new servers (ubuntu 18.04) and samba 4.11. and i reproduced the problem immediate. 3. Christian in Comment#3 has the same problem. 4. Louis (samba developer) reproduced it on his testsystem (comment#7) i did more testes: - new Domain on my testsystem with samba 4.10 -> works fine - upgrade to 4.11 -> the problem exists - downgrade: demote one DC downgrade to 4.10 re-join as DC demote the other DC downgreade to 4.10 re-join -> works fine a New test: - New Domain on samba 4.10 -> works fine as expected - upgrade only 1 DC to samba 4.11 The result: inserting records on the node with 4.10 all works fine, but inserting on the node with 4.11 the problems returns.
Hi, are there any news? for me this is a haevy bug, so i can not upgrade samba. regards, heinz
Created attachment 16195 [details] a failed attempt to run Louis's proof of concept in a testenv Louis, I tried to adapt your script to a testenv, but I couldn't get it to fail to work properly (tried on 4.12 and master). Is there some bit I missed? For anyone who has not done this before, to get a Samba testenv, in a samba source tree, go $ sudo apt install xterm $ ./configure.developer && make -j && make testenv and it should pop up a little xterm window with its own fake network. If you can't see the bug in the testenv, there is either a failure of the environment's fakery, or some significant difference between your set up and that of the testenv (which defaults to using ./st/ad_dc/ -- e.g. st/ad_dc/etc/smb.conf). Or the script is wrong.
(In reply to Douglas Bagnall from comment #15) I reproduced the error from samba 4.11 to 4.12 rc3 . Now I've tested it on Samba 4.12.6 and the problem doesn't seem to exist in this version anymore. @Christian, @Louis : can you confirm this?
Hai, sorry for the late reply, i missed this one. I've rerun my tests also on 4.12.6 and this looks good now, I'm not able to reproduce this again. I cant retest it on lower samba version, since all my DC-s are running 4.12.6 now. but im currently working on the 4.13.0rc4 build, when thats done i can test that one. Also, i made a few small changes to that test script. The new version, only thing i noticed it that the speed of adding records might have caused the previous problem. Which is why you see the 1sec delay in the script below. #!bin/bash # Run this script 2 times on an AD-DC. # Wait few min, verify AD-DC replications. DC_FQDN="$(hostname -f)" DC_NAME="$(hostname -s)" # After the first run change the "test1/test2 to test01/test02" # and run it again. # Atest... (testing A-records) ZONE_TEST1="Atest.domain1.tld" # Atest 2. just and other domain to test. ZONE_TEST2="Atest.domain2.tld" # used iprange/reverse zoned. ZONE_ARPA="0.19.172" ZONE_IP="172.19.0" # Test auth # !! AUTH before you run this script. if [[ -f /tmp/krb5cc_0 ]] then #true echo "found: /tmp/krb5cc_0, kinit Administrator skipped." else #false kinit Administrator fi # Create zone # test with FQDN (thest against Fully Qualified Domain Name) echo "samba-tool dns zonecreate $DC_FQDN $ZONE_TEST1" samba-tool dns zonecreate $DC_FQDN $ZONE_TEST1 # test with NAME (single lable name.) echo "samba-tool dns zonecreate $DC_NAME $ZONE_TEST2" samba-tool dns zonecreate $DC_NAME $ZONE_TEST2 # Create reverse zone echo "samba-tool dns zonecreate $DC_FQDN $ZONE_ARPA.in-addr.arpa" samba-tool dns zonecreate $DC_FQDN $ZONE_ARPA.in-addr.arpa # Create A record, PTR record and add a CNAME echo "Testing on DC $(hostname -s): $DC_FQDN zone : $ZONE_TEST1" for x in 1 2 3 4 5 6 7 8 9 10 do echo "Creating A record: samba-tool dns add $DC_FQDN $ZONE_TEST1 testA${x} A $ZONE_IP.${x}" samba-tool dns add $DC_FQDN $ZONE_TEST1 testA${x} A $ZONE_IP.${x} echo "creating PTR record: samba-tool dns add $DC_FQDN $ZONE_ARPA.in-addr.arpa ${x} PTR testA${x}.$ZONE_TEST1" samba-tool dns add $DC_FQDN $ZONE_ARPA.in-addr.arpa testA${x}.$ZONE_TEST1 PTR $x echo "Creating CNAME record : samba-tool dns add $DC_FQDN $ZONE_TEST1 testC${x} CNAME testA${x}.$ZONE_TEST1" samba-tool dns add $DC_FQDN $ZONE_TEST1 testC${x} CNAME testA${x}.$ZONE_TEST1 echo #echo "waiting 1 sec." #sleep 1 done # repeat above for the other zones for x in 1 2 3 4 5 6 7 8 9 10 do echo "Creating A record: samba-tool dns add $DC_FQDN $ZONE_TEST2 testA${x} A $ZONE_IP.${x}" samba-tool dns add $DC_FQDN $ZONE_TEST2 testA${x} A $ZONE_IP.${x} echo echo "creating PTR record: samba-tool dns add $DC_FQDN $ZONE_ARPA.in-addr.arpa ${x} PTR testA${x}.$ZONE_TEST2" samba-tool dns add $DC_FQDN $ZONE_ARPA.in-addr.arpa testA${x}.$ZONE_TEST2 PTR $x > echo echo "Creating CNAME record : samba-tool dns add $DC_FQDN $ZONE_TEST2 testC${x} CNAME testA${x}.$ZONE_TEST2" samba-tool dns add $DC_FQDN $ZONE_TEST2 testC${x} CNAME testA${x}.$ZONE_TEST2 echo #echo "waiting 1 sec." #sleep 1 done # Query all records. samba-tool dns query $DC_FQDN $ZONE_TEST1 $ZONE_TEST1 all samba-tool dns query $DC_NAME $ZONE_TEST2 $ZONE_TEST2 all And review the record again. so far for me, im unable to reproduce the problem.