Created attachment 9575 [details] Level 10 debug log of an unsuccessfull adrestore try I do some research for writing a new Wiki HowTo about restoring AD objects and the AD recycle bin, what is really a usefull feature, but I encoutered a problem/bug: When I delete e. g. an user, the object is moved correctly into the CN=Deleted Objects container. But when I try to restore it on any way like adrestore (Sysinternals), Lazarus (Freeware) or by hand with lpd.exe, it fails. The restore isn't successfull, because when the object should be transfered back to it's last known parent, the following error appears: Modify of 'distinguishedName' on CN=demo3\0ADEL:c443ca4f-9335-484a-9144-5b2d334fc948,CN=Deleted Objects,DC=samdom,DC=example,DC=com not permitted, must use 'rename' operation instead It would be really great if someone could have a look at this. If we get the restoring AD deleted objects function or finally the complete AD recycle bin working, it would be a great benefit for all Samba users.
Created attachment 9576 [details] level 10 debug log of deleting an user This is just a level 10 debug log, of a delete user event (I deleted the user "demo3") from ADUC. The user object was in CN=Deleted Objects after the deletion.
Hi Marc, as you see in https://git.samba.org/samba.git/?p=samba.git;a=blob;f=lib/ldb/modules/rdn_name.c;h=f44ea71f660c4e34335c25e4570e9ce42059cb3b;hb=master the check is enforced by rdn_name_modify(). I would prefer to not change this LDB module (adding bypass flags ecc.) since it is LDB internal but rather try other ways to circumvent it (eg. to internally use a rename and not a modify operation).
Hello Matthias, (In reply to comment #2) > as you see in > https://git.samba.org/samba.git/?p=samba.git;a=blob;f=lib/ldb/modules/rdn_name.c;h=f44ea71f660c4e34335c25e4570e9ce42059cb3b;hb=master > the check is enforced by rdn_name_modify(). > > I would prefer to not change this LDB module (adding bypass flags ecc.) > since it is LDB internal but rather try other ways to circumvent it (eg. > to internally use a rename and not a modify operation). It would be great if there could be way to get this working, as the AD-Recycle-Bin or at least the Deleted Objects recovery is a really great feature of AD, for recovering without restoring offline or the whole AD. And the half side is already working. :-) Can you have a look at it (I'm not a programmer)? Please let me know, if I can provide any further help. E. g. testing or providing details about the Deleted Objects stuff like a step-by-step-guide for manual recovery with e. g. ldp.exe.
Okay, http://support.microsoft.com/?kbid=840001 says that a delete restore on AD works like this: dn: ...,CN=Deleted Objects,DC=... changetype: modify delete: isDeleted - replace: distinguishedName distinguishedName: <new DN> We do not support it yet and it requires some work. As a workaround for Samba you could try this one: - rename (=move) the object away from the "CN=Deleted Objects" container back to the original position - remove the "isDeleted" attribute and re-add the correct "objectCategory" attribute Concrete example: ./bin/ldbrename -H ldap://127.0.0.21 -Uadministrator%locDCpass1 "cn=testuser\0ADEL:2f1577df-eb9f-4d17-99d8-5a2e42e178d9,CN=Deleted Objects,DC=samba,DC=example,DC=com" "cn=testuser,cn=users,dc=samba,dc=example,dc=com" ./bin/ldbedit -H ldap://127.0.0.21 -Uadministrator%locDCpass1 -b "cn=testuser,cn=users,dc=samba,dc=example,dc=com" --show-deleted In the editor you may remove the "isDeleted" line and re-add the "objectCategory" attribute with value "CN=Person,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com"
Hello Matthias, thanks for that example workaround. I tried it and it works. (In reply to comment #4) > ./bin/ldbedit -H ldap://127.0.0.21 -Uadministrator%locDCpass1 -b > "cn=testuser,cn=users,dc=samba,dc=example,dc=com" --show-deleted Only here is a small typo. The "cn=Users"... is case sensitive on this place. :-) I'll use that all in the new Wiki HowTo I'm working on and also place a link to this bug report, so users can use the "official" way, when there's a fix for the modify/rename operation.
To properly restore deleted objects, we should implement support for special modify request - aka. Tombstone reanimation A branch to track current effort in impelementing this feature: https://github.com/kamenim/samba/compare/tombstone-reanimation-wip For easier review, there is a pull request to outline only the differences to master: https://github.com/kamenim/samba/pull/2 At the moment, we have tests and implementation ready. All the patches though are based on Jelmers initial patches for unittest and the branch doesn't rebase cleanly on master at the moment. What's left: * rebase on master at some point * perhaps some more tests and tests tidy up * reviews cheers
Created attachment 10825 [details] 4.2 patch cherry-picked from master
The patches in the blocking bug #11137 and #9704 need to be applied first, otherwise it will not patch cleanly.
(correction) The patches in the blocking bug #11137 and #9404 need to be applied first, otherwise it will not patch cleanly.
I applied the patches from bug #11137 and bug #9404 + the one from here to 4.2.0. Then I recompiled the patched sources on both DCs in my test environment. I started Samba on both DCs and deleted user demo01 via ADUC on DC1. I can find the deleted user on both DCs for a few minutes, before it's totally lost in space. But the record on DC1 have more attributes than on DC2. It looks like, it is already recycled on DC2, but not on DC1. And when the garbage collector comes after a few minutes, it's gone on both. On DC1 (where I have deleted the user): See attached DC1.txt On DC2: See attached DC2.txt To save you the time to compare: This are the attributes of the deleted entry, that are additional on DC1 but _not_ on DC2: badPwdCount: 0 codePage: 0 countryCode: 0 badPasswordTime: 0 lastLogoff: 0 lastLogon: 0 primaryGroupID: 513 accountExpires: 9223372036854775807 logonCount: 0 userPrincipalName: demo01@samdom.example.com pwdLastSet: 130542401790000000 uidNumber: 10001 loginShell: /bin/bash unixHomeDirectory: /home/demo01 gidNumber: 10000 msSFU30Name: demo01 unixUserPassword: ABCD!efgh12345$67890 givenName: Hans sn: Dampf displayName: Hans Dampf telephoneNumber: 0123456 mail: Hans.Dampf@example.com profilePath: \\DC1\profiles\demo01 This are the attributes of the deleted entry, that are additional on DC2 but _not_ on DC1: objectGUID: 925a163c-917f-413a-81d3-8a8545aaf8af isRecycled: TRUE Something more I've discovered, but don't understand is, that # ldbsearch -H ldap://localhost -U administrator --show-deleted "CN=demo01\\0ADEL:925a163c-917f-413a-81d3-8a8545aaf8af,CN=Deleted*" returns nothing. But # ldbsearch -H ldap://localhost -U administrator --show-deleted "CN=demo01\\0ADEL:925a163c-917f-413a-81d3-8a8545aaf8af*" returns the record with its attributes!? I can't use the full DN with ldbsearch, ldbedit or ldbrename.
Created attachment 10868 [details] DC1.txt Result of ldbsearch executed on the DC I've deleted the user on.
Created attachment 10869 [details] DC2.txt Result of ldbsearch executed on the second DC (not the one I've deleted the record on)
I don't think these patches were intended to revolve bug #10403 but I agree, they may not be much use with this regression in place.
Comment on attachment 10825 [details] 4.2 patch cherry-picked from master Wow Andrew, thanks for doing this backport work!
Created attachment 10878 [details] 4.2 patch cherry-picked from master (with ldap server patch) I missed a change to our LDAP server that impacts the test suite (but not real-world operation). This version passes a full autobuild on sn-devel, with the other patches this depends on.
Comment on attachment 10878 [details] 4.2 patch cherry-picked from master (with ldap server patch) We should first fix the flakey test before backporting this stuff. See https://git.samba.org/autobuild.flakey/2015-03-17-1952/samba.stdout https://git.samba.org/autobuild.flakey/2015-03-18-0456/samba.stdout [1657(10627)/1735 at 1h37m36s] samba4.tombstone_reanimation.python(fl2003dc) Test Container reanimation UNEXPECTED(failure): samba4.tombstone_reanimation.python.tombstone_reanimation.RestoreContainerObjectTestCase.test_container(fl2003dc) REASON: Exception: Exception: Traceback (most recent call last): File "bin/python/samba/tests/__init__.py", line 113, in run testMethod() File "/memdisk/autobuild/fl/b2199/samba/source4/dsdb/tests/python/tombstone_reanimation.py", line 541, in test_container self.assertAttributesEqual(obj, attr_orig, obj_restore, attr_rest) File "/memdisk/autobuild/fl/b2199/samba/source4/dsdb/tests/python/tombstone_reanimation.py", line 86, in assertAttributesEqual self.assertEqual(attrs_orig, attrs_rest, "Actual object does not have expected attributes") AssertionError: Actual object does not have expected attributes FAILED (1 failures, 0 errors and 0 unexpected successes in 0 testsuites)
Created attachment 10887 [details] failed autobuild output Attaching the failed autobuild output so it is not lost while we work on this
Fixed in Samba 4.5.0rc1 by fb9af9727f4a9b1664c5a9e541981d239c72b0cf
I reopen this bug, because recovering deleted objects still does not work in 4.5 RC2.
Created attachment 12389 [details] Log output from ldbrename If I follow the steps we describe in the Wiki (https://wiki.samba.org/index.php/The_AD_recycle_bin#On_a_Samba_DC_2), I find the objects using the ldbsearch command. However, it's not possible to move the object using ldbrename anymore: # ldbrename -H ldap://localhost -Uadministrator "CN=demo08\\0ADEL:85f2ba20-7474-42e5-a9f5-c6cb6356bc42,CN=Deleted Objects,DC=samdom,DC=example,DC=com" "cn=demo08,cn=Users,dc=samdom,dc=example,dc=com" Password for [SAMDOM\administrator]: rename of 'CN=demo08\0ADEL:85f2ba20-7474-42e5-a9f5-c6cb6356bc42,CN=Deleted Objects,DC=samdom,DC=example,DC=com' to 'cn=demo08,cn=Users,dc=samdom,dc=example,dc=com' failed - LDAP error 32 LDAP_NO_SUCH_OBJECT - <00002030: ldb_wait from ../source4/ldap_server/ldap_backend.c:483 with LDB_WAIT_ALL: No such object (32)> <> Log attached.
Created attachment 12390 [details] Log output from adrestore -r I tried recovering a deleted user object using the Sysinternal tool "adrestore -r". Even if it shows that the restore succeeded, it failed. If you re-run "adrestore" to scan for deleted objects, it still shows the object as deleted and able to recover. Log attached.
(In reply to Marc Muehlfeld from comment #21) Hi Marc, I've done this sequence of operations from a Windows machine with a Samba DC. The only way I'm able to reproduce this bug is where restoring an object would cause a duplicate DN. In this case, it exhibits exactly the behaviour you've described. With a single object, no duplicate DN, it seems to work.
(In reply to Bob Campbell from comment #22) > The only way I'm able to reproduce this bug is where restoring an object > would cause a duplicate DN. In this case, it exhibits exactly the behaviour > you've described. With a single object, no duplicate DN, it seems to work. I'm not sure, what you mean. I saw you edited the Wiki and replaced the ldbrename with ldbmodify. However I created a new user, deleted it and tried to bring the object back using the ldbmodify way you described and fail now with a different error: [root@DC1 ~]# samba-tool user add demoUser01 Passw0rd Note: samba-tool user add is deprecated. Please use samba-tool user create for the same function. User 'demoUser01' created successfully [root@DC1 ~]# samba-tool user delete demoUser01 Deleted user demoUser01 [root@DC1 ~]# ldbsearch -H ldap://localhost -U administrator --show-deleted cn=demoUser01\\0ADEL:* Password for [SAMDOM\administrator]: # record 1 dn: CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com objectClass: top objectClass: person objectClass: organizationalPerson objectClass: user instanceType: 4 whenCreated: 20160822091207.0Z uSNCreated: 4026 objectGUID: ebc71f0d-8d54-474d-b6cc-c0e85d3f028d badPwdCount: 0 codePage: 0 countryCode: 0 badPasswordTime: 0 lastLogoff: 0 lastLogon: 0 primaryGroupID: 513 objectSid: S-1-5-21-469703510-2364959079-1506205053-1130 accountExpires: 9223372036854775807 logonCount: 0 sAMAccountName: demoUser01 userPrincipalName: demoUser01@samdom.example.com pwdLastSet: 131163307277087850 userAccountControl: 512 isDeleted: TRUE lastKnownParent: CN=Users,DC=samdom,DC=example,DC=com msDS-LastKnownRDN: demoUser01 cn:: ZGVtb1VzZXIwMQpERUw6ZWJjNzFmMGQtOGQ1NC00NzRkLWI2Y2MtYzBlODVkM2YwMjhk name:: ZGVtb1VzZXIwMQpERUw6ZWJjNzFmMGQtOGQ1NC00NzRkLWI2Y2MtYzBlODVkM2YwMjhk whenChanged: 20160822091252.0Z uSNChanged: 4029 distinguishedName: CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN =Deleted Objects,DC=samdom,DC=example,DC=com # Referral ref: ldap://samdom.example.com/CN=Configuration,DC=samdom,DC=example,DC=com # Referral ref: ldap://samdom.example.com/DC=DomainDnsZones,DC=samdom,DC=example,DC=com # Referral ref: ldap://samdom.example.com/DC=ForestDnsZones,DC=samdom,DC=example,DC=com # returned 4 records # 1 entries # 3 referrals [root@DC1 ~]# ldbmodify --show-deleted -H ldap://localhost -Uadministrator Password for [SAMDOM\administrator]: dn: CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com changetype: modify delete: isDeleted - replace: distinguishedName distinguishedName: cn=demoUser01,CN=Users,DC=samdom,DC=example,DC=com - ERR: (No such attribute) "LDAP error 16 LDAP_NO_SUCH_ATTRIBUTE - <attribute 'isRecycled': no such attribute for delete on 'CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com'> <>" on DN CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com at block before line 7 Modify failed after processing 0 records This run on a 4.5rc2 Samba DC. After the update to 4.5, dbcheck was executed and everything 4.5 introduces to fix, was fixed. Please let me know if you need any further information, log, etc.
One more information: I provisioned a new Samba AD with 4.5rc2 (my normal test environment is regularly updated from one version to the next). If I run the same steps on the 4.5rc2 provisioned AD, the output of the ldbsearch command and the error I receive is different: [root@DC1 ~]# samba-tool user add demoUser01 Passw0rd Note: samba-tool user add is deprecated. Please use samba-tool user create for the same function. User 'demoUser01' created successfully [root@DC1 ~]# samba-tool user delete demoUser01 Deleted user demoUser01 [root@DC1 ~]# ldbsearch -H ldap://localhost -U administrator --show-deleted cn=demoUser01\\0ADEL:* Password for [SAMDOM\administrator]: # record 1 dn: CN=demoUser01\0ADEL:58929388-5d7c-45e6-ba78-a7793b9cb6a0,CN=Deleted Objects,DC=samdom,DC=example,DC=com objectClass: top objectClass: person objectClass: organizationalPerson objectClass: user instanceType: 4 whenCreated: 20160822093632.0Z uSNCreated: 3771 objectGUID: 58929388-5d7c-45e6-ba78-a7793b9cb6a0 objectSid: S-1-5-21-2276609257-3734461420-1769380943-1103 sAMAccountName: demoUser01 userAccountControl: 512 isDeleted: TRUE lastKnownParent: CN=Users,DC=samdom,DC=example,DC=com isRecycled: TRUE cn:: ZGVtb1VzZXIwMQpERUw6NTg5MjkzODgtNWQ3Yy00NWU2LWJhNzgtYTc3OTNiOWNiNmEw name:: ZGVtb1VzZXIwMQpERUw6NTg5MjkzODgtNWQ3Yy00NWU2LWJhNzgtYTc3OTNiOWNiNmEw whenChanged: 20160822093642.0Z uSNChanged: 3774 distinguishedName: CN=demoUser01\0ADEL:58929388-5d7c-45e6-ba78-a7793b9cb6a0,CN =Deleted Objects,DC=samdom,DC=example,DC=com # Referral ref: ldap://samdom.example.com/CN=Configuration,DC=samdom,DC=example,DC=com # Referral ref: ldap://samdom.example.com/DC=DomainDnsZones,DC=samdom,DC=example,DC=com # Referral ref: ldap://samdom.example.com/DC=ForestDnsZones,DC=samdom,DC=example,DC=com # returned 4 records # 1 entries # 3 referrals [root@DC1 ~]# ldbmodify --show-deleted -H ldap://localhost -Uadministrator Password for [SAMDOM\administrator]: Password for [SAMDOM\administrator]: dn: CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com changetype: modify delete: isDeleted - replace: distinguishedName distinguishedName: cn=demoUser01,CN=Users,DC=samdom,DC=example,DC=com - ERR: (Operations error) "LDAP error 1 LDAP_OPERATIONS_ERROR - <00002020: operations error at ../source4/dsdb/samdb/ldb_modules/tombstone_reanimate.c:372> <>" on DN CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com at block before line 7 Modify failed after processing 0 records
(In reply to Marc Muehlfeld from comment #24 & comment #23) The second comment you posted should just be due to using the wrong DN, unless it was just pasted wrong here (the DN shown in ldb search [CN=demoUser01\0ADEL:58929388-5d7c-45e6-ba78-a7793b9cb6a0,CN=Deleted] is different to the one in ldb modify [CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted]). The first comment is interesting. Do you have the recycle bin enabled? We think that we've found where the problem is. Windows' specification states that, if the recycle bin is enabled, tombstone reanimation SHOULD fail to work. It doesn't at the moment, causing issues like this. We'll have a look into this.
(In reply to Bob Campbell from comment #25) > The second comment you posted should just be due to using the wrong DN... Sorry, you're right. The error is the same like on an updated environment. -> Please ignore comment#24. > The first comment is interesting. Do you have the recycle bin enabled? Yes, it was enabled. I revert my VM snapshot and re-checked. I raised the forest+domain level to 2008_R2 and afterwards enabled the recycle bin. > We think that we've found where the problem is. Windows' specification states > that, if the recycle bin is enabled, tombstone reanimation SHOULD fail to work. > It doesn't at the moment, causing issues like this. We'll have a look into this. After reading this, I provisioned a new domain _and did not enable the recycle bin_. In this case the ldbmodify works like you described it in the Wiki. It would be great if you could get it working with recycle bin enabled too.
Thanks. Recycle bin is not a supported feature right now (some code to implement bits of it, but no tests, so unsupported).
(In reply to Andrew Bartlett from comment #27) I think I might have run this script a while ago since it was in the sources. Is there a way to reverse the changes?