Bug 10371 - Tombstone reanimation not supported
Tombstone reanimation not supported
Status: RESOLVED FIXED
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB
4.1.4
x64 Linux
: P5 regression
: 4.5
Assigned To: Andrew Bartlett
Samba QA Contact
:
Depends on: 9404 11137
Blocks:
  Show dependency treegraph
 
Reported: 2014-01-11 21:28 UTC by Marc Muehlfeld
Modified: 2016-08-25 23:50 UTC (History)
5 users (show)

See Also:


Attachments
Level 10 debug log of an unsuccessfull adrestore try (1.45 MB, text/x-log)
2014-01-11 21:28 UTC, Marc Muehlfeld
no flags Details
level 10 debug log of deleting an user (625.79 KB, text/x-log)
2014-01-11 21:30 UTC, Marc Muehlfeld
no flags Details
4.2 patch cherry-picked from master (200.07 KB, patch)
2015-03-06 04:38 UTC, Andrew Bartlett
kamenim: review+
Details
DC1.txt (1.78 KB, text/plain)
2015-03-12 21:09 UTC, Marc Muehlfeld
no flags Details
DC2.txt (1.31 KB, text/plain)
2015-03-12 21:10 UTC, Marc Muehlfeld
no flags Details
4.2 patch cherry-picked from master (with ldap server patch) (202.31 KB, patch)
2015-03-18 03:53 UTC, Andrew Bartlett
abartlet: review? (kamenim)
metze: review-
Details
failed autobuild output (134.35 KB, text/plain)
2015-03-19 03:44 UTC, Andrew Bartlett
no flags Details
Log output from ldbrename (31.33 KB, application/x-gzip)
2016-08-19 18:08 UTC, Marc Muehlfeld
no flags Details
Log output from adrestore -r (50.18 KB, application/x-gzip)
2016-08-19 18:10 UTC, Marc Muehlfeld
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marc Muehlfeld 2014-01-11 21:28:00 UTC
Created attachment 9575 [details]
Level 10 debug log of an unsuccessfull adrestore try

I do some research for writing a new Wiki HowTo about restoring AD objects and the AD recycle bin, what is really a usefull feature, but I encoutered a problem/bug:


When I delete e. g. an user, the object is moved correctly into the CN=Deleted Objects container. But when I try to restore it on any way like adrestore (Sysinternals), Lazarus (Freeware) or by hand with lpd.exe, it fails.


The restore isn't successfull, because when the object should be transfered back to it's last known parent, the following error appears:

Modify of 'distinguishedName' on CN=demo3\0ADEL:c443ca4f-9335-484a-9144-5b2d334fc948,CN=Deleted Objects,DC=samdom,DC=example,DC=com not permitted, must use 'rename' operation instead


It would be really great if someone could have a look at this. If we get the restoring AD deleted objects function or finally the complete AD recycle bin working, it would be a great benefit for all Samba users.
Comment 1 Marc Muehlfeld 2014-01-11 21:30:14 UTC
Created attachment 9576 [details]
level 10 debug log of deleting an user

This is just a level 10 debug log, of a delete user event (I deleted the user "demo3") from ADUC. The user object was in CN=Deleted Objects after the deletion.
Comment 2 Matthias Dieter Wallnöfer 2014-01-12 09:43:55 UTC
Hi Marc,

as you see in
https://git.samba.org/samba.git/?p=samba.git;a=blob;f=lib/ldb/modules/rdn_name.c;h=f44ea71f660c4e34335c25e4570e9ce42059cb3b;hb=master
the check is enforced by rdn_name_modify().

I would prefer to not change this LDB module (adding bypass flags ecc.)
since it is LDB internal but rather try other ways to circumvent it (eg.
to internally use a rename and not a modify operation).
Comment 3 Marc Muehlfeld 2014-01-12 10:34:31 UTC
Hello Matthias,


(In reply to comment #2)
> as you see in
> https://git.samba.org/samba.git/?p=samba.git;a=blob;f=lib/ldb/modules/rdn_name.c;h=f44ea71f660c4e34335c25e4570e9ce42059cb3b;hb=master
> the check is enforced by rdn_name_modify().
> 
> I would prefer to not change this LDB module (adding bypass flags ecc.)
> since it is LDB internal but rather try other ways to circumvent it (eg.
> to internally use a rename and not a modify operation).


It would be great if there could be way to get this working, as the AD-Recycle-Bin or at least the Deleted Objects recovery is a really great feature of AD, for recovering without restoring offline or the whole AD. And the half side is already working. :-)


Can you have a look at it (I'm not a programmer)? Please let me know, if I can provide any further help. E. g. testing or providing details about the Deleted Objects stuff like a step-by-step-guide for manual recovery with e. g. ldp.exe.
Comment 4 Matthias Dieter Wallnöfer 2014-01-12 22:00:04 UTC
Okay, http://support.microsoft.com/?kbid=840001 says that a delete restore on AD works like this:

dn: ...,CN=Deleted Objects,DC=...
changetype: modify
delete: isDeleted
-
replace: distinguishedName
distinguishedName: <new DN>

We do not support it yet and it requires some work.

As a workaround for Samba you could try this one:

- rename (=move) the object away from the "CN=Deleted Objects" container back to the original position
- remove the "isDeleted" attribute and re-add the correct "objectCategory" attribute

Concrete example:

./bin/ldbrename -H ldap://127.0.0.21 -Uadministrator%locDCpass1 "cn=testuser\0ADEL:2f1577df-eb9f-4d17-99d8-5a2e42e178d9,CN=Deleted Objects,DC=samba,DC=example,DC=com" "cn=testuser,cn=users,dc=samba,dc=example,dc=com"

./bin/ldbedit -H ldap://127.0.0.21 -Uadministrator%locDCpass1 -b "cn=testuser,cn=users,dc=samba,dc=example,dc=com" --show-deleted

In the editor you may remove the "isDeleted" line and re-add the "objectCategory" attribute with value "CN=Person,CN=Schema,CN=Configuration,DC=samba,DC=example,DC=com"
Comment 5 Marc Muehlfeld 2014-01-13 21:42:51 UTC
Hello Matthias,

thanks for that example workaround. I tried it and it works. 

(In reply to comment #4)
> ./bin/ldbedit -H ldap://127.0.0.21 -Uadministrator%locDCpass1 -b
> "cn=testuser,cn=users,dc=samba,dc=example,dc=com" --show-deleted

Only here is a small typo. The "cn=Users"... is case sensitive on this place. :-)


I'll use that all in the new Wiki HowTo I'm working on and also place a link to this bug report, so users can use the "official" way, when there's a fix for the modify/rename operation.
Comment 6 Kamen Mazdrashki 2014-11-27 16:02:42 UTC
To properly restore deleted objects, we should implement
support for special modify request - aka. Tombstone reanimation

A branch to track current effort in impelementing this feature:
https://github.com/kamenim/samba/compare/tombstone-reanimation-wip

For easier review, there is a pull request to outline 
only the differences to master:
https://github.com/kamenim/samba/pull/2

At the moment, we have tests and implementation ready.
All the patches though are based on Jelmers initial patches
for unittest and the branch doesn't rebase cleanly on master
at the moment.

What's left:
* rebase on master at some point
* perhaps some more tests and tests tidy up
* reviews

cheers
Comment 7 Andrew Bartlett 2015-03-06 04:38:57 UTC
Created attachment 10825 [details]
4.2 patch cherry-picked from master
Comment 8 Andrew Bartlett 2015-03-06 04:39:58 UTC
The patches in the blocking bug #11137 and #9704 need to be applied first, otherwise it will not patch cleanly.
Comment 9 Andrew Bartlett 2015-03-06 04:40:43 UTC
(correction) The patches in the blocking bug #11137 and #9404 need to be applied first, otherwise it will not patch cleanly.
Comment 10 Marc Muehlfeld 2015-03-12 21:08:15 UTC
I applied the patches from bug #11137 and bug #9404 + the one from here to 4.2.0. Then I recompiled the patched sources on both DCs in my test environment. I started Samba on both DCs and deleted user demo01 via ADUC on DC1. I can find the deleted user on both DCs for a few minutes, before it's totally lost in space. But the record on DC1 have more attributes than on DC2. It looks like, it is already recycled on DC2, but not on DC1. And when the garbage collector comes after a few minutes, it's gone on both.

On DC1 (where I have deleted the user): See attached DC1.txt
On DC2: See attached DC2.txt

To save you the time to compare:

This are the attributes of the deleted entry, that are additional on DC1 but _not_ on DC2:
badPwdCount: 0
codePage: 0
countryCode: 0
badPasswordTime: 0
lastLogoff: 0
lastLogon: 0
primaryGroupID: 513
accountExpires: 9223372036854775807
logonCount: 0
userPrincipalName: demo01@samdom.example.com
pwdLastSet: 130542401790000000
uidNumber: 10001
loginShell: /bin/bash
unixHomeDirectory: /home/demo01
gidNumber: 10000
msSFU30Name: demo01
unixUserPassword: ABCD!efgh12345$67890
givenName: Hans
sn: Dampf
displayName: Hans Dampf
telephoneNumber: 0123456
mail: Hans.Dampf@example.com
profilePath: \\DC1\profiles\demo01

This are the attributes of the deleted entry, that are additional on DC2 but _not_ on DC1:
objectGUID: 925a163c-917f-413a-81d3-8a8545aaf8af
isRecycled: TRUE




Something more I've discovered, but don't understand is, that 
# ldbsearch -H ldap://localhost -U administrator --show-deleted "CN=demo01\\0ADEL:925a163c-917f-413a-81d3-8a8545aaf8af,CN=Deleted*"                                   
returns nothing. But
# ldbsearch -H ldap://localhost -U administrator --show-deleted "CN=demo01\\0ADEL:925a163c-917f-413a-81d3-8a8545aaf8af*"
returns the record with its attributes!? I can't use the full DN with ldbsearch, ldbedit or ldbrename.
Comment 11 Marc Muehlfeld 2015-03-12 21:09:32 UTC
Created attachment 10868 [details]
DC1.txt

Result of ldbsearch executed on the DC I've deleted the user on.
Comment 12 Marc Muehlfeld 2015-03-12 21:10:09 UTC
Created attachment 10869 [details]
DC2.txt

Result of ldbsearch executed on the second DC (not the one I've deleted the record on)
Comment 13 Andrew Bartlett 2015-03-13 03:50:00 UTC
I don't think these patches were intended to revolve bug #10403 but I agree, they may not be much use with this regression in place.
Comment 14 Kamen Mazdrashki 2015-03-14 00:59:11 UTC
Comment on attachment 10825 [details]
4.2 patch cherry-picked from master

Wow Andrew, thanks for doing this backport work!
Comment 15 Andrew Bartlett 2015-03-18 03:53:09 UTC
Created attachment 10878 [details]
4.2 patch cherry-picked from master (with ldap server patch)

I missed a change to our LDAP server that impacts the test suite (but not real-world operation).  This version passes a full autobuild on sn-devel, with the other patches this depends on.
Comment 16 Stefan Metzmacher 2015-03-18 07:14:23 UTC
Comment on attachment 10878 [details]
4.2 patch cherry-picked from master (with ldap server patch)

We should first fix the flakey test before backporting this stuff.

See
https://git.samba.org/autobuild.flakey/2015-03-17-1952/samba.stdout
https://git.samba.org/autobuild.flakey/2015-03-18-0456/samba.stdout

[1657(10627)/1735 at 1h37m36s] samba4.tombstone_reanimation.python(fl2003dc)
Test Container reanimation
UNEXPECTED(failure): samba4.tombstone_reanimation.python.tombstone_reanimation.RestoreContainerObjectTestCase.test_container(fl2003dc)
REASON: Exception: Exception: Traceback (most recent call last):
  File "bin/python/samba/tests/__init__.py", line 113, in run
    testMethod()
  File "/memdisk/autobuild/fl/b2199/samba/source4/dsdb/tests/python/tombstone_reanimation.py", line 541, in test_container
    self.assertAttributesEqual(obj, attr_orig, obj_restore, attr_rest)
  File "/memdisk/autobuild/fl/b2199/samba/source4/dsdb/tests/python/tombstone_reanimation.py", line 86, in assertAttributesEqual
    self.assertEqual(attrs_orig, attrs_rest, "Actual object does not have expected attributes")
AssertionError: Actual object does not have expected attributes

FAILED (1 failures, 0 errors and 0 unexpected successes in 0 testsuites)
Comment 17 Andrew Bartlett 2015-03-19 03:44:25 UTC
Created attachment 10887 [details]
failed autobuild output

Attaching the failed autobuild output so it is not lost while we work on this
Comment 18 Andrew Bartlett 2016-07-29 03:24:58 UTC
Fixed in Samba 4.5.0rc1 by fb9af9727f4a9b1664c5a9e541981d239c72b0cf
Comment 19 Marc Muehlfeld 2016-08-19 18:08:09 UTC
I reopen this bug, because recovering deleted objects still does not work in 4.5 RC2.
Comment 20 Marc Muehlfeld 2016-08-19 18:08:55 UTC
Created attachment 12389 [details]
Log output from ldbrename

If I follow the steps we describe in the Wiki (https://wiki.samba.org/index.php/The_AD_recycle_bin#On_a_Samba_DC_2), I find the objects using the ldbsearch command. However, it's not possible to move the object using ldbrename anymore:

# ldbrename -H ldap://localhost -Uadministrator "CN=demo08\\0ADEL:85f2ba20-7474-42e5-a9f5-c6cb6356bc42,CN=Deleted Objects,DC=samdom,DC=example,DC=com" "cn=demo08,cn=Users,dc=samdom,dc=example,dc=com"
Password for [SAMDOM\administrator]:
rename of 'CN=demo08\0ADEL:85f2ba20-7474-42e5-a9f5-c6cb6356bc42,CN=Deleted Objects,DC=samdom,DC=example,DC=com' to 'cn=demo08,cn=Users,dc=samdom,dc=example,dc=com' failed - LDAP error 32 LDAP_NO_SUCH_OBJECT -  <00002030: ldb_wait from ../source4/ldap_server/ldap_backend.c:483 with LDB_WAIT_ALL: No such object (32)> <>


Log attached.
Comment 21 Marc Muehlfeld 2016-08-19 18:10:40 UTC
Created attachment 12390 [details]
Log output from adrestore -r

I tried recovering a deleted user object using the Sysinternal tool "adrestore -r". Even if it shows that the restore succeeded, it failed. If you re-run "adrestore" to scan for deleted objects, it still shows the object as deleted and able to recover.

Log attached.
Comment 22 Bob Campbell 2016-08-21 23:42:43 UTC
(In reply to Marc Muehlfeld from comment #21)
Hi Marc,

I've done this sequence of operations from a Windows machine with a Samba DC. The only way I'm able to reproduce this bug is where restoring an object would cause a duplicate DN. In this case, it exhibits exactly the behaviour you've described. With a single object, no duplicate DN, it seems to work.
Comment 23 Marc Muehlfeld 2016-08-22 09:26:41 UTC
(In reply to Bob Campbell from comment #22)
> The only way I'm able to reproduce this bug is where restoring an object 
> would cause a duplicate DN. In this case, it exhibits exactly the behaviour 
> you've described. With a single object, no duplicate DN, it seems to work.


I'm not sure, what you mean.

I saw you edited the Wiki and replaced the ldbrename with ldbmodify. However I created a new user, deleted it and tried to bring the object back using the ldbmodify way you described and fail now with a different error:






[root@DC1 ~]# samba-tool user add demoUser01 Passw0rd
Note: samba-tool user add is deprecated.  Please use samba-tool user create for the same function.
User 'demoUser01' created successfully




[root@DC1 ~]# samba-tool user delete demoUser01
Deleted user demoUser01




[root@DC1 ~]# ldbsearch -H ldap://localhost -U administrator --show-deleted cn=demoUser01\\0ADEL:*
Password for [SAMDOM\administrator]:
# record 1
dn: CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
instanceType: 4
whenCreated: 20160822091207.0Z
uSNCreated: 4026
objectGUID: ebc71f0d-8d54-474d-b6cc-c0e85d3f028d
badPwdCount: 0
codePage: 0
countryCode: 0
badPasswordTime: 0
lastLogoff: 0
lastLogon: 0
primaryGroupID: 513
objectSid: S-1-5-21-469703510-2364959079-1506205053-1130
accountExpires: 9223372036854775807
logonCount: 0
sAMAccountName: demoUser01
userPrincipalName: demoUser01@samdom.example.com
pwdLastSet: 131163307277087850
userAccountControl: 512
isDeleted: TRUE
lastKnownParent: CN=Users,DC=samdom,DC=example,DC=com
msDS-LastKnownRDN: demoUser01
cn:: ZGVtb1VzZXIwMQpERUw6ZWJjNzFmMGQtOGQ1NC00NzRkLWI2Y2MtYzBlODVkM2YwMjhk
name:: ZGVtb1VzZXIwMQpERUw6ZWJjNzFmMGQtOGQ1NC00NzRkLWI2Y2MtYzBlODVkM2YwMjhk
whenChanged: 20160822091252.0Z
uSNChanged: 4029
distinguishedName: CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN
 =Deleted Objects,DC=samdom,DC=example,DC=com

# Referral
ref: ldap://samdom.example.com/CN=Configuration,DC=samdom,DC=example,DC=com

# Referral
ref: ldap://samdom.example.com/DC=DomainDnsZones,DC=samdom,DC=example,DC=com

# Referral
ref: ldap://samdom.example.com/DC=ForestDnsZones,DC=samdom,DC=example,DC=com

# returned 4 records
# 1 entries
# 3 referrals




[root@DC1 ~]# ldbmodify --show-deleted -H ldap://localhost -Uadministrator
Password for [SAMDOM\administrator]:
dn: CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com
changetype: modify
delete: isDeleted
-
replace: distinguishedName
distinguishedName: cn=demoUser01,CN=Users,DC=samdom,DC=example,DC=com
-
ERR: (No such attribute) "LDAP error 16 LDAP_NO_SUCH_ATTRIBUTE -  <attribute 'isRecycled': no such attribute for delete on 'CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com'> <>" on DN CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com at block before line 7
Modify failed after processing 0 records







This run on a 4.5rc2 Samba DC. After the update to 4.5, dbcheck was executed and everything 4.5 introduces to fix, was fixed. 


Please let me know if you need any further information, log, etc.
Comment 24 Marc Muehlfeld 2016-08-22 09:46:44 UTC
One more information: I provisioned a new Samba AD with 4.5rc2 (my normal test environment is regularly updated from one version to the next). If I run the same steps on the 4.5rc2 provisioned AD, the output of the ldbsearch command and the error I receive is different:





[root@DC1 ~]# samba-tool user add demoUser01 Passw0rd
Note: samba-tool user add is deprecated.  Please use samba-tool user create for the same function.
User 'demoUser01' created successfully




[root@DC1 ~]# samba-tool user delete demoUser01
Deleted user demoUser01




[root@DC1 ~]# ldbsearch -H ldap://localhost -U administrator --show-deleted cn=demoUser01\\0ADEL:*
Password for [SAMDOM\administrator]:
# record 1
dn: CN=demoUser01\0ADEL:58929388-5d7c-45e6-ba78-a7793b9cb6a0,CN=Deleted Objects,DC=samdom,DC=example,DC=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
instanceType: 4
whenCreated: 20160822093632.0Z
uSNCreated: 3771
objectGUID: 58929388-5d7c-45e6-ba78-a7793b9cb6a0
objectSid: S-1-5-21-2276609257-3734461420-1769380943-1103
sAMAccountName: demoUser01
userAccountControl: 512
isDeleted: TRUE
lastKnownParent: CN=Users,DC=samdom,DC=example,DC=com
isRecycled: TRUE
cn:: ZGVtb1VzZXIwMQpERUw6NTg5MjkzODgtNWQ3Yy00NWU2LWJhNzgtYTc3OTNiOWNiNmEw
name:: ZGVtb1VzZXIwMQpERUw6NTg5MjkzODgtNWQ3Yy00NWU2LWJhNzgtYTc3OTNiOWNiNmEw
whenChanged: 20160822093642.0Z
uSNChanged: 3774
distinguishedName: CN=demoUser01\0ADEL:58929388-5d7c-45e6-ba78-a7793b9cb6a0,CN
 =Deleted Objects,DC=samdom,DC=example,DC=com

# Referral
ref: ldap://samdom.example.com/CN=Configuration,DC=samdom,DC=example,DC=com

# Referral
ref: ldap://samdom.example.com/DC=DomainDnsZones,DC=samdom,DC=example,DC=com

# Referral
ref: ldap://samdom.example.com/DC=ForestDnsZones,DC=samdom,DC=example,DC=com

# returned 4 records
# 1 entries
# 3 referrals




[root@DC1 ~]# ldbmodify --show-deleted -H ldap://localhost -Uadministrator
Password for [SAMDOM\administrator]:
Password for [SAMDOM\administrator]:
dn: CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com
changetype: modify
delete: isDeleted
-
replace: distinguishedName
distinguishedName: cn=demoUser01,CN=Users,DC=samdom,DC=example,DC=com
-
ERR: (Operations error) "LDAP error 1 LDAP_OPERATIONS_ERROR -  <00002020: operations error at ../source4/dsdb/samdb/ldb_modules/tombstone_reanimate.c:372> <>" on DN CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted Objects,DC=samdom,DC=example,DC=com at block before line 7
Modify failed after processing 0 records
Comment 25 Bob Campbell 2016-08-23 01:31:53 UTC
(In reply to Marc Muehlfeld from comment #24 & comment #23)
The second comment you posted should just be due to using the wrong DN, unless it was just pasted wrong here (the DN shown in ldb search [CN=demoUser01\0ADEL:58929388-5d7c-45e6-ba78-a7793b9cb6a0,CN=Deleted] is different to the one in ldb modify [CN=demoUser01\0ADEL:ebc71f0d-8d54-474d-b6cc-c0e85d3f028d,CN=Deleted]).

The first comment is interesting. Do you have the recycle bin enabled?
We think that we've found where the problem is. Windows' specification states that, if the recycle bin is enabled, tombstone reanimation SHOULD fail to work. It doesn't at the moment, causing issues like this. We'll have a look into this.
Comment 26 Marc Muehlfeld 2016-08-25 23:38:21 UTC
(In reply to Bob Campbell from comment #25)
> The second comment you posted should just be due to using the wrong DN...

Sorry, you're right. The error is the same like on an updated environment.
-> Please ignore comment#24.



> The first comment is interesting. Do you have the recycle bin enabled?

Yes, it was enabled. I revert my VM snapshot and re-checked. I raised the forest+domain level to 2008_R2 and afterwards enabled the recycle bin.



> We think that we've found where the problem is. Windows' specification states 
> that, if the recycle bin is enabled, tombstone reanimation SHOULD fail to work. 
> It doesn't at the moment, causing issues like this. We'll have a look into this.

After reading this, I provisioned a new domain _and did not enable the recycle bin_. In this case the ldbmodify works like you described it in the Wiki. It would be great if you could get it working with recycle bin enabled too.
Comment 27 Andrew Bartlett 2016-08-25 23:50:06 UTC
Thanks.  Recycle bin is not a supported feature right now (some code to implement bits of it, but no tests, so unsupported).