Bug 10294 - Failed to Join Domain - uncaught exception - attribute 'isRecycled'
Summary: Failed to Join Domain - uncaught exception - attribute 'isRecycled'
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB (show other bugs)
Version: 4.1.2
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks: 10077
  Show dependency treegraph
 
Reported: 2013-11-28 03:14 UTC by Tim Wong
Modified: 2014-07-17 18:24 UTC (History)
1 user (show)

See Also:


Attachments
4.1 patch cherry-picked from master (1.14 KB, patch)
2014-07-07 22:08 UTC, Andrew Bartlett
abartlet: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Wong 2013-11-28 03:14:51 UTC
I got the below while I join another ubuntu 32 bit samba 4.1.2 to my exisiting domain (also 4.1.2).

The error found that my previous JTHDC2, which previously crashed and removed from Domain, but it still have some records inside. run samba-tools dbcheck --fix also doesnt fix the problems.

ERROR(ldb): uncaught exception - attribute 'isRecycled': invalid modify flags on                                                                                         'CN=NTDS Settings\0ADEL:5264f614-f999-4989-b65a-5e1a38f5f108,CN=JTHDC2\0ADEL:71                                                                                        1f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Site                                                                                        s,CN=Configuration,DC=jtiasa,DC=jayatiasa,DC=local': 0x0
  File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",                                                                                         line 175, in _run
    return self.run(*args, **kwargs)
  File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/domain.py", li                                                                                        ne 552, in run
    machinepass=machinepass, use_ntvfs=use_ntvfs, dns_backend=dns_backend)
  File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1172,                                                                                         in join_DC
    ctx.do_join()
  File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1077,                                                                                         in do_join
    ctx.join_replicate()
  File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 851, i                                                                                        n join_replicate
    ctx.local_samdb.transaction_commit()

please help and thanks in advance !
Comment 1 Leonid Rybka 2013-12-09 08:32:58 UTC
(In reply to comment #0)
> I got the below while I join another ubuntu 32 bit samba 4.1.2 to my exisiting
> domain (also 4.1.2).
> 
> The error found that my previous JTHDC2, which previously crashed and removed
> from Domain, but it still have some records inside. run samba-tools dbcheck
> --fix also doesnt fix the problems.
> 
> ERROR(ldb): uncaught exception - attribute 'isRecycled': invalid modify flags
> on                                                                             
>            'CN=NTDS
> Settings\0ADEL:5264f614-f999-4989-b65a-5e1a38f5f108,CN=JTHDC2\0ADEL:71         
>                                                                              
> 1f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Site
>                                                                                
>        s,CN=Configuration,DC=jtiasa,DC=jayatiasa,DC=local': 0x0
>   File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/__init__.py",
>                                                                                
>         line 175, in _run
>     return self.run(*args, **kwargs)
>   File "/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/domain.py",
> li                                                                             
>           ne 552, in run
>     machinepass=machinepass, use_ntvfs=use_ntvfs, dns_backend=dns_backend)
>   File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1172,
>                                                                                
>         in join_DC
>     ctx.do_join()
>   File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 1077,
>                                                                                
>         in do_join
>     ctx.join_replicate()
>   File "/usr/local/samba/lib/python2.7/site-packages/samba/join.py", line 851,
> i                                                                              
>          n join_replicate
>     ctx.local_samdb.transaction_commit()
> 
> please help and thanks in advance !

Got the same problem while joining 4.1.2 to existing Windows Server 2012 R2 with Recycle Bin enabled.
Provision stops on committing sam.d database with the same uncaught exception.
In our case removing those 0ADEL objects on windows server fixed provision problem.
If it is windows server you provision from then powershell commands like those will fix provision(DOUBLE CHECK THEM BEFORE EXECUTE!!! do not do it if you not sure what you are removing from AD!):

Get-ADObject 'CN=NTDS Settings\0ADEL:5264f614-f999-4989-b65a-5e1a38f5f108,CN=JTHDC2\0ADEL:711f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=jtiasa,DC=jayatiasa,DC=local' -IncludeDeletedObjects | Remove-ADObject

and...

Get-ADObject 'CN=JTHDC2\0ADEL:711f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=jtiasa,DC=jayatiasa,DC=local' -IncludeDeletedObjects | Remove-ADObject

After that in ldp.exe those objects will appear with isRecycled attribute set to TRUE and provision will not crash on them.
Comment 2 Tim Wong 2013-12-11 06:39:36 UTC
(In reply to comment #1)
> 
> Got the same problem while joining 4.1.2 to existing Windows Server 2012 R2
> with Recycle Bin enabled.
> Provision stops on committing sam.d database with the same uncaught exception.
> In our case removing those 0ADEL objects on windows server fixed provision
> problem.
> If it is windows server you provision from then powershell commands like those
> will fix provision(DOUBLE CHECK THEM BEFORE EXECUTE!!! do not do it if you not
> sure what you are removing from AD!):
> 
> Get-ADObject 'CN=NTDS
> Settings\0ADEL:5264f614-f999-4989-b65a-5e1a38f5f108,CN=JTHDC2\0ADEL:711f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=jtiasa,DC=jayatiasa,DC=local'
> -IncludeDeletedObjects | Remove-ADObject
> 
> and...
> 
> Get-ADObject
> 'CN=JTHDC2\0ADEL:711f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=jtiasa,DC=jayatiasa,DC=local'
> -IncludeDeletedObjects | Remove-ADObject
> 
> After that in ldp.exe those objects will appear with isRecycled attribute set
> to TRUE and provision will not crash on them.

For our setup currently using 2 server with Samba-4.1.2, this message happened when we join ubuntu server as DC. As such, we don't have any Windows server in this environment. As such, we cannot use powershell remove the object. The "JTHDC2" is the also a test joined server, but it doesn't go through proper way to demote it, because the server crash after running series of software testing & configuration.

Please tell me how to remove this record by using samba-tools or any linux command. thanks a lot
Comment 3 Leonid Rybka 2013-12-11 14:30:55 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > 
> > Got the same problem while joining 4.1.2 to existing Windows Server 2012 R2
> > with Recycle Bin enabled.
> > Provision stops on committing sam.d database with the same uncaught exception.
> > In our case removing those 0ADEL objects on windows server fixed provision
> > problem.
> > If it is windows server you provision from then powershell commands like those
> > will fix provision(DOUBLE CHECK THEM BEFORE EXECUTE!!! do not do it if you not
> > sure what you are removing from AD!):
> > 
> > Get-ADObject 'CN=NTDS
> > Settings\0ADEL:5264f614-f999-4989-b65a-5e1a38f5f108,CN=JTHDC2\0ADEL:711f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=jtiasa,DC=jayatiasa,DC=local'
> > -IncludeDeletedObjects | Remove-ADObject
> > 
> > and...
> > 
> > Get-ADObject
> > 'CN=JTHDC2\0ADEL:711f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=jtiasa,DC=jayatiasa,DC=local'
> > -IncludeDeletedObjects | Remove-ADObject
> > 
> > After that in ldp.exe those objects will appear with isRecycled attribute set
> > to TRUE and provision will not crash on them.
> 
> For our setup currently using 2 server with Samba-4.1.2, this message happened
> when we join ubuntu server as DC. As such, we don't have any Windows server in
> this environment. As such, we cannot use powershell remove the object. The
> "JTHDC2" is the also a test joined server, but it doesn't go through proper way
> to demote it, because the server crash after running series of software testing
> & configuration.
> 
> Please tell me how to remove this record by using samba-tools or any linux
> command. thanks a lot

Well, that is sad. Though i`m not quite sure that this is the right place to share samba usage experience, i`ll write some more of it.

Removing 0ADELs from sam.ldb on samba DCs is next big problem that i`m trying to deal with. In theory removal procedure with samba tools (not samba-tool) as simple, as help pages says:
  ldbdel <options> <DN...>

Practically it does not remove objects in most cases. Especially 0ADELs.
One of the resolvable error is that ldbdel exits with child existence warning. That is quite simple to resolve.
But the most strange behavior of ldbdel is error message "entry does not exist!" while in ldbsearch and ldbedit you can clearly see that entry. What i have noticed in common about such objects is that they do not have (or maybe ldbsearch display them in that way) "ObjectClass" field(s). And that is very strange too, as i can not imagine how ldap entry can exist without ObjectClass.

Anyway, the most righteous way to remove deleted objects is to wait for tombstone lifetime period and deleted object lifetime period. That is how MSAD works. I don`t know if samba doing the same way.
Command below will show you both lifetime period(days):

/usr/local/samba/bin/ldbsearch -H /usr/local/samba/private/sam.ldb -b 'cn=directory service,cn=windows nt,cn=services,cn=configuration,DC=<YourDomainDC>' tombstonelifetime msDS-DeletedObjectLifetime

Try to wait. And do not do any provision for it will create new deleted objects. If your object still in there, well, try to inspect it with ldbsearch:

/usr/local/samba/bin/ldbsearch -H /usr/local/samba/private/sam.ldb -b 'CN=NTDS
Settings\0ADEL:5264f614-f999-4989-b65a-5e1a38f5f108,CN=JTHDC2\0ADEL:71                                                                             
1f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Site'

Then if it looks normal try to remove it with ldbdel.
However i must warn you that for us any manipulation with deleted objects on samba leads to dreplsrv process crash with internal error. And that is yet another problem.
Comment 4 Tim Wong 2013-12-12 07:07:15 UTC
The lifetime is 180 days, so I have to wait half year in order to join another server as DC?

ldbsearch return no such base DN

Any fix or patch for this, at least shorten the lifetime.

> 
> Well, that is sad. Though i`m not quite sure that this is the right place to
> share samba usage experience, i`ll write some more of it.
> 
> Removing 0ADELs from sam.ldb on samba DCs is next big problem that i`m trying
> to deal with. In theory removal procedure with samba tools (not samba-tool) as
> simple, as help pages says:
>   ldbdel <options> <DN...>
> 
> Practically it does not remove objects in most cases. Especially 0ADELs.
> One of the resolvable error is that ldbdel exits with child existence warning.
> That is quite simple to resolve.
> But the most strange behavior of ldbdel is error message "entry does not
> exist!" while in ldbsearch and ldbedit you can clearly see that entry. What i
> have noticed in common about such objects is that they do not have (or maybe
> ldbsearch display them in that way) "ObjectClass" field(s). And that is very
> strange too, as i can not imagine how ldap entry can exist without ObjectClass.
> 
> Anyway, the most righteous way to remove deleted objects is to wait for
> tombstone lifetime period and deleted object lifetime period. That is how MSAD
> works. I don`t know if samba doing the same way.
> Command below will show you both lifetime period(days):
> 
> /usr/local/samba/bin/ldbsearch -H /usr/local/samba/private/sam.ldb -b
> 'cn=directory service,cn=windows
> nt,cn=services,cn=configuration,DC=<YourDomainDC>' tombstonelifetime
> msDS-DeletedObjectLifetime
> 
> Try to wait. And do not do any provision for it will create new deleted
> objects. If your object still in there, well, try to inspect it with ldbsearch:
> 
> /usr/local/samba/bin/ldbsearch -H /usr/local/samba/private/sam.ldb -b 'CN=NTDS
> Settings\0ADEL:5264f614-f999-4989-b65a-5e1a38f5f108,CN=JTHDC2\0ADEL:71          
> 1f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Site'
> 
> Then if it looks normal try to remove it with ldbdel.
> However i must warn you that for us any manipulation with deleted objects on
> samba leads to dreplsrv process crash with internal error. And that is yet
> another problem.
Comment 5 Leonid Rybka 2013-12-14 09:23:46 UTC
(In reply to comment #4)
> The lifetime is 180 days, so I have to wait half year in order to join another
> server as DC?
> 
> ldbsearch return no such base DN
> 
> Any fix or patch for this, at least shorten the lifetime.
> 
> > 
> > Well, that is sad. Though i`m not quite sure that this is the right place to
> > share samba usage experience, i`ll write some more of it.
> > 
> > Removing 0ADELs from sam.ldb on samba DCs is next big problem that i`m trying
> > to deal with. In theory removal procedure with samba tools (not samba-tool) as
> > simple, as help pages says:
> >   ldbdel <options> <DN...>
> > 
> > Practically it does not remove objects in most cases. Especially 0ADELs.
> > One of the resolvable error is that ldbdel exits with child existence warning.
> > That is quite simple to resolve.
> > But the most strange behavior of ldbdel is error message "entry does not
> > exist!" while in ldbsearch and ldbedit you can clearly see that entry. What i
> > have noticed in common about such objects is that they do not have (or maybe
> > ldbsearch display them in that way) "ObjectClass" field(s). And that is very
> > strange too, as i can not imagine how ldap entry can exist without ObjectClass.
> > 
> > Anyway, the most righteous way to remove deleted objects is to wait for
> > tombstone lifetime period and deleted object lifetime period. That is how MSAD
> > works. I don`t know if samba doing the same way.
> > Command below will show you both lifetime period(days):
> > 
> > /usr/local/samba/bin/ldbsearch -H /usr/local/samba/private/sam.ldb -b
> > 'cn=directory service,cn=windows
> > nt,cn=services,cn=configuration,DC=<YourDomainDC>' tombstonelifetime
> > msDS-DeletedObjectLifetime
> > 
> > Try to wait. And do not do any provision for it will create new deleted
> > objects. If your object still in there, well, try to inspect it with ldbsearch:
> > 
> > /usr/local/samba/bin/ldbsearch -H /usr/local/samba/private/sam.ldb -b 'CN=NTDS
> > Settings\0ADEL:5264f614-f999-4989-b65a-5e1a38f5f108,CN=JTHDC2\0ADEL:71          
> > 1f391a-d545-4d8b-be43-ee13cc867cb0,CN=Servers,CN=Default-First-Site-Name,CN=Site'
> > 
> > Then if it looks normal try to remove it with ldbdel.
> > However i must warn you that for us any manipulation with deleted objects on
> > samba leads to dreplsrv process crash with internal error. And that is yet
> > another problem.

Life time can be changed like any other MSAD object. Through ADSI edit from windows client, or ldbedit on linux shell.
That`s strange. Are you sure you searching it on provision source DC?
I think if you find it and add "isRecycled: TRUE" to it your provision will be good.

I want to make it clear. I`m just using samba. No, i don`t have any patches.
Let us stick to the subject. i think it is a real bug.
Comment 6 Andrew Bartlett 2013-12-15 01:47:16 UTC
To stop the beating around the bush.  Yes, this is a real bug, in Samba 4.1 but not 4.0, due to handling of incoming deleted objects.

I even started on a patch, but the issue is that I can't figure out how an object can be be isRecycled and need to be deleted locally (which is what this routine is trying to do, fix up objects that are not deleted on the other server, but are already considered deleted on Samba).
Comment 7 Stefan Metzmacher 2014-07-07 11:10:54 UTC
(In reply to comment #6)
> To stop the beating around the bush.  Yes, this is a real bug, in Samba 4.1 but
> not 4.0, due to handling of incoming deleted objects.
> 
> I even started on a patch, but the issue is that I can't figure out how an
> object can be be isRecycled and need to be deleted locally (which is what this
> routine is trying to do, fix up objects that are not deleted on the other
> server, but are already considered deleted on Samba).

Andrew, where can I find this patch?
Comment 8 Andrew Bartlett 2014-07-07 22:08:10 UTC
Created attachment 10077 [details]
4.1 patch cherry-picked from master

You found and fixed it in 5b22222421c77c8c379c828c5da7e6c8c38cfb88.
Comment 9 Stefan Metzmacher 2014-07-07 22:25:23 UTC
Comment on attachment 10077 [details]
4.1 patch cherry-picked from master

Ah, that one :-)
Can you assign the bug to Karolin, for the next release?
Comment 10 Karolin Seeger 2014-07-11 07:56:02 UTC
Pushed to autobuild-v4-1-test.
Comment 11 Karolin Seeger 2014-07-17 18:24:42 UTC
Pushed to v4-1-test.
Closing out bug report.

Thanks!