Bug 14902 - User with multiple spaces (eg Fred<space><space>Nurk) become un-deletable after 4.15.0, 4.14.10, 4.13.14
Summary: User with multiple spaces (eg Fred<space><space>Nurk) become un-deletable aft...
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB (show other bugs)
Version: unspecified
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Andrew Bartlett
QA Contact: Samba QA Contact
: 15029 (view as bug list)
Depends on:
Reported: 2021-11-11 21:06 UTC by Douglas Bagnall
Modified: 2022-05-04 04:57 UTC (History)
5 users (show)

See Also:

Patch for v4-15-test (2.82 KB, patch)
2021-12-07 10:26 UTC, Stefan Metzmacher
slow: review+
Patch for v4-14-test (2.82 KB, patch)
2021-12-07 10:27 UTC, Stefan Metzmacher
slow: review+
Patch for v4-13-test (2.82 KB, patch)
2021-12-07 10:27 UTC, Stefan Metzmacher
slow: review+

Note You need to log in before you can comment on or make changes to this bug.
Description Douglas Bagnall 2021-11-11 21:06:13 UTC
As reported by Denis Cardon on the Samba mailing list (https://lists.samba.org/archive/samba/2021-November/238452.html), slightly edited here:

We have had this issue a few time today with latest 4.14 when upgrading client installations (I didn't have time to check if it was latest 4.14.10 or if it happened in some earlier version).

If you have DN strings with consecutive space characters (yeah, it shouldn't happen, but if one can do it, it will be done), then the upgrade will break a few things.

In the replication you'll get this kind of error message :

[2021/11/10 15:15:33.150632,  1] ../../source4/dsdb/repl/replicated_objects.c:904(dsdb_replicated_objects_commit)
  Failed to apply records: operational_search_post_process failed for attribute 'parentGUID' - No such Base DN: CN=USERNAME  Romain,OU=Sync Azure,DC=mydomain,DC=lan: Operations error
[2021/11/10 15:15:33.150754,  0] ../../source4/dsdb/repl/drepl_out_helpers.c:1184(dreplsrv_op_pull_source_apply_changes_trigger)

If you try a samba-tool dbcheck --cross-ncs, you'll may get this kind of error :

ERROR: Object CN=USERNAME  Romain,OU=Sync Azure,DC=mydomain,DC=lan disappeared during check

Another symptom is that the search with an attribute (like samba-tool user show dcardon) does work, but a ldbsearch with a DN like below (beware of the two spaces) does not work
'CN=denis  cardon,OU=test,DC=test,DC=lan'

If you have this case, a reindex should fix it (it need to be run on each DC)
samba-tool dbcheck --reindex

Another option is to fix this before upgrade, or if it is already upgraded, downgrade, fix and then upgrade.

If you have the case where you have two quasi-identical entries, one with two space and one with only one (ie CN=denis cardon, and CN=denis cardon), then you have to delete one of them before re-indexing (yeah we have seen this one today also).

There seems to be a discrepancy in the way multiple spaces are handled in the index and in the DN string itself.

Note : if you recreate an entry with multiple consecutive spaces after upgrade it seems to work though...
Comment 1 Douglas Bagnall 2021-11-11 21:49:24 UTC
I suspect this is due to fixes for bug 14656 which were backported in the November 2021 security release to 4.14.10 (and 4.13).

Those patches changed the way spaces are handled in searches and maybe indexes. I think they are probably correct, at least insofar as they do what the old code was trying to do.

However, for the indexes, it might be that the old index has treated values with the incorrect space collapsing, which would I think have mapped  
'CN=denis  cardon' to  'CN=denis ccardon', while the new code uses one space 'CN=denis cardon'. So the index doesn't match.

The real bug is this hypothesis is that the values are not being escaped first ('CN=denis\ \ cardon') for the indexing.

Before 'CN=denis  cardon' and 'CN=denis ccardon' were quasi-identical pair, and we never/rarely saw it because no-one is called ccardon.
Comment 2 Andrew Bartlett 2021-12-01 00:06:22 UTC
The workaround for now is to run:

samba-tool dbcheck --reindex

We should perhaps force a re-index in a future version upgrade (by bumping SAMDB_INDEXING_VERSION to ensure this is done)

/* change this when we change something in our schema code that
 * requires a re-index of the database

However this will force a reindex regardless which is CPU intensive.
Comment 3 Andrew Bartlett 2021-12-01 00:25:12 UTC
Given that users use 'samba-tool dbcheck' to assure themselves that the DB is not at fault, a future task could be to upgrade dbcheck to search for each object by string DN, as that would detect this in the future.

Sadly dbcheck is not comprehensive, essentially being a set of fixes for things we know have gone wrong in the past.
Comment 4 Samba QA Contact 2021-12-03 12:55:23 UTC
This bug was referenced in samba master:

Comment 5 Stefan Metzmacher 2021-12-07 10:26:41 UTC
Created attachment 17048 [details]
Patch for v4-15-test
Comment 6 Stefan Metzmacher 2021-12-07 10:27:07 UTC
Created attachment 17049 [details]
Patch for v4-14-test
Comment 7 Stefan Metzmacher 2021-12-07 10:27:47 UTC
Created attachment 17050 [details]
Patch for v4-13-test
Comment 8 Ralph Böhme 2021-12-07 16:53:55 UTC
Reassigning to Jule for inclusion in 4.13, 4.14 and 4.15.
Comment 9 Stefan Metzmacher 2021-12-08 09:30:48 UTC
(In reply to Ralph Böhme from comment #8)

Pushed to autobuild-v4-{15,14,13}-test
Comment 10 Samba QA Contact 2021-12-08 10:55:34 UTC
This bug was referenced in samba v4-15-test:

Comment 11 Samba QA Contact 2021-12-08 14:58:53 UTC
This bug was referenced in samba v4-14-test:

Comment 12 Samba QA Contact 2021-12-08 14:59:10 UTC
This bug was referenced in samba v4-15-stable (Release samba-4.15.3):

Comment 13 Samba QA Contact 2021-12-08 16:55:12 UTC
This bug was referenced in samba v4-13-test:

Comment 14 Jule Anger 2021-12-08 18:05:48 UTC
Closing out bug report.

Comment 15 Andrew Bartlett 2021-12-08 21:26:35 UTC
Reopening to consider a further fix per comment #2
Comment 16 Samba QA Contact 2021-12-15 14:25:14 UTC
This bug was referenced in samba v4-13-stable (Release samba-4.13.15):

Comment 17 Samba QA Contact 2021-12-15 14:53:35 UTC
This bug was referenced in samba v4-14-stable (Release samba-4.14.11):

Comment 18 Andrew Bartlett 2022-05-04 04:57:42 UTC
*** Bug 15029 has been marked as a duplicate of this bug. ***