Bug 8077 - userParameters needs not to be converted to utf8
Summary: userParameters needs not to be converted to utf8
Status: ASSIGNED
Alias: None
Product: Samba 4.0
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB (show other bugs)
Version: unspecified
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Andrew Bartlett
QA Contact: samba4-qa@samba.org
URL:
Keywords:
: 9835 11610 11987 (view as bug list)
Depends on: 10130 10374
Blocks: 11881 9835
  Show dependency treegraph
 
Reported: 2011-04-12 10:09 UTC by Stefan Metzmacher
Modified: 2016-11-29 18:53 UTC (History)
10 users (show)

See Also:


Attachments
Binary form of the base64 version of userParameters attribute (191 bytes, application/octet-stream)
2011-06-20 12:48 UTC, Matthieu Patou
no flags Details
Trace from Windows XP while manipulating session tab in dsa.msc (228.01 KB, application/x-gzip)
2011-06-20 19:10 UTC, Matthieu Patou
no flags Details
First Patch for v4-1-test (1.27 KB, patch)
2013-09-27 07:29 UTC, Stefan Metzmacher
asn: review+
Details
First Patch for v4-0-test (1.27 KB, patch)
2013-09-27 07:38 UTC, Stefan Metzmacher
asn: review+
Details
patch for master (2.35 KB, patch)
2013-10-04 03:34 UTC, Andrew Bartlett
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Metzmacher 2011-04-12 10:09:34 UTC
While doing migration tests from samba3 to ad I noticed
that it's wrong to copy just the utf16 array (that comes via SAMR)
into the userParameters attribute. The terminal server settings
will be returned corrupted when returned by a windows server,
as it will convert the userParameters attribute from utf8 to utf16
when returning it in SAMR and NETLOGON.

But note that the userParameters attribute contains 0 bytes within
the utf8 stream, so it's not a normal null terminated utf8 string.

We need some tests which demonstrate this behavior as this has also
impact on the schema syntax verification on originating updates.

Andrew, you may want to look at that while your in charset land:-)

metze
Comment 1 Andrew Bartlett 2011-04-12 23:00:29 UTC
If you can get me a pair of UTF8 and UTF16 (ldap and SAMR) representation examples, base64 encoded, I'll be very glad to add the charset level tests. 

We may even be able to add a python test to confirm the interactions between LDAP and SAMR.
Comment 2 Matthieu Patou 2011-06-20 12:47:25 UTC
Andrew,

here is the LDAP representation in base64.
userParameters:: ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI
 CAgUAUaCAFDdHhDZmdQcmVzZW5045S15pSx5oiw44GiGAgBQ3R4Q2ZnRmxhZ3Mx44Cw44Gm44Cy44
 C5EggBQ3R4U2hhZG9344Sw44Cw44Cw44CwKAgBQ3R4TWF4Q29ubmVjdGlvblRpbWXjgLDjoaLmjLT
 mhLAqAgFDdHhNaW5FbmNyeXB0aW9uTGV2ZWzjhLA=


That's an attribute coming from a Windows 2008R2 server, it looks like a utf-8 (at least it didn't look like utf-16).
Comment 3 Matthieu Patou 2011-06-20 12:48:45 UTC
Created attachment 6597 [details]
Binary form of the base64 version of userParameters attribute
Comment 4 Matthieu Patou 2011-06-20 12:50:01 UTC
And it mess with samba-tool dbcheck:

ERROR: Normalisation error for attribute userParameters in CN=User, DC=home, DC=matws,DC=net
value '                                                CtxCfgPresent551e0bCtxCfgFlags100f0e0CtxShadow0100000*CtxMinEncryptionLevel0' should be ' '
Comment 5 Stefan Metzmacher 2011-06-20 12:50:43 UTC
Can you also paste the hexdump -C of the binary blob.
Comment 6 Matthieu Patou 2011-06-20 19:10:03 UTC
So I redid the full stuff:

here is the hexdump
0000000 2020 2020 2020 2020 2020 2020 2020 2020
*
0000030 0550 081a 4301 7874 6643 5067 6572 6573
0000040 746e 94e3 e6b5 b194 88e6 e3b0 a281 0818
0000050 4301 7874 6643 4667 616c 7367 e331 b080
0000060 81e3 e3a6 b280 80e3 12b9 0108 7443 5378
0000070 6168 6f64 e377 b084 80e3 e3b0 b080 80e3
0000080 1cb0 0108 7443 4d78 7861 6449 656c 6954
0000090 656d 80e3 e3b0 a2a1 8ce6 e6b4 b084 022a
00000a0 4301 7874 694d 456e 636e 7972 7470 6f69
00000b0 4c6e 7665 6c65 84e3 00b0               
00000b9

of

dn: CN=test,CN=Users,DC=w2k8r2,DC=home,DC=matws,DC=net                          
userParameters:: ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI  
 CAgUAUaCAFDdHhDZmdQcmVzZW5045S15pSx5oiw44GiGAgBQ3R4Q2ZnRmxhZ3Mx44Cw44Gm44Cy44  
 C5EggBQ3R4U2hhZG9344Sw44Cw44Cw44CwHAgBQ3R4TWF4SWRsZVRpbWXjgLDjoaLmjLTmhLAqAgF  
 DdHhNaW5FbmNyeXB0aW9uTGV2ZWzjhLA=     


In the attached trace see the SetUserInfo where we can clearly see that this RPC + QueryUserInfo exchange information encoded in UTF-16.

So
Comment 7 Matthieu Patou 2011-06-20 19:10:44 UTC
Created attachment 6599 [details]
Trace from Windows XP while manipulating session tab in dsa.msc
Comment 8 Matthieu Patou 2011-07-06 22:14:42 UTC
So in order to reproduce:

* Start dsa.msc 
* Select administrator user
* Click on the Session tab
* Set/Change a value
* Click Ok


As soon as you click ok the value is changed is the database (through a RPC call) and you get the values I put in the attachment.
Comment 9 Andrew Bartlett 2013-06-27 08:36:57 UTC
*** Bug 9835 has been marked as a duplicate of this bug. ***
Comment 10 Karolin Seeger 2013-08-30 08:28:01 UTC
Is this a blocker for 4.1.0 or 4.2?
Comment 11 Karolin Seeger 2013-08-30 09:55:23 UTC
(In reply to comment #10)
> Is this a blocker for 4.1.0 or 4.2?

Comment from Metze:

https://bugzilla.samba.org/show_bug.cgi?id=8077 (userParameters needs to
be stored as utf8)
is a real bad bug, I'd wait till the last minute before deferring it,
but it'll require a lot of work to fix this :-(
Comment 12 Stefan Metzmacher 2013-09-27 07:29:41 UTC
Created attachment 9243 [details]
First Patch for v4-1-test
Comment 13 Andreas Schneider 2013-09-27 07:34:06 UTC
Comment on attachment 9243 [details]
First Patch for v4-1-test

LGTM
Comment 14 Stefan Metzmacher 2013-09-27 07:38:34 UTC
Created attachment 9244 [details]
First Patch for v4-0-test
Comment 15 Stefan Metzmacher 2013-09-27 08:01:10 UTC
Karolin, please keep this bug open, as we need more work to fix this completely.
Comment 16 Karolin Seeger 2013-09-27 08:08:07 UTC
(In reply to comment #13)
> Comment on attachment 9243 [details]
> First Patch for v4-1-test
> 
> LGTM

Pushed to autobuild-v4-1-test.
Comment 17 Karolin Seeger 2013-09-30 10:11:42 UTC
(In reply to comment #16)
> (In reply to comment #13)
> > Comment on attachment 9243 [details] [details]
> > First Patch for v4-1-test
> > 
> > LGTM
> 
> Pushed to autobuild-v4-1-test.

Pushed to v4-1-test.
Comment 18 Karolin Seeger 2013-09-30 10:11:57 UTC
Pushed to autobuild-v4-0-test.
Comment 19 Karolin Seeger 2013-10-01 08:14:03 UTC
(In reply to comment #15)
> Karolin, please keep this bug open, as we need more work to fix this
> completely.

Metze/Andrew, freeze for 4.1.0 final will be on Friday, October 4.
Will the additional patch be available and reviewed until Friday?
If not, can I go ahead with the release anyway? Or do we need to delay again?
Comment 20 Andrew Bartlett 2013-10-03 00:36:40 UTC
At this point the proposed change is to change the DRS syntax to be binary, not string, and to have the LDAP server return the 'binary' blob, as a stop-gap to not to returning it at all.  (The conversion to 'utf8' may be lossy).
Comment 21 Andrew Bartlett 2013-10-04 03:34:50 UTC
Created attachment 9251 [details]
patch for master

This is the patch to just force the syntax.  Trying to do the 'SAMR is special' patch just ended up creating more trouble than it is worth. 

What we do however also need is tests (at least manual to get past the 4.1 release) and dbcheck patches to fix up all the corrupt databases.
Comment 22 Stefan Metzmacher 2013-10-04 08:45:27 UTC
I just tested (within TDB_NO_FSYNC=1 buildnice make -j testenv
SELFTEST_TESTENV=vampire_dc)
that a value of

userParameters::
IAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgA
 CAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAUAAFABoACA
 ABAEMAdAB4AEMAZgBnAFAAcgBlAHMAZQBuAHQANTUxZTBiYjAYAAgAAQBDAHQAeABDAGYAZwBGAGw
 AYQBnAHMAMQAwMGYwZTBmNxIACAABAEMAdAB4AFMAaABhAGQAbwB3ADAxMDAwMDAwKgACAAEAQwB0
 AHgATQBpAG4ARQBuAGMAcgB5AHAAdABpAG8AbgBMAGUAdgBlAGwAMDEgAEgAAQBDAHQAeABXAEYAU
 AByAG8AZgBpAGwAZQBQAGEAdABoADJmNzc2NTJmNjQ2ZjJmNmU2Zjc0MmY3MjY1NzA2YzY5NjM2MT
 c0NjUyZjc0Njg2OTczMmY2MTc0NzQ3MjY5NjI3NTc0NjUwMA==

0000   20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00     . . . . . . . .
0010   20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00     . . . . . . . .
0020   20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00     . . . . . . . .
0030   20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00     . . . . . . . .
0040   20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00     . . . . . . . .
0050   20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00     . . . . . . . .
0060   50 00 05 00 1A 00 08 00 01 00 43 00 74 00 78 00    P.........C.t.x.
0070   43 00 66 00 67 00 50 00 72 00 65 00 73 00 65 00    C.f.g.P.r.e.s.e.
0080   6E 00 74 00 35 35 31 65 30 62 62 30 18 00 08 00    n.t.551e0bb0....
0090   01 00 43 00 74 00 78 00 43 00 66 00 67 00 46 00    ..C.t.x.C.f.g.F.
00A0   6C 00 61 00 67 00 73 00 31 00 30 30 66 30 65 30    l.a.g.s.1.00f0e0
00B0   66 37 12 00 08 00 01 00 43 00 74 00 78 00 53 00    f7......C.t.x.S.
00C0   68 00 61 00 64 00 6F 00 77 00 30 31 30 30 30 30    h.a.d.o.w.010000
00D0   30 30 2A 00 02 00 01 00 43 00 74 00 78 00 4D 00    00*.....C.t.x.M.
00E0   69 00 6E 00 45 00 6E 00 63 00 72 00 79 00 70 00    i.n.E.n.c.r.y.p.
00F0   74 00 69 00 6F 00 6E 00 4C 00 65 00 76 00 65 00    t.i.o.n.L.e.v.e.
0100   6C 00 30 31 20 00 48 00 01 00 43 00 74 00 78 00    l.01 .H...C.t.x.
0110   57 00 46 00 50 00 72 00 6F 00 66 00 69 00 6C 00    W.F.P.r.o.f.i.l.
0120   65 00 50 00 61 00 74 00 68 00 32 66 37 37 36 35    e.P.a.t.h.2f7765
0130   32 66 36 34 36 66 32 66 36 65 36 66 37 34 32 66    2f646f2f6e6f742f
0140   37 32 36 35 37 30 36 63 36 39 36 33 36 31 37 34    7265706c69636174
0150   36 35 32 66 37 34 36 38 36 39 37 33 32 66 36 31    652f746869732f61
0160   37 34 37 34 37 32 36 39 36 32 37 35 37 34 36 35    7474726962757465
0170   30 30

Gets correctly replicated without your patch.

So I don't think there's no need for an urgent fix for 4.1.0.

We should try to create a real fix for this in master and think about
backporting it once we're happy with the final result.

=> remove this as blocker for 4.1.0
Comment 23 Karolin Seeger 2013-12-10 15:30:30 UTC
Any news on this one?
Comment 24 Stefan Metzmacher 2014-07-08 14:23:05 UTC
Please someone requires to modify 'userParameters' via LDAP as UTF8,
let leave a note here, so that we'll be aware of the need and can work on
a solution.
Comment 25 trenta 2016-07-15 08:31:04 UTC
Hi Stefan,

We are tying to migrate samba 3 to samba 4 AD (4.4.5) and we detected that with latest version of samba 4 error persist, trying to modify userParameters attribute in ldap  with

new LDAPModification(LDAPModification.REPLACE, new LDAPAttribute("userParameters", userEntry.getAttribute("userParameters").getByteValue()))

we receive this error:
LDAPException: Violación Del Constreñimiento (19) Violación Del Constreñimiento
LDAPException: Server Message: 0000202F: samldb: setting userParameters is not supported over LDAP, see https://bugzilla.samba.org/show_bug.cgi?id=8077 at ../source4/dsdb/samdb/ldb_modules/samldb.c:3104


We need to update this attribute because with samba 3 we have data and we need to migrate to keep same features.

Can you solve  for samba 4.4?

thanks
Comment 26 Andrew Bartlett 2016-07-28 23:53:26 UTC
*** Bug 11610 has been marked as a duplicate of this bug. ***
Comment 27 Andrew Bartlett 2016-07-28 23:53:58 UTC
*** Bug 11987 has been marked as a duplicate of this bug. ***
Comment 28 Andrew Bartlett 2016-07-28 23:59:30 UTC
As per bug 11881 What this needs a a test script that confirms correct behaviour over:

LDAP
SAMR
NETLOGON
passdb
DRS replciation
and then confirming that dbcheck doesn't mangle it even more

Once we have such a test script, fixing up the various parts should not be too hard.
Comment 29 Yuriy Tabolin 2016-10-06 13:29:08 UTC
I have the same problem on samba 4.2.3. Recently I have created new user and some magic way this user now has attribute userParameters 
userParameters::  IAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgA
 CAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAIAAgACAAUAAEABoACA
 ABAEMAdAB4AEMAZgBnAFAAcgBlAHMAZQBuAHQANTUxZTBiYjAYAAgAAQBDAHQAeABDAGYAZwBGAGw
 AYQBnAHMAMQAwMGYwZTBmNxIACAABAEMAdAB4AFMAaABhAGQAbwB3ADAxMDAwMDAwKgACAAEAQwB0
 AHgATQBpAG4ARQBuAGMAcgB5AHAAdABpAG8AbgBMAGUAdgBlAGwAMDE=
How appeared this attribute I don't know.

Now I can't clear this attribute or modify it. Is there any way to clear userParameters not deleting and re-create whole user?
Comment 30 Adam Tauno Williams 2016-11-29 18:53:09 UTC
Yuriy Tabolin -  I have been able to *delete* the attribute using ldbmodify, which fixed the user account and I was subsequently able to edit it via MMC etc..

[root@larkin27 ~]# cat fix.ldif 
dn: CN=adam,OU=MorrisonInd,OU=Industries Users,DC=micore,DC=us
changetype: modify
delete: userParameters

[root@larkin27 ~]# ldbmodify -H /var/lib/samba/private/sam.ldb  fix.ldif 
Modified 1 records successfully

Remember that LDIF files need to end with a single blank line.