Bug 7594 - "wbinfo -u" and "wbinfo -g" gives no output (log=>ndr_pull_error)
"wbinfo -u" and "wbinfo -g" gives no output (log=>ndr_pull_error)
Status: RESOLVED FIXED
Product: Samba 3.5
Classification: Unclassified
Component: Winbind
3.5.4
x64 Linux
: P3 major
: ---
Assigned To: Karolin Seeger
Samba QA Contact
:
: 7560 7796 7819 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-29 06:02 UTC by chstbrieuc22
Modified: 2010-12-08 01:34 UTC (History)
6 users (show)

See Also:


Attachments
log files (tar format) with log level 10 (281.88 KB, application/x-tar)
2010-07-29 06:08 UTC, chstbrieuc22
no flags Details
Proposed patchset for fixing the bug. (15.58 KB, patch)
2010-11-01 09:07 UTC, Michael Adam
no flags Details
Updated patchset (6.85 KB, patch)
2010-11-01 10:43 UTC, Michael Adam
metze: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description chstbrieuc22 2010-07-29 06:02:40 UTC
Hello,

after compiling  Samba  3.5.4 on RHEL 5.5(x86_64/2.6.18-194.8.1.el5) 
 "wbinfo -g" an "wbinfo -u" gives no output.

Log Entry:


Connection to A.Directory is working as expected :

[root@nas05 log]# net ads testjoin
Join is OK

[root@nas05 log]# wbinfo -t
checking the trust secret for domain GCA_CH22 via RPC calls succeeded

wbinfo -i is working :
[root@nas05 log]# net ads testjoin
Join is OK


how to get wbinfo - u  an wnfinfo working ?

thanks you
Comment 1 chstbrieuc22 2010-07-29 06:08:56 UTC
Created attachment 5876 [details]
log files (tar format) with log level 10
Comment 2 chstbrieuc22 2010-07-29 06:24:08 UTC
I'm correcting my last sentence :

How to get "wbinfo -u"  and "wbinfo -g" working ?



Comment 3 Dietrich Streifert 2010-10-29 01:16:33 UTC
I'm hitting the same problem on solaris 9 x86 and sparc with samba 3.5.6 (active directory on windows 2003 R2 SP2 with rfc2307 schema extension, openssl 0.9.8o, libiconv 1.13.1, heimdal 1.4, openldap 2.4.23) for both "wbinfo -g" and "wbinfo -u".

wbinfo -t and net ads testjoin give positive results.

The same testbed except of using samba 3.4.9 does not show the problem: "wbinfo -g" and "wbinfo -u" work as expected. Names services using nss_winbind.so are working.

The ndr_pull_error line seems to be a subsequent "unable to display the error message" error.

The relevant lines in log.winbindd are

[2010/10/28 17:51:31.512980,  6] winbindd/winbindd.c:768(new_connection)
  accepted socket 23
[2010/10/28 17:51:31.513254,  3] winbindd/winbindd_lookupsid.c:51(winbindd_lookupsid_send)
  lookupsid S-1-5-21-XXXXXXXXXX-YYYYYYYYYY-ZZZZZZZZZZ-513
[2010/10/28 17:51:31.513468,  1] ../librpc/ndr/ndr.c:395(ndr_pull_error)
  ndr_pull_error(1): String terminator not present or outside string boundaries
[2010/10/28 17:51:31.513536,  5] winbindd/winbindd_lookupsid.c:94(winbindd_lookupsid_recv)
  Could not lookup sid S-1-5-21-XXXXXXXXXX-YYYYYYYYYY-ZZZZZZZZZZ-513: NT_STATUS_ARRAY_BOUNDS_EXCEEDED
[2010/10/28 17:51:31.513729,  6] winbindd/winbindd.c:816(winbind_client_request_read)
  closing socket 22, client exited

Comment 4 Volker Lendecke 2010-10-29 01:18:20 UTC
Stefan Metzmacher and Michael Adam are right now working on this problem. It's a deep problem with our NDR libraries and charset conversion routines.

Stay tuned.

Volker
Comment 5 Stefan Metzmacher 2010-10-29 03:20:07 UTC
I assume you're using a "unix charset" which is not "UTF-8"?
Comment 6 chstbrieuc22 2010-10-29 03:36:31 UTC
hello,

Yes, we are using unix charset = ISO8859-15

[global]
        unix charset = ISO8859-15
        display charset = UTF-8
...
Comment 7 Michael Adam 2010-10-29 03:47:21 UTC
(In reply to comment #6)
> hello,
> 
> Yes, we are using unix charset = ISO8859-15
> 
> [global]
>         unix charset = ISO8859-15
>         display charset = UTF-8

Oh, is the display charset set explicitly or through the locale?
Because I am seeing this in the logs:
"Substituting charset 'UTF-8' for LOCALE"

Anyhow. While this is certainly a bug: Is there
a special reason to set unix charset to a different
value than display charset?
Comment 8 Dietrich Streifert 2010-10-29 03:57:45 UTC
(In reply to comment #5)
> I assume you're using a "unix charset" which is not "UTF-8"?
> 
Yes we use to configure in smb.conf:

        unix charset = ISO8859-1
        dos charset = 850

I (maybe) managed to track down the issue to the following circumstances:

We have users in the AD with accented characters in the gecos RFC-2307 attribute (please note the:

  gecos: Großmüller\, Karlo,OU=Abt001

I observerd the following iconv conversion error in log.wb-DOMAIN:

[2010/10/29 10:34:26.510806,  3] lib/charcnv.c:268(convert_string_internal)
  E2BIG: convert_string(ISO8859-1,UTF8): srclen=25 destlen=26 - 'ßmüller\, Karlo,OU=Abt001'
[2010/10/29 10:34:26.514246,  4] winbindd/winbindd_dual.c:1525(fork_domain_child)
  Finished processing child request 63



(In reply to comment #5)
> I assume you're using a "unix charset" which is not "UTF-8"?
> 

Comment 9 chstbrieuc22 2010-10-29 04:25:14 UTC
(In reply to comment #7)
> (In reply to comment #6)
> > hello,
> > 
> > Yes, we are using unix charset = ISO8859-15
> > 
> > [global]
> >         unix charset = ISO8859-15
> >         display charset = UTF-8
> 
> Oh, is the display charset set explicitly or through the locale?
> Because I am seeing this in the logs:
> "Substituting charset 'UTF-8' for LOCALE"
> 
> Anyhow. While this is certainly a bug: Is there
> a special reason to set unix charset to a different
> value than display charset?
> 

I agree, there is no special reason to set DISPLAY CHARSET to UTF-8.
(LANG=fr_FR.UTF-8)
so, i have set "display charset" to locale value in smb.conf file.

After restarting samba and winwind process, i always have the same issue on "wbinfo - u" and "wbinfo -g" 
Comment 10 Michael Adam 2010-10-29 04:34:37 UTC
(In reply to comment #9)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > hello,
> > > 
> > > Yes, we are using unix charset = ISO8859-15
> > > 
> > > [global]
> > >         unix charset = ISO8859-15
> > >         display charset = UTF-8
> > 
> > Oh, is the display charset set explicitly or through the locale?
> > Because I am seeing this in the logs:
> > "Substituting charset 'UTF-8' for LOCALE"
> > 
> > Anyhow. While this is certainly a bug: Is there
> > a special reason to set unix charset to a different
> > value than display charset?
> > 
> 
> I agree, there is no special reason to set DISPLAY CHARSET to UTF-8.
> (LANG=fr_FR.UTF-8)
> so, i have set "display charset" to locale value in smb.conf file.

Hmmm, the question is rather:
When your locale is UTF-8, why do you set the unix charset to ISO?
If you would have it set to UTF-8, the error would vanish...

Cheers - Michael

> After restarting samba and winwind process, i always have the same issue on
> "wbinfo - u" and "wbinfo -g" 


Comment 11 chstbrieuc22 2010-10-29 04:56:53 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #7)
> > > (In reply to comment #6)
> > > > hello,
> > > > 
> > > > Yes, we are using unix charset = ISO8859-15
> > > > 
> > > > [global]
> > > >         unix charset = ISO8859-15
> > > >         display charset = UTF-8
> > > 
> > > Oh, is the display charset set explicitly or through the locale?
> > > Because I am seeing this in the logs:
> > > "Substituting charset 'UTF-8' for LOCALE"
> > > 
> > > Anyhow. While this is certainly a bug: Is there
> > > a special reason to set unix charset to a different
> > > value than display charset?
> > > 
> > 
> > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8.
> > (LANG=fr_FR.UTF-8)
> > so, i have set "display charset" to locale value in smb.conf file.
> 
> Hmmm, the question is rather:
> When your locale is UTF-8, why do you set the unix charset to ISO?
> If you would have it set to UTF-8, the error would vanish...
> 
> Cheers - Michael
> 
> > After restarting samba and winwind process, i always have the same issue on
> > "wbinfo - u" and "wbinfo -g" 
> 
With  unix charset set to ISO8859-15, accentued characters (éééèèùùàà...) are correctly displayed in "windows explorer" but in this case,  Wbinfo -u and  -g are not functional.

If unixcharset set to  UTF-8, accentued characters are  not correctly displayed (eg: "é"=>"_") but wbinfo becomes fonctional... this is strange

How to get the proper display of accentued characters in Windows Explorer and a command wbinfo functional ?



Comment 12 Michael Adam 2010-10-29 06:13:09 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > (In reply to comment #9)
> > > (In reply to comment #7)
> > > > (In reply to comment #6)
> > > > > hello,
> > > > > 
> > > > > Yes, we are using unix charset = ISO8859-15
> > > > > 
> > > > > [global]
> > > > >         unix charset = ISO8859-15
> > > > >         display charset = UTF-8
> > > > 
> > > > Oh, is the display charset set explicitly or through the locale?
> > > > Because I am seeing this in the logs:
> > > > "Substituting charset 'UTF-8' for LOCALE"
> > > > 
> > > > Anyhow. While this is certainly a bug: Is there
> > > > a special reason to set unix charset to a different
> > > > value than display charset?
> > > > 
> > > 
> > > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8.
> > > (LANG=fr_FR.UTF-8)
> > > so, i have set "display charset" to locale value in smb.conf file.
> > 
> > Hmmm, the question is rather:
> > When your locale is UTF-8, why do you set the unix charset to ISO?
> > If you would have it set to UTF-8, the error would vanish...
> > 
> > Cheers - Michael
> > 
> > > After restarting samba and winwind process, i always have the same issue on
> > > "wbinfo - u" and "wbinfo -g" 
> > 
> With  unix charset set to ISO8859-15, accentued characters
> (éééèèùùàà...) are correctly displayed in "windows explorer" but in
> this case,  Wbinfo -u and  -g are not functional.
> 
> If unixcharset set to  UTF-8, accentued characters are  not correctly displayed
> (eg: "é"=>"_") but wbinfo becomes fonctional... this is strange
> 
> How to get the proper display of accentued characters in Windows Explorer and a
> command wbinfo functional ?

It is not so strange. The problem is this:

1. Even though your locale is set to UTF-8, you seem to have
   file(names) with ISO encoding in the directories shared
   by samba. (Possibly because they were copied from an older
   installation, or because you just had this unix charset
   setting from the beginning.)

   This is why the explorer shows these chars correctly only
   when the unix charset is set to ISO: This is the encoding
   used (among other things) for unix file system interaction.

   When you use an encoding that is different from the encoding
   on disk, you get broken file names.

2. The "unix charset" is used internally in Samba for many
   things. For instance also the User and Group names retrieved
   from windows are also converted to the unix charset.

   This should work, but there is a bug in the code that is used
   (among other places) for winbindd parent-child-communication
   in samba >= 3.5.0. This bug makes the conversions fail
   when the unix charset is different from UTF-8.

   This is the bug that Volker mentionend above Metze and I are
   currently hunting. We need to use one of the recorded bugs
   as the only one and make the other instances duplicates...

So. What you could do is this:
You could convert your filenames on disk to
UTF-8 encoding using the tool convmv  and then
change your unix charset to "UTF-8".
This should work for now.

Cheers - Michael
Cheers - Michael
Comment 13 chstbrieuc22 2010-10-29 07:29:32 UTC
(In reply to comment #12)
> (In reply to comment #11)
> > (In reply to comment #10)
> > > (In reply to comment #9)
> > > > (In reply to comment #7)
> > > > > (In reply to comment #6)
> > > > > > hello,
> > > > > > 
> > > > > > Yes, we are using unix charset = ISO8859-15
> > > > > > 
> > > > > > [global]
> > > > > >         unix charset = ISO8859-15
> > > > > >         display charset = UTF-8
> > > > > 
> > > > > Oh, is the display charset set explicitly or through the locale?
> > > > > Because I am seeing this in the logs:
> > > > > "Substituting charset 'UTF-8' for LOCALE"
> > > > > 
> > > > > Anyhow. While this is certainly a bug: Is there
> > > > > a special reason to set unix charset to a different
> > > > > value than display charset?
> > > > > 
> > > > 
> > > > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8.
> > > > (LANG=fr_FR.UTF-8)
> > > > so, i have set "display charset" to locale value in smb.conf file.
> > > 
> > > Hmmm, the question is rather:
> > > When your locale is UTF-8, why do you set the unix charset to ISO?
> > > If you would have it set to UTF-8, the error would vanish...
> > > 
> > > Cheers - Michael
> > > 
> > > > After restarting samba and winwind process, i always have the same issue on
> > > > "wbinfo - u" and "wbinfo -g" 
> > > 
> > With  unix charset set to ISO8859-15, accentued characters
> > (éééèèùùàà...) are correctly displayed in "windows explorer" but in
> > this case,  Wbinfo -u and  -g are not functional.
> > 
> > If unixcharset set to  UTF-8, accentued characters are  not correctly displayed
> > (eg: "é"=>"_") but wbinfo becomes fonctional... this is strange
> > 
> > How to get the proper display of accentued characters in Windows Explorer and a
> > command wbinfo functional ?
> 
> It is not so strange. The problem is this:
> 
> 1. Even though your locale is set to UTF-8, you seem to have
>    file(names) with ISO encoding in the directories shared
>    by samba. (Possibly because they were copied from an older
>    installation, or because you just had this unix charset
>    setting from the beginning.)
> 
>    This is why the explorer shows these chars correctly only
>    when the unix charset is set to ISO: This is the encoding
>    used (among other things) for unix file system interaction.
> 
>    When you use an encoding that is different from the encoding
>    on disk, you get broken file names.
> 
> 2. The "unix charset" is used internally in Samba for many
>    things. For instance also the User and Group names retrieved
>    from windows are also converted to the unix charset.
> 
>    This should work, but there is a bug in the code that is used
>    (among other places) for winbindd parent-child-communication
>    in samba >= 3.5.0. This bug makes the conversions fail
>    when the unix charset is different from UTF-8.
> 
>    This is the bug that Volker mentionend above Metze and I are
>    currently hunting. We need to use one of the recorded bugs
>    as the only one and make the other instances duplicates...
> 
> So. What you could do is this:
> You could convert your filenames on disk to
> UTF-8 encoding using the tool convmv  and then
> change your unix charset to "UTF-8".
> This should work for now.
> 
> Cheers - Michael
> Cheers - Michael
> 

(In reply to comment #12)
> (In reply to comment #11)
> > (In reply to comment #10)
> > > (In reply to comment #9)
> > > > (In reply to comment #7)
> > > > > (In reply to comment #6)
> > > > > > hello,
> > > > > > 
> > > > > > Yes, we are using unix charset = ISO8859-15
> > > > > > 
> > > > > > [global]
> > > > > >         unix charset = ISO8859-15
> > > > > >         display charset = UTF-8
> > > > > 
> > > > > Oh, is the display charset set explicitly or through the locale?
> > > > > Because I am seeing this in the logs:
> > > > > "Substituting charset 'UTF-8' for LOCALE"
> > > > > 
> > > > > Anyhow. While this is certainly a bug: Is there
> > > > > a special reason to set unix charset to a different
> > > > > value than display charset?
> > > > > 
> > > > 
> > > > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8.
> > > > (LANG=fr_FR.UTF-8)
> > > > so, i have set "display charset" to locale value in smb.conf file.
> > > 
> > > Hmmm, the question is rather:
> > > When your locale is UTF-8, why do you set the unix charset to ISO?
> > > If you would have it set to UTF-8, the error would vanish...
> > > 
> > > Cheers - Michael
> > > 
> > > > After restarting samba and winwind process, i always have the same issue on
> > > > "wbinfo - u" and "wbinfo -g" 
> > > 
> > With  unix charset set to ISO8859-15, accentued characters
> > (éééèèùùàà...) are correctly displayed in "windows explorer" but in
> > this case,  Wbinfo -u and  -g are not functional.
> > 
> > If unixcharset set to  UTF-8, accentued characters are  not correctly displayed
> > (eg: "é"=>"_") but wbinfo becomes fonctional... this is strange
> > 
> > How to get the proper display of accentued characters in Windows Explorer and a
> > command wbinfo functional ?
> 
> It is not so strange. The problem is this:
> 
> 1. Even though your locale is set to UTF-8, you seem to have
>    file(names) with ISO encoding in the directories shared
>    by samba. (Possibly because they were copied from an older
>    installation, or because you just had this unix charset
>    setting from the beginning.)
> 
>    This is why the explorer shows these chars correctly only
>    when the unix charset is set to ISO: This is the encoding
>    used (among other things) for unix file system interaction.
> 
>    When you use an encoding that is different from the encoding
>    on disk, you get broken file names.
> 
> 2. The "unix charset" is used internally in Samba for many
>    things. For instance also the User and Group names retrieved
>    from windows are also converted to the unix charset.
> 
>    This should work, but there is a bug in the code that is used
>    (among other places) for winbindd parent-child-communication
>    in samba >= 3.5.0. This bug makes the conversions fail
>    when the unix charset is different from UTF-8.
> 
>    This is the bug that Volker mentionend above Metze and I are
>    currently hunting. We need to use one of the recorded bugs
>    as the only one and make the other instances duplicates...
> 
> So. What you could do is this:
> You could convert your filenames on disk to
> UTF-8 encoding using the tool convmv  and then
> change your unix charset to "UTF-8".
> This should work for now.
> 
> Cheers - Michael
> Cheers - Michael
> 
Before noon, we found and tested the procedure perl "convmv" that converts the character file names.
convmv -f iso-8859-15 -t utf-8 - notest-r /nas
it takes 23 minutes for 100,000 files on an HP server DL360g6 E5540.
So we will convert 4.5 million files before upgrading to version 3.5.6
Thank you
Comment 14 Michael Adam 2010-11-01 06:25:18 UTC
*** Bug 7560 has been marked as a duplicate of this bug. ***
Comment 15 Michael Adam 2010-11-01 06:27:45 UTC
Before anyone starts convmv-ing millions of files:

I have by now a patch (against master) that fixes the problem for me.
Needs some samba4-testing and backporting still.
I will add the patch here for review soon.

Stay tuned - Michael
Comment 16 Michael Adam 2010-11-01 09:07:43 UTC
Created attachment 6041 [details]
Proposed patchset for fixing the bug.

This patchset (to be applied to current git v3-5-test with "git am") fixes the bug for me. This is ported back from a patchset to master.

Please test!

I have not yet pushed the master patchset since it also touches samba4 code (since the librpc/ndr/ndr_string.c code with the ndr_charset_length() function that has been fixed in this patchset has been merged to the code shared between samba3 and samba4. So I am still awaiting review for that.

The master patchset can be take from my master3-charset branch at git://git.samba.org/obnox/samba/samba-obnox.git

Cheers - Michael
Comment 17 Michael Adam 2010-11-01 10:43:39 UTC
Created attachment 6042 [details]
Updated patchset

Here is an updated more minmal patchset.
It adds less code and does less reformatting.
(The next_codepoint_ext() function is not needed,
just a version of strlen_m_ext is introdcuced that
accepts the destination charset...)

The master patchset has also been updated (cleaned...)
but there there more general version of strlen_m_ext is
preserved.

Cheers - Michael
Comment 18 Stefan Metzmacher 2010-11-02 03:59:14 UTC
Comment on attachment 6042 [details]
Updated patchset

Looks good, for v3-5
Comment 19 Jeremy Allison 2010-11-02 13:05:30 UTC
Michael, can you push your changes to master and v3-6-test as well please !
Thanks,
Jeremy.
Comment 20 Michael Adam 2010-11-02 21:58:33 UTC
(In reply to comment #19)
> Michael, can you push your changes to master and v3-6-test as well please !
> Thanks,
> Jeremy.

Jeremy,

yes, I surely want to do that but:

* The master changes are slightly more general
  - adding source and desitnation charset parameters to 
    strlen_m_ext()
  - by adding a source charset to next_codepoint()
* Since the librpc/ndr/ndr_string.c is now common code (s3+s4),
  I have to add a s4 variant of strlen_m_ext() with the same
  signature as the s3 one.

The pathes are in my master3-charset branch.
I compiled and tested them and it looks fine
to me, but I would like to get sign-off by 
Metze or some other s4 guru before pushing.

Cheers - Michael
Comment 21 Michael Adam 2010-11-04 10:54:22 UTC
(In reply to comment #19)
> Michael, can you push your changes to master and v3-6-test as well please !
> Thanks,
> Jeremy.

Ok, done.
Comment 22 Michael Adam 2010-11-04 10:55:06 UTC
Assigning Bug to Karolin.
Please add the patch to 3.5

Cheers - Michael
Comment 23 Karolin Seeger 2010-11-11 04:58:21 UTC
Pushed to v3-5-test.
Closing out bug report.

Thanks!
Comment 24 rem 2010-11-17 03:13:57 UTC
*** Bug 7796 has been marked as a duplicate of this bug. ***
Comment 25 Michael Adam 2010-12-07 08:59:18 UTC
*** Bug 7819 has been marked as a duplicate of this bug. ***