Hello, after compiling Samba 3.5.4 on RHEL 5.5(x86_64/2.6.18-194.8.1.el5) "wbinfo -g" an "wbinfo -u" gives no output. Log Entry: Connection to A.Directory is working as expected : [root@nas05 log]# net ads testjoin Join is OK [root@nas05 log]# wbinfo -t checking the trust secret for domain GCA_CH22 via RPC calls succeeded wbinfo -i is working : [root@nas05 log]# net ads testjoin Join is OK how to get wbinfo - u an wnfinfo working ? thanks you
Created attachment 5876 [details] log files (tar format) with log level 10
I'm correcting my last sentence : How to get "wbinfo -u" and "wbinfo -g" working ?
I'm hitting the same problem on solaris 9 x86 and sparc with samba 3.5.6 (active directory on windows 2003 R2 SP2 with rfc2307 schema extension, openssl 0.9.8o, libiconv 1.13.1, heimdal 1.4, openldap 2.4.23) for both "wbinfo -g" and "wbinfo -u". wbinfo -t and net ads testjoin give positive results. The same testbed except of using samba 3.4.9 does not show the problem: "wbinfo -g" and "wbinfo -u" work as expected. Names services using nss_winbind.so are working. The ndr_pull_error line seems to be a subsequent "unable to display the error message" error. The relevant lines in log.winbindd are [2010/10/28 17:51:31.512980, 6] winbindd/winbindd.c:768(new_connection) accepted socket 23 [2010/10/28 17:51:31.513254, 3] winbindd/winbindd_lookupsid.c:51(winbindd_lookupsid_send) lookupsid S-1-5-21-XXXXXXXXXX-YYYYYYYYYY-ZZZZZZZZZZ-513 [2010/10/28 17:51:31.513468, 1] ../librpc/ndr/ndr.c:395(ndr_pull_error) ndr_pull_error(1): String terminator not present or outside string boundaries [2010/10/28 17:51:31.513536, 5] winbindd/winbindd_lookupsid.c:94(winbindd_lookupsid_recv) Could not lookup sid S-1-5-21-XXXXXXXXXX-YYYYYYYYYY-ZZZZZZZZZZ-513: NT_STATUS_ARRAY_BOUNDS_EXCEEDED [2010/10/28 17:51:31.513729, 6] winbindd/winbindd.c:816(winbind_client_request_read) closing socket 22, client exited
Stefan Metzmacher and Michael Adam are right now working on this problem. It's a deep problem with our NDR libraries and charset conversion routines. Stay tuned. Volker
I assume you're using a "unix charset" which is not "UTF-8"?
hello, Yes, we are using unix charset = ISO8859-15 [global] unix charset = ISO8859-15 display charset = UTF-8 ...
(In reply to comment #6) > hello, > > Yes, we are using unix charset = ISO8859-15 > > [global] > unix charset = ISO8859-15 > display charset = UTF-8 Oh, is the display charset set explicitly or through the locale? Because I am seeing this in the logs: "Substituting charset 'UTF-8' for LOCALE" Anyhow. While this is certainly a bug: Is there a special reason to set unix charset to a different value than display charset?
(In reply to comment #5) > I assume you're using a "unix charset" which is not "UTF-8"? > Yes we use to configure in smb.conf: unix charset = ISO8859-1 dos charset = 850 I (maybe) managed to track down the issue to the following circumstances: We have users in the AD with accented characters in the gecos RFC-2307 attribute (please note the: gecos: Großmüller\, Karlo,OU=Abt001 I observerd the following iconv conversion error in log.wb-DOMAIN: [2010/10/29 10:34:26.510806, 3] lib/charcnv.c:268(convert_string_internal) E2BIG: convert_string(ISO8859-1,UTF8): srclen=25 destlen=26 - 'ßmüller\, Karlo,OU=Abt001' [2010/10/29 10:34:26.514246, 4] winbindd/winbindd_dual.c:1525(fork_domain_child) Finished processing child request 63 (In reply to comment #5) > I assume you're using a "unix charset" which is not "UTF-8"? >
(In reply to comment #7) > (In reply to comment #6) > > hello, > > > > Yes, we are using unix charset = ISO8859-15 > > > > [global] > > unix charset = ISO8859-15 > > display charset = UTF-8 > > Oh, is the display charset set explicitly or through the locale? > Because I am seeing this in the logs: > "Substituting charset 'UTF-8' for LOCALE" > > Anyhow. While this is certainly a bug: Is there > a special reason to set unix charset to a different > value than display charset? > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8. (LANG=fr_FR.UTF-8) so, i have set "display charset" to locale value in smb.conf file. After restarting samba and winwind process, i always have the same issue on "wbinfo - u" and "wbinfo -g"
(In reply to comment #9) > (In reply to comment #7) > > (In reply to comment #6) > > > hello, > > > > > > Yes, we are using unix charset = ISO8859-15 > > > > > > [global] > > > unix charset = ISO8859-15 > > > display charset = UTF-8 > > > > Oh, is the display charset set explicitly or through the locale? > > Because I am seeing this in the logs: > > "Substituting charset 'UTF-8' for LOCALE" > > > > Anyhow. While this is certainly a bug: Is there > > a special reason to set unix charset to a different > > value than display charset? > > > > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8. > (LANG=fr_FR.UTF-8) > so, i have set "display charset" to locale value in smb.conf file. Hmmm, the question is rather: When your locale is UTF-8, why do you set the unix charset to ISO? If you would have it set to UTF-8, the error would vanish... Cheers - Michael > After restarting samba and winwind process, i always have the same issue on > "wbinfo - u" and "wbinfo -g"
(In reply to comment #10) > (In reply to comment #9) > > (In reply to comment #7) > > > (In reply to comment #6) > > > > hello, > > > > > > > > Yes, we are using unix charset = ISO8859-15 > > > > > > > > [global] > > > > unix charset = ISO8859-15 > > > > display charset = UTF-8 > > > > > > Oh, is the display charset set explicitly or through the locale? > > > Because I am seeing this in the logs: > > > "Substituting charset 'UTF-8' for LOCALE" > > > > > > Anyhow. While this is certainly a bug: Is there > > > a special reason to set unix charset to a different > > > value than display charset? > > > > > > > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8. > > (LANG=fr_FR.UTF-8) > > so, i have set "display charset" to locale value in smb.conf file. > > Hmmm, the question is rather: > When your locale is UTF-8, why do you set the unix charset to ISO? > If you would have it set to UTF-8, the error would vanish... > > Cheers - Michael > > > After restarting samba and winwind process, i always have the same issue on > > "wbinfo - u" and "wbinfo -g" > With unix charset set to ISO8859-15, accentued characters (éééèèùùàà...) are correctly displayed in "windows explorer" but in this case, Wbinfo -u and -g are not functional. If unixcharset set to UTF-8, accentued characters are not correctly displayed (eg: "é"=>"_") but wbinfo becomes fonctional... this is strange How to get the proper display of accentued characters in Windows Explorer and a command wbinfo functional ?
(In reply to comment #11) > (In reply to comment #10) > > (In reply to comment #9) > > > (In reply to comment #7) > > > > (In reply to comment #6) > > > > > hello, > > > > > > > > > > Yes, we are using unix charset = ISO8859-15 > > > > > > > > > > [global] > > > > > unix charset = ISO8859-15 > > > > > display charset = UTF-8 > > > > > > > > Oh, is the display charset set explicitly or through the locale? > > > > Because I am seeing this in the logs: > > > > "Substituting charset 'UTF-8' for LOCALE" > > > > > > > > Anyhow. While this is certainly a bug: Is there > > > > a special reason to set unix charset to a different > > > > value than display charset? > > > > > > > > > > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8. > > > (LANG=fr_FR.UTF-8) > > > so, i have set "display charset" to locale value in smb.conf file. > > > > Hmmm, the question is rather: > > When your locale is UTF-8, why do you set the unix charset to ISO? > > If you would have it set to UTF-8, the error would vanish... > > > > Cheers - Michael > > > > > After restarting samba and winwind process, i always have the same issue on > > > "wbinfo - u" and "wbinfo -g" > > > With unix charset set to ISO8859-15, accentued characters > (éééèèùùàà...) are correctly displayed in "windows explorer" but in > this case, Wbinfo -u and -g are not functional. > > If unixcharset set to UTF-8, accentued characters are not correctly displayed > (eg: "é"=>"_") but wbinfo becomes fonctional... this is strange > > How to get the proper display of accentued characters in Windows Explorer and a > command wbinfo functional ? It is not so strange. The problem is this: 1. Even though your locale is set to UTF-8, you seem to have file(names) with ISO encoding in the directories shared by samba. (Possibly because they were copied from an older installation, or because you just had this unix charset setting from the beginning.) This is why the explorer shows these chars correctly only when the unix charset is set to ISO: This is the encoding used (among other things) for unix file system interaction. When you use an encoding that is different from the encoding on disk, you get broken file names. 2. The "unix charset" is used internally in Samba for many things. For instance also the User and Group names retrieved from windows are also converted to the unix charset. This should work, but there is a bug in the code that is used (among other places) for winbindd parent-child-communication in samba >= 3.5.0. This bug makes the conversions fail when the unix charset is different from UTF-8. This is the bug that Volker mentionend above Metze and I are currently hunting. We need to use one of the recorded bugs as the only one and make the other instances duplicates... So. What you could do is this: You could convert your filenames on disk to UTF-8 encoding using the tool convmv and then change your unix charset to "UTF-8". This should work for now. Cheers - Michael Cheers - Michael
(In reply to comment #12) > (In reply to comment #11) > > (In reply to comment #10) > > > (In reply to comment #9) > > > > (In reply to comment #7) > > > > > (In reply to comment #6) > > > > > > hello, > > > > > > > > > > > > Yes, we are using unix charset = ISO8859-15 > > > > > > > > > > > > [global] > > > > > > unix charset = ISO8859-15 > > > > > > display charset = UTF-8 > > > > > > > > > > Oh, is the display charset set explicitly or through the locale? > > > > > Because I am seeing this in the logs: > > > > > "Substituting charset 'UTF-8' for LOCALE" > > > > > > > > > > Anyhow. While this is certainly a bug: Is there > > > > > a special reason to set unix charset to a different > > > > > value than display charset? > > > > > > > > > > > > > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8. > > > > (LANG=fr_FR.UTF-8) > > > > so, i have set "display charset" to locale value in smb.conf file. > > > > > > Hmmm, the question is rather: > > > When your locale is UTF-8, why do you set the unix charset to ISO? > > > If you would have it set to UTF-8, the error would vanish... > > > > > > Cheers - Michael > > > > > > > After restarting samba and winwind process, i always have the same issue on > > > > "wbinfo - u" and "wbinfo -g" > > > > > With unix charset set to ISO8859-15, accentued characters > > (éééèèùùàà...) are correctly displayed in "windows explorer" but in > > this case, Wbinfo -u and -g are not functional. > > > > If unixcharset set to UTF-8, accentued characters are not correctly displayed > > (eg: "é"=>"_") but wbinfo becomes fonctional... this is strange > > > > How to get the proper display of accentued characters in Windows Explorer and a > > command wbinfo functional ? > > It is not so strange. The problem is this: > > 1. Even though your locale is set to UTF-8, you seem to have > file(names) with ISO encoding in the directories shared > by samba. (Possibly because they were copied from an older > installation, or because you just had this unix charset > setting from the beginning.) > > This is why the explorer shows these chars correctly only > when the unix charset is set to ISO: This is the encoding > used (among other things) for unix file system interaction. > > When you use an encoding that is different from the encoding > on disk, you get broken file names. > > 2. The "unix charset" is used internally in Samba for many > things. For instance also the User and Group names retrieved > from windows are also converted to the unix charset. > > This should work, but there is a bug in the code that is used > (among other places) for winbindd parent-child-communication > in samba >= 3.5.0. This bug makes the conversions fail > when the unix charset is different from UTF-8. > > This is the bug that Volker mentionend above Metze and I are > currently hunting. We need to use one of the recorded bugs > as the only one and make the other instances duplicates... > > So. What you could do is this: > You could convert your filenames on disk to > UTF-8 encoding using the tool convmv and then > change your unix charset to "UTF-8". > This should work for now. > > Cheers - Michael > Cheers - Michael > (In reply to comment #12) > (In reply to comment #11) > > (In reply to comment #10) > > > (In reply to comment #9) > > > > (In reply to comment #7) > > > > > (In reply to comment #6) > > > > > > hello, > > > > > > > > > > > > Yes, we are using unix charset = ISO8859-15 > > > > > > > > > > > > [global] > > > > > > unix charset = ISO8859-15 > > > > > > display charset = UTF-8 > > > > > > > > > > Oh, is the display charset set explicitly or through the locale? > > > > > Because I am seeing this in the logs: > > > > > "Substituting charset 'UTF-8' for LOCALE" > > > > > > > > > > Anyhow. While this is certainly a bug: Is there > > > > > a special reason to set unix charset to a different > > > > > value than display charset? > > > > > > > > > > > > > I agree, there is no special reason to set DISPLAY CHARSET to UTF-8. > > > > (LANG=fr_FR.UTF-8) > > > > so, i have set "display charset" to locale value in smb.conf file. > > > > > > Hmmm, the question is rather: > > > When your locale is UTF-8, why do you set the unix charset to ISO? > > > If you would have it set to UTF-8, the error would vanish... > > > > > > Cheers - Michael > > > > > > > After restarting samba and winwind process, i always have the same issue on > > > > "wbinfo - u" and "wbinfo -g" > > > > > With unix charset set to ISO8859-15, accentued characters > > (éééèèùùàà...) are correctly displayed in "windows explorer" but in > > this case, Wbinfo -u and -g are not functional. > > > > If unixcharset set to UTF-8, accentued characters are not correctly displayed > > (eg: "é"=>"_") but wbinfo becomes fonctional... this is strange > > > > How to get the proper display of accentued characters in Windows Explorer and a > > command wbinfo functional ? > > It is not so strange. The problem is this: > > 1. Even though your locale is set to UTF-8, you seem to have > file(names) with ISO encoding in the directories shared > by samba. (Possibly because they were copied from an older > installation, or because you just had this unix charset > setting from the beginning.) > > This is why the explorer shows these chars correctly only > when the unix charset is set to ISO: This is the encoding > used (among other things) for unix file system interaction. > > When you use an encoding that is different from the encoding > on disk, you get broken file names. > > 2. The "unix charset" is used internally in Samba for many > things. For instance also the User and Group names retrieved > from windows are also converted to the unix charset. > > This should work, but there is a bug in the code that is used > (among other places) for winbindd parent-child-communication > in samba >= 3.5.0. This bug makes the conversions fail > when the unix charset is different from UTF-8. > > This is the bug that Volker mentionend above Metze and I are > currently hunting. We need to use one of the recorded bugs > as the only one and make the other instances duplicates... > > So. What you could do is this: > You could convert your filenames on disk to > UTF-8 encoding using the tool convmv and then > change your unix charset to "UTF-8". > This should work for now. > > Cheers - Michael > Cheers - Michael > Before noon, we found and tested the procedure perl "convmv" that converts the character file names. convmv -f iso-8859-15 -t utf-8 - notest-r /nas it takes 23 minutes for 100,000 files on an HP server DL360g6 E5540. So we will convert 4.5 million files before upgrading to version 3.5.6 Thank you
*** Bug 7560 has been marked as a duplicate of this bug. ***
Before anyone starts convmv-ing millions of files: I have by now a patch (against master) that fixes the problem for me. Needs some samba4-testing and backporting still. I will add the patch here for review soon. Stay tuned - Michael
Created attachment 6041 [details] Proposed patchset for fixing the bug. This patchset (to be applied to current git v3-5-test with "git am") fixes the bug for me. This is ported back from a patchset to master. Please test! I have not yet pushed the master patchset since it also touches samba4 code (since the librpc/ndr/ndr_string.c code with the ndr_charset_length() function that has been fixed in this patchset has been merged to the code shared between samba3 and samba4. So I am still awaiting review for that. The master patchset can be take from my master3-charset branch at git://git.samba.org/obnox/samba/samba-obnox.git Cheers - Michael
Created attachment 6042 [details] Updated patchset Here is an updated more minmal patchset. It adds less code and does less reformatting. (The next_codepoint_ext() function is not needed, just a version of strlen_m_ext is introdcuced that accepts the destination charset...) The master patchset has also been updated (cleaned...) but there there more general version of strlen_m_ext is preserved. Cheers - Michael
Comment on attachment 6042 [details] Updated patchset Looks good, for v3-5
Michael, can you push your changes to master and v3-6-test as well please ! Thanks, Jeremy.
(In reply to comment #19) > Michael, can you push your changes to master and v3-6-test as well please ! > Thanks, > Jeremy. Jeremy, yes, I surely want to do that but: * The master changes are slightly more general - adding source and desitnation charset parameters to strlen_m_ext() - by adding a source charset to next_codepoint() * Since the librpc/ndr/ndr_string.c is now common code (s3+s4), I have to add a s4 variant of strlen_m_ext() with the same signature as the s3 one. The pathes are in my master3-charset branch. I compiled and tested them and it looks fine to me, but I would like to get sign-off by Metze or some other s4 guru before pushing. Cheers - Michael
(In reply to comment #19) > Michael, can you push your changes to master and v3-6-test as well please ! > Thanks, > Jeremy. Ok, done.
Assigning Bug to Karolin. Please add the patch to 3.5 Cheers - Michael
Pushed to v3-5-test. Closing out bug report. Thanks!
*** Bug 7796 has been marked as a duplicate of this bug. ***
*** Bug 7819 has been marked as a duplicate of this bug. ***