Created attachment 12000 [details] How Nautilus shows local incorrectly encoded file names I use samba to share files from a server running Ubuntu 14.04 LTS x64. The version is reported as 4.1.6+dfsg. There are public shares and home shares. They are accessed from multiple devices and computers including WinXP virtual machines, Win 7 64 bit and 32 bit PCs, a laptop running Ubuntu 14.04 IA32, a Samsung smart TV, an OUYA console and other Android devices, an iPad... When at last one entry in a directory has an invalid unicode encoding, some entries can be missing when the directory listing is obtained using smbclient. I use smbclient version 4.1.6+dfsg which is the most up to date version released for Ubuntu 14.04 LTS. How to reproduce: A file with an incorrect encoding can be quite easily created using Emacs. 1-Create a normal file with 'touch', putting Unicode characters in it. 2-Open Emacs and in "Options/MULE/Set Coding Systems/For File Name" choose "raw". 3-Using Emacs dired mode in "Immediate" menu choose "Edit file name". 4-Emacs should display the unicode sequence with octal escapes \350\... 5-Move the cursor to the octal escape and delete the first octal character 6-Save the buffer to rename the file. 7-ls the folder with bash: the incorrectly encoded file name should contain question marks now. 8-On the system where the file resides, browse to the folder and check that Nautilus displays the file with question marks in black diamonds replacing the bad characters and "(invalid encoding)" appended. Note that "(invalid encoding)" is localized and may differ on your system. 9-Using Nautilus on another system, navigate to the same directory exported as a samba share: the directory now shows up empty. 10-Using smbclient from the same or another system, connect to the share, type in the password if needed, navigate to the correct folder and type "dir": the file does not show up. There is also an error message saying: cli_list: Error: unable to parse name from info level 260 11-Now create a few other files in the same folder: give them simple names. They are not listed either! 11-Create many files: I used a for loop to create files a a2 a3 a4 a5 a6 b b2 ... After creating 156 normal files, smbclient and nautilus list only "s5". 12-Delete s5: no file is listed. Create 26 new files a7 to z7, k7 is listed. In many cases it is now very obvious that files and folders are missing. In addition if the commande "dir a*" is given to smbclient, the answer is correct: it lists a a2 a3 a4 a5 a6 a7 (in another order). My bad file name starts with 'b', so I tried "dir b*". I got the error message: cli_list: Error: unable to parse name from info level 260 NT_STATUS_NO_MEMORY listing \Test\b* The bug severity is major because it affects silently filesystem copies and some backup systems. Some users might be unable to recover their operations from backup disks if whole files and directories are missing. Part of the blame is on Ubuntu's Nautilus, which unlike smbclient, does not show any error message. But the bug is the incorrect handling of badly encoded file names. I guess that Samba must decode and recode file names to adapt Unix names to the more restrictive rules of Windows. One bad name should not trigger the disappearance of tens of other files.
ouch - this is still an issue and a really bad one; just tested with current 4.15.7, easy to reproduce: # touch a b c d (in an empty directory) # touch `echo ü | iconv -f utf8 -t latin1` -> dicrectory listing via smb: a b d (c and mis-encoded ü missing) another test in another empty directory: touch a b c d e f g h i j k l m n o p q r s t u v w x y z touch `echo Äpfel | iconv -f utf8 -t latin1` -> directory listing via smb: f b (yes, all the others are not listed!)
The server is returning bad UCS2 data here, which breaks the protocol. Can you test what a Windows client lists if the server returns a badly formatted name like this ? Does it list the directory minus the bad name or stop listing ?
I should have mentioned. I tested with current Windows 10, it lists the same files as smbclient in the tested cases.
So we're behaving in the same way as Windows does, yes ?
smbclient and windows as client behave the same in this situation, yes, but the client isn't the problem. smbd with wrong-encoded files is causing the problem. Shouldn't smbd send files, that are illegally encoded (e.g. "unix charset" to UCS2 conversion failed) just return as mangled names?
Hmmm. Anything that prevents the use of mangled names is good to me :-). What does the DEBUG level 10 log say ? The code in smbd does this: 1514 status = srvstr_push(base_data, flags2, p, 1515 fname, PTR_DIFF(end_data, p), 1516 STR_TERMINATE_ASCII, &len); 1517 if (!NT_STATUS_IS_OK(status)) { 1518 return status; 1519 } so should be failing the push of the bad string into the buffer returned to the client. I guess I'm not sure where the error is on this one. The report states: cli_list: Error: unable to parse name from info level 260 NT_STATUS_NO_MEMORY listing \Test\b* which implies the server is sending back a bad UCS2 name. But the code in the server implies that smbd_marshall_dir_entry() looks like it should fail when encountering a bad name. Having messed up encoding for filename on the server isn't something we should be encouraging, so IMHO we could just close this as "wontfix". II really don't want to make the server have to cope with this by mangling the names. The mangled names code really should eventually go away by mapping into the unicode private space instead (like the Linux kernel cifsfs client does).
if we declase this problem as WONTFIX; then anybody with shell or NFS access to the shared data can easily DoS the Samba server by creating some latin1 encoded files.
As I said, first we need to know where the problem lies and without debug logs we don't yet know if it's in client or server. Are we seeing this error: 2008 if (NT_STATUS_EQUAL(status, NT_STATUS_ILLEGAL_CHARACTER)) { 2009 DEBUG(1,("Conversion error: illegal character: %s\n", 2010 smb_fname_str_dbg(smb_fname))); 2011 } in the server logs ? In smb2_query_directory_next_entry() we should be running into: 581 if (!NT_STATUS_IS_OK(status)) { 582 if (NT_STATUS_EQUAL(status, NT_STATUS_ILLEGAL_CHARACTER)) { 583 /* 584 * Bad character conversion on name. Ignore this 585 * entry. 586 */ 587 return false; which should just skip returning the bad name.
the "Conversion error: illegal character" appears, yes. I just found out that this is just a problem with the fruit AND its native encoding option: vfs objects = catia fruit fruit:encoding = native makes this reproducible.
(In reply to Björn Jacke from comment #9) Still need more details to decide where this is. Looking like a server issue yeah ? Can you upload a wireshark capture file of it working (no fruit module) and failing (with fruit module) so I can see what is actually being returned ?