My setup: current archlinux systems, both the server and the client have samba version 3.4.3. I'm sharing files (videos) with a simple setup and 'share' security privileges. When viewing the share using either smbclient or KDE dolphin's smb:// protocol, invalid Windows chars like colons (":") or or question marks ("?") result in mangling. I suspect libsmbclient as a result of both interfaces having this issue. Mounting with CIFS does not mangle the filenames.
After asking on the samba irc channel on freenode, user kukks confirmed this behavior on the git version as of the 18th of December, 2009. testparm indicates that unix extensions are enabled and I have attempted messing with the charset options several times (although I was informed this was not the problem), to no avail.
Contrary to the similar names, smbclient is not based on libsmbclient. The mangling occurs much deeper than either of those.
Jeremy, to you to work on or reassign.
*** Bug 5666 has been marked as a duplicate of this bug. ***