I compiled Samba 3.0.25pre1 on AIX 5.3. Server and Windows-Clients use codepage ISO-8859-15. I read in Samba-3 description, Samba tries to use UTF-8 for translate international characters. I tried to switch the Server to UTF-8. Samba-Clients work, but I have some problems with Unix-Applications. I switched Server back to ISO-8859-15 and set dos and unix-charset in smb.conf to ASCII to avoid unnecessary character translation. If I open Samba-Filesystem on Windows-Client with Windows-Explorer, german umlauts are displayed correct, but I cant open any files with umlauts in file-names on the Windows-Client with any application. It looks for me, Samba does an unnecessary internal character recoding. If I open a dos-Window and check files in Samba-Filesystem with dir /x the old 8.3-Filenames of files with international characters contain strange letters, other letters as Windows would give it. I hope, its possible to fix this bug. I never had this problems in Samba 1 or 2.
you should never set dos/unix charset to ascii as that should break any "special" character handling. dos charset should be kept at your language's default that Windows 9x/DOS clients use (cp850 or other strance codepage). unix charset should stay at utf-8 for best interoperability with unicode win* clients. If you need 8bit filenames on the server because you need non-UTF-8 filenames via NFS or for local unix applications you can use iso8859-1 but that is non-optimal for samba. When you change the unix charset from the previous setting to a new one you will have to convert the existing files with convmv. On AIX make sure to have samba compiled agains libiconv as IBM ships a poor iconv implementation with AIX that does not support enough charsets, I would consider it broken.