I just tested with the mapchars options because I added support for those mappings to convmv. The weird thing I found out while testing is that it looks to me like mapchars and nomapchars are swapped:
on the local filesystem:
when mounted just with cifs vfs and option "nounix":
(same with explicit mount options "nounix,mapchars")
But when mounted with cifs vfs and options "nounix,nomapchars":
So it looks like a mapping is being done, when nothing or "nomapchars" is used and NO mapping is done when "mapchars" is used. The other way round would have been expected.
IMHO it would be okay, if "mapchars" would be made the default. From the behaviour actually it is already the default, but the naming is wrong :-)
This is a little confusing so I want to make sure I understand the question
"mapposix" is the mount option that is used by default and is the mapping used by Macs (and Linux SMB2/SMB3 for a few years) and Windows 'Services for Mac' by default
mapposix maps \ < > * ? among other characters to the remap range 0xF000 ...
There is an older less commonly used 'SFU' mapping from Interix's 'Service for Unix' (which we can handle if needed with the rarely used 'mapchars' mount option but it is rarely encountered).
Files natively created with '|' in them (ie locally in ext4 e.g.) will show up with mangled name. See below with 'mapchars'
# ls /mnt1
100M 4078 FB2HRH~M filewithpipe
Note that filewith|pipe created locally shows up mangled by Samba while the one created over default smb3 shows us with only the '|' looking strange.
While with the default "filewith|pipe' looks fine (the one created locally in ext4 is still mangled)
# ls /mnt1
100M 4078 FB2HRH~M 'filewith|pipe'
Seems to me that mapchars works as expected as does the more common mapposix. Do you see a problem with one of these?
Note that if you created files with macs or with Windows SFM you want to use the default ("mapposix") and only used "mapchars" mount option if you need SFU mapping instead of SFM reason (which should be a rare case).
Presumably if you create files with default ('SFM' ie "mapposix") mapping you want to make sure to continue to use default mount options and not switch it to a different mapping of the same range (ie "mapchars") where files created with "mapposix" could potentially appear to have different names.
I don't see anymore the problem I initially reported. Maybe I screwed something up on the server side with the mapping also. I'll close this bug report. Thanks!