With the following share definition:
path = /srv/share1
browseable = yes
read only = no
guest ok = no
create mask = 0770
force create mode = 0660
directory mask = 2770
force directory mode = 2770
I create a directory:
cd /srv/share1; mkdir modes && chmod 2770 modes
There from a Windows client I connect to the share and create some files & directories:
New text document.txt*
Entries marked with * have their default names (picked by Windows) untouched.
Here is what happens:
└── [drwxrws---] ./modes
├── [-rwxr-----] ./modes/New text document.txt
├── [drwxrws---] ./modes/New folder
├── [drwxrws---] ./modes/test
└── [-rwxrw----] ./modes/test.txt
Despite force create mode the "New text document.txt" does not have group-write permission, while another file with custom name has the permissions set as desired.
When I rename "New text document.txt" to anything else the permissions are fixed immediately:
└── [-rwxrw----] ./modes/test2.txt
This was not observable with Samba 4.1.6.