When iterating through a directory, smbd_dirptr_get_entry() calls openat_pathref_fsp_nosymlink() on all directory entries, including files and subdirectories.  It's expected to create some sort of file descriptor for the directory entry, regardless of file permissions, so it uses O_PATH if available, which does this properly, and it tries O_SEARCH if that isn't available.

On some platforms (FreeBSD and AIX at least), O_SEARCH is ignored if the path specified to openat() doesn't identify a directory, so it presumably works there.  On others (Solaris/Illumos at least), openat() will return ENOTDIR unless the path identifies a directory.  This means smbd_dirptr_get_entry() will only return entries for directories, so clients cannot see any files.

Additionally, on Illumos at least, the ENOTDIR return in the openat() syscall happens after a call to VOP_ACCESS(), which in the case of O_SEARCH will check if the process has execute permission on the target file, so openat(fd, path, O_SEARCH) will actually return EACCES if the process doesn't have execute permission, and ENOTDIR if it does.

If there's any information I can provide to clarify what's going on here, I can certainly try to.