If a lot of clients (a few hundred) access (open/close) the same path at the
time they all try to grab a lock on the locking.tdb record for the inode.

The following (recently added) test demonstrates the behavior change between
different Samba versions. I was testing on a AMD Ryzen 9 5950X 16-Core Processor
using for cpu in $(seq 0 31); do cpufreq-set -c $cpu -g performance; done
and exporting /dev/shm.

smbtorture //127.0.0.1/m -Utest%test smb2.create.bench-path-contention-shared \
        --option='torture:bench_path=' \
        --option="torture:timelimit=60" \
        --option="torture:nprocs=256"


On the server I used:
- v4-6-test:
  open[num/s=4977,avslat=0.025800,minlat=0.000036,maxlat=0.383098]
  close[num/s=4975,avslat=0.025981,minlat=0.000021,maxlat=0.355635]

- v4-11-test:
  open[num/s=4515,avslat=0.028414,minlat=0.000039,maxlat=0.084519]
  close[num/s=4513,avslat=0.028810,minlat=0.000021,maxlat=0.074114]

  This is little bit less than 4.6, but no major change.

- v4-12-test:
  open[num/s=36458,avslat=0.003613,minlat=0.000040,maxlat=0.015829]
  close[num/s=36456,avslat=0.003590,minlat=0.000023,maxlat=0.015742]

  This was a bit unexpected to me as ~36500 ops/s is the hardware limit
  I also get by using fileid and nolock. The reason is that 4.12
  had a separate share_entries.tdb, which means there's no real contention
  on the locking.tdb entry. And we never hit the dbwrap_watch code path.

  This is also the number we get when disabling lock contention
  via vfs_fileid. It seems to be the maximum for that test case on
  the server.

- v4-13-test - master:
  open[num/s=54,avslat=2.207252,minlat=0.006880,maxlat=26.734115]
  close[num/s=52,avslat=2.303759,minlat=0.007108,maxlat=25.945049]

  The change to g_lock re-added a thundering herd problem again
  because we always go via the dbwrap_watch when someone else
  hold the g_lock already. Currently any g_lock modification
  g_lock_lock(), modify of the payload and g_lock_unlock()
  wakes all watchers and then all but one comeback and
  see the record is still locked.

With the changes in https://gitlab.com/samba-team/samba/-/merge_requests/2608
we have an optimized interaction between g_lock and dbwrap_watch and we get the
following result:

  open[num/s=10249,avslat=0.012551,minlat=0.000072,maxlat=0.051488] 
  close[num/s=10247,avslat=0.012576,minlat=0.000031,maxlat=0.051464]

This is more than twice than 4.6 and 200 times than 4.13-master.


For the future are some ideas to improve this further by avoiding the current pattern of:

1. lck = get_share_mode_lock()
         1.1 g_lock_lock()
2. do some syscalls
3. set_share_mode()
    3.1 g_lock_writev_data()
4. TALLOC_FREE(lck)
    4.1 g_lock_unlock()

In a non conflicting case we could have a g_lock_do_locked()
that directly stores our share_mode entry if there's not existing g_lock,
Then we can have a share_mode_keep(lck), that tells the destructor to do
nothing, otherwise the destructor would remove the optimistic share_mode_entry
again. With that we would only fallback to the existing pattern is there's a conflict or an existing g_lock.

For the fallback we could even defer set_share_mode() and have
a g_lock_writev_data_and_unlock() function.

This would hopefully get us back to the 4.12 case where we had share_entries.tdb.

The idea of the share_entries.tdb can also comeback in a way
that we split locking.tdb records based cluster nodes
and maybe on a modulo operation on the pid. It means
the entry header (struct share_mode_data) with share_mode_flags flags
will be mirrored in all entries and only on conflict we need to check all
individual records. (The single opener case should still be optimized and only work on a single record).

This bug was referenced in samba master:

f26b22cc8e784af1dc64e0aa9a2c948385e2b667
3f88b700a9e73b2371cba3bb867c4c625bc36c7c
5af37ae6970674573bb6f32c6c0f347718b8a2a9
77db4b666f0eb6668b6e0bc3d445048980542bfd
cdf1c37a90c24cc32953865a0bb909fd71fbbf05
420a595c1baa6062fcb7f672cf032287b4521a76
9356b1701c43c9ec2a71252a31e09555a1f5a3ea
7226d0b365625ab0e5ff0ffe8b8fe3924ae1a569
6702b3b0da729b824455a5efc7b50ebc03f835e8
cb012e45c91b4b7d1572864b7e60de36c96a38af
b3f6668f93b73a5980dc418c6b1dacde330f7497
2342489f526a43fe2d56f3859b6df0e26538870d
c0febbd3e19a1c40c48d93f38f9223a7feab8f30
095fafbe0c901180b30253b9adb6b2de3bf827f2
eb89748ee4a9c58da8064b41d0253e00c30c213f
726f468ccdb432c35dadef1aa4af6a041cbf46f2
6e45da1a38d31c18e38397f354596f6c4654c8b1
5021abff88ccaeb8b7a2b274e4ddc9ea046b2c34
6b173bf1569b4c099223a090f021465eee04747e
39cdcec49c6578430d91c450645de1522fe26c2a
1c84980d7c3ca33c61c8137902af2fc7a0988824
8908af569551def4e06c9aa179cf77e0beaa2638
2129d352ae0d90867ebe8101529817a12223d5ee
908eea1202459c8ffdf8e43eb310cf8690693824
1fb9db8c994b60720ea82167b5a91808293640e0
cc9c8b8e7e0e2ab8512fb443b89780a11bc966ae
044e018e9a243466bdbf075304a2715f36c0199a
2eb6a209493e228e433a565154c7c4cd076c0a4a
50163da3096d71c08e9c3ff13dd3ecb252a60f4e
f62beaa2c25043a1b3dde408ce7d8f7327ac3e13
b865bb28abedd18d5db0ae3ea31908c75ee1129b
52720516998091206399e9b2387d8ee8265e4660
2e92267920ddee7b96f32e243eb7b21124ed554f
20f3fd021911a118228815cb32d436069b0273d1
e33143099b37a4d79a1cd45ca43a5a5d9d63b261
67af3586d989be9d6a8fe7e7789250451b03f2bb
98269bd5f31a2521b756e0a20fba82e9122582f7
6e701d02ee2d0fc304157395c451d3b972128cfc
9d9991166322477781f20372ffd7c19d1632276c
d5c7e2e27385a4730ff8674650700bd25b77f975