On somewhat long-running samba AD DC instances (4.5.8-Debian, Stretch), we're seeming massive RAM utilization even with little/no clients connected (1+ GB, possibly more, if OOM killer wouldn't intervene). However, as far as I can see (testparm), aio is on its defaults, and should be disabled:
ps, testparm outputs: https://up.tao.at/-16a27a843f64f278719a4ee6639011f2/ps-testparm-1.txt
smbd pool-usage: https://up.tao.at/-13984af5bf5d36c3da17685a7f76bc8b/pool-usage.txt
The above data is from last week (reported to mailing list then), and OOM killer has slain the offending process since. Thankfully(?) the bug also popped up on another machine, so I can provide additional debug data if needed:
ps, testparm outputs: https://up.tao.at/-887f2fc8673b6dfb676f3d25db394460/ps-testparm-2.txt
smbd pool-usage: https://up.tao.at/-51345a0eba4958b0daddcadde31aed62/pool-usage-2.txt
FWIW, we can only reproduce the issue on one of our physical sites, where we only have "BDCs" (i.e., not holding any FSMO roles) that serve Windows clients. The other site with the FSMO holder and another "BDC", serving mixed Linux/OSX clients, does not seem to show the problems on either DC. All four DCs are running Debian Stretch with Debian's default Samba build.