While trying to make Samba4 work as AD DC on a 128meg ARM box, I stumbled over a couple of samba processes using around 12 megabytes each. The memory usage as reported by top seems to decrease over time, but at startup time this forces the system to start swapping. Simo pointed out that the tdbs are mmapped and a likely cause for this. However, the suggestion of setting "use mmap = no" in smb.conf doesn't help as S4 does not recognize this parameter.
As per our discussion on IRC, running as a single process should be the goal on this platform. That seemed to fix things up pretty well for you. If you still think it's too bloated, feel free to re-open.