Rsync could use system loadavg (/proc/loadavg) measure to throttle disk operations on both sites. This especially includes file table operations as those are more costly. Checking loadavg every file is not a good idea, but checking it every x seconds (when next file accessed) could be ok. If max load is exceeded operations should freeze for few seconds (possibly another parameter) --load-check-interval-min = 5 (seconds) --max-load = 8 --load-freeze = 10(seconds)
io wait seems to be more reasonable than system load. system load simply telly how many processes are waiting for _something_, which may not be related to disk io at all.