It would be good if rsync used posix_fadvise with POSIX_FADV_DONTNEED to notify the kernel when it no longer needs file contents that have been synced to prevent useful cache contents being dropped in favor of never-reused rsynced files.
Tobias Oetiker wrote a patch & documentation about this:
What is the status of this? Grepping the source for fadvise turns up nothing. It seems the patch has even been ported up to 3.0.7 
I don't feel this patch is general enough nor straight-forward enough to make it into rsync. It would be better for someone to write a a pre-load library that could be used with any copy program rather than putting this into rsync.
The rsync filling up kernel cache is a problem on bigger backup servers. These days POSIX_FADV_DONTNEED is commonly implemented in unix systems.
Anyway as temporary/not optimal workaround: https://github.com/Feh/nocache (if not wanting to patch rsync).