I'm backing up my computer to a server.  I've got almost a terabyte of data in  100k of files, and they don't change much each day.  When rsync runs, it traverses the destination directory on the server to find changes, which chews up a lot of network bandwidth, cpu time, and disk seeks compared to the amount of actual data to send, if any.

After reading the manual page, the --write-batch/--read-batch commands look promising.  As I'm the only one writing to the destination directory, I could cache the current state of the destination directory in a local file, then generate a batch file based on that.  This would cut out 99% of the overhead we're seeing now, reducing a 6-hour rsync to maybe 15 minutes (ie: the time to traverse the source filesystem).

Alternatively I could do:
   find /source-directory -type f -ls | sort > new.txt
   comm -3 old.txt new.txt | gawk -F"\t" '{print $NF}' > todo.txt
   rsync -av --files-from=todo.txt /remote-dest-directory && mv new.txt old.txt
and pray there are no filenames with funky characters in them.

Is this caching feature simple enough to implement?  

Thanks!