When changing times on a symlink in Mac OS X, the system actually sets the times on the symlink destination file. If the destination file does not exist, syscall returns error. This behaviour causes rsync to report many "failed to set times on" errors when source/destination directories contain symlinks which do not point to existing files. It would be nice if (on Mac OS X only) setting of times to symlinks would be appropriately guarded or not executed at all since it does not behave correctly.
Rsync only tries to set times on symlinks if the lutimes() function exists. On my version of OS X (Panther) the setting of the time on a symlink worked fine. If your system says it has lutimes() but it doesn't work, comment out the HAVE_LUTIMES in config.h and recompile.