It should be possible to set port for ssh transfer.
It is possible:
rsync -ave 'ssh -p 4242' host:/src/ /dest/
However, you're much better off using the ~/.ssh/config file to define what port
"host" uses so you don't need to specify it manually each time.