Created attachment 11225 [details] patch to implement --line-buffered option Behavior change with --line-buffered would be primarily to --progress - which would output a newline after percentage update instead of just a carriage-return. During a normal operation with smaller files you'd never notice the difference, but with large files (say recurrent sync of ISO images or similar) - you'd get a MUCH more usable output trace in the build system and logs instead of it all being merged onto one "line" of output.
--outbuf=N|L|B set out buffering to None, Line, or Block
Perhaps the naming is not correct on my suggested option (and I'll admit, I completely missed the outbuf option) - unfortunately, outbuf doesn't actually solve the problem. The goal is to get incremental progress output while running rsync through a build system or similar. As near as I can tell, with outbuf, it still writes out a "single line" into the output stream, so you get result like this: custom.iso ^M 32,768 0% 0.00kB/s 0:00:00 ^M 264,765,440 29% 252.47MB/s 0:00:02 ^M 508,592,128 57% 242.62MB/s 0:00:01 ^M 778,338,304 87% 247.58MB/s 0:00:00 ^M 890,107,904 100% 250.03MB/s 0:00:03 (xfr#1, to-chk=0/1) Which doesn't mesh nicely with build tools, or simple stuff like: rsync ..... | logger -t nightly-sync Which results in a mess like: ... Jul 4 12:52:59 skyhawk rsync-test: sending incremental file list Jul 4 12:52:59 skyhawk rsync-test: custom.iso Jul 4 12:53:01 skyhawk CROND[13401]: (nneul) CMD (/local/thermo/grab-status.pl >> /local/thermo/status.log 2>&1) 17,764,896 1% 597.84kB/s 0:24:20 0% 0.00kB/s 0:00:00 33,587,200 3% 641.25kB/s 0:22:16 % 613.17kB/s 0:23:42 Jul 4 12:54:01 skyhawk CROND[13448]: (nneul) CMD (/local/thermo/grab-status.pl >> /local/thermo/status.log 2>&1) ... Instead of (output with my suggested patch): Jul 4 12:54:46 skyhawk rsync-test: custom.iso Jul 4 12:54:46 skyhawk rsync-test: 32,768 0% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1) Jul 4 12:54:47 skyhawk rsync-test: 2,818,048 0% 2.31MB/s 0:00:01 (xfr#1, to-chk=0/1) Jul 4 12:54:48 skyhawk rsync-test: 3,538,944 0% 1.56MB/s 0:00:02 (xfr#1, to-chk=0/1) Jul 4 12:54:49 skyhawk rsync-test: 4,128,768 0% 1.24MB/s 0:00:03 (xfr#1, to-chk=0/1) Jul 4 12:54:50 skyhawk rsync-test: 4,784,128 0% 1.06MB/s 0:00:04 (xfr#1, to-chk=0/1) Jul 4 12:54:51 skyhawk rsync-test: 5,439,488 0% 1003.42kB/s 0:00:05 (xfr#1, to-chk=0/1) Jul 4 12:54:52 skyhawk rsync-test: 6,094,848 0% 945.38kB/s 0:00:06 (xfr#1, to-chk=0/1) Perhaps "line-progress" or something?
On Sat, 04 Jul 2015 17:56:25 +0000 samba-bugs@samba.org wrote: > --- Comment #2 from Nathan Neulinger <nneul@neulinger.org> --- > Perhaps the naming is not correct on my suggested option (and I'll > admit, I completely missed the outbuf option) - unfortunately, outbuf > doesn't actually solve the problem. > > The goal is to get incremental progress output while running rsync > through a build system or similar. What would happen if you piped the rsync output through tr and changed \r to \n? Karl <kop@meme.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein
Unfortunately, that just moves the buffering issue to tr instead of rsync.
With some other tweaks to my local scripts I was able to make outbuf=L work. I'd still like to recommend that there be some way to request that the --progress output use CR instead of LF for piped output though. Maybe a special "--outbuf=M" for "machine processed" - which would both set to line buffered AND use full CR.