Bug 11378 - Please add a '--line-buffered' option to rsync to make logging/output more friendly with pipes/syslog/CI systems/etc.
Please add a '--line-buffered' option to rsync to make logging/output more fr...
Status: RESOLVED WONTFIX
Product: rsync
Classification: Unclassified
Component: core
3.1.1
All All
: P5 enhancement
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2015-07-02 18:00 UTC by Nathan Neulinger
Modified: 2015-07-23 13:38 UTC (History)
0 users

See Also:


Attachments
patch to implement --line-buffered option (2.14 KB, patch)
2015-07-02 18:00 UTC, Nathan Neulinger
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nathan Neulinger 2015-07-02 18:00:19 UTC
Created attachment 11225 [details]
patch to implement --line-buffered option

Behavior change with --line-buffered would be primarily to --progress - which would output a newline after percentage update instead of just a carriage-return.

During a normal operation with smaller files you'd never notice the difference, but with large files (say recurrent sync of ISO images or similar) - you'd get a MUCH more usable output trace in the build system and logs instead of it all being merged onto one "line" of output.
Comment 1 Wayne Davison 2015-07-04 17:06:14 UTC
            --outbuf=N|L|B          set out buffering to None, Line, or Block
Comment 2 Nathan Neulinger 2015-07-04 17:56:25 UTC
Perhaps the naming is not correct on my suggested option (and I'll admit, I completely missed the outbuf option) - unfortunately, outbuf doesn't actually solve the problem. 

The goal is to get incremental progress output while running rsync through a build system or similar.

As near as I can tell, with outbuf, it still writes out a "single line" into the output stream, so you get result like this:

custom.iso
^M         32,768   0%    0.00kB/s    0:00:00  ^M    264,765,440  29%  252.47MB/s    0:00:02  ^M    508,592,128  57%  242.62MB/s    0:00:01  ^M    778,338,304  87%  247.58MB/s    0:00:00  ^M    890,107,904 100%  250.03MB/s    0:00:03 (xfr#1, to-chk=0/1)


Which doesn't mesh nicely with build tools, or simple stuff like:

   rsync ..... | logger -t nightly-sync

Which results in a mess like:

...
Jul  4 12:52:59 skyhawk rsync-test: sending incremental file list
Jul  4 12:52:59 skyhawk rsync-test: custom.iso
Jul  4 12:53:01 skyhawk CROND[13401]: (nneul) CMD (/local/thermo/grab-status.pl >> /local/thermo/status.log 2>&1)
     17,764,896   1%  597.84kB/s    0:24:20  0%    0.00kB/s    0:00:00  
     33,587,200   3%  641.25kB/s    0:22:16  %  613.17kB/s    0:23:42  
Jul  4 12:54:01 skyhawk CROND[13448]: (nneul) CMD (/local/thermo/grab-status.pl >> /local/thermo/status.log 2>&1)
...


Instead of (output with my suggested patch):

Jul  4 12:54:46 skyhawk rsync-test: custom.iso
Jul  4 12:54:46 skyhawk rsync-test: 32,768   0%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/1)
Jul  4 12:54:47 skyhawk rsync-test: 2,818,048   0%    2.31MB/s    0:00:01 (xfr#1, to-chk=0/1)
Jul  4 12:54:48 skyhawk rsync-test: 3,538,944   0%    1.56MB/s    0:00:02 (xfr#1, to-chk=0/1)
Jul  4 12:54:49 skyhawk rsync-test: 4,128,768   0%    1.24MB/s    0:00:03 (xfr#1, to-chk=0/1)
Jul  4 12:54:50 skyhawk rsync-test: 4,784,128   0%    1.06MB/s    0:00:04 (xfr#1, to-chk=0/1)
Jul  4 12:54:51 skyhawk rsync-test: 5,439,488   0% 1003.42kB/s    0:00:05 (xfr#1, to-chk=0/1)
Jul  4 12:54:52 skyhawk rsync-test: 6,094,848   0%  945.38kB/s    0:00:06 (xfr#1, to-chk=0/1)


Perhaps "line-progress" or something?
Comment 3 Karl O. Pinc 2015-07-04 18:56:21 UTC
On Sat, 04 Jul 2015 17:56:25 +0000
samba-bugs@samba.org wrote:

> --- Comment #2 from Nathan Neulinger <nneul@neulinger.org> ---
> Perhaps the naming is not correct on my suggested option (and I'll
> admit, I completely missed the outbuf option) - unfortunately, outbuf
> doesn't actually solve the problem. 
> 
> The goal is to get incremental progress output while running rsync
> through a build system or similar.

What would happen if you piped the rsync output through
tr and changed \r to \n?



Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein
Comment 4 Nathan Neulinger 2015-07-07 15:33:48 UTC
Unfortunately, that just moves the buffering issue to tr instead of rsync.
Comment 5 Nathan Neulinger 2015-07-23 13:38:24 UTC
With some other tweaks to my local scripts I was able to make outbuf=L work. I'd still like to recommend that there be some way to request that the --progress output use CR instead of LF for piped output though.

Maybe a special "--outbuf=M" for "machine processed" - which would both set to line buffered AND use full CR.