How can I convert to uppercase stderr output but not stdout?

Question

I'm using rsync to backup some files:

rsync -PaSh --stats --delete -e "-i key -p XXXX" "/source/" username@192.168.0.1:/destination/ 2> output.txt | grep -e 'bytes  received' -e 'total bytes' -e files -e 'total file size:' >> output.txt

because there are thousands of files, I only want to see any errors and a summary at the end.

The above command outputs this:

rsync: delete_file: unlink(test/test.txt) failed: Permission denied (13)
Number of files: 12 (reg: 10, dir: 2)
Number of created files: 0
Number of regular files transferred: 0
sent 382 bytes  received 137 bytes  41.52 bytes/sec

I want to convert to uppercase any errors only (to draw attention to them) and leave the summary unchanged.

So it would look like this:

RSYNC: DELETE_FILE: UNLINK(TEST/TEST.TXT) FAILED: PERMISSION DENIED (13)
Number of files: 12 (reg: 10, dir: 2)
Number of created files: 0
Number of regular files transferred: 0
sent 382 bytes  received 137 bytes  41.52 bytes/sec

How can I achieve this?

Thanks

Side comment/question not mentioned in my answer: wouldn't it be easier to just tail the rsync stdout to get the summary info? Or does it get displayed elsewhere besides the very end of the rsync run? (If it would work that's a lot better than scanning the whole thing with grep.) — B Layer, Jan 06 '18 at 20:36

B Layer · Accepted Answer · 2018-01-06T22:43:46.390

3

You can effectively swap stdin and stdout with the help of a temporary file descriptor (fd 3 here):

cmd 3>&2 2>&1 1>&3- | tr '[:lower:]' '[:upper]'

That says "point new fd at the place stderr points to, point stderr where stdout points, point stdout to where stderr originally pointed"...easy, right? :)

Or in other words: now the stderr from cmd will go through the pipe like stdout while the original stdout has been redirected to stderr. And the tr command will do the upper-case conversion.

Update: Though you accepted the answer above your wishlist included the ability to manipulate stdout and stderr simultaneously. So let's try to tackle that...keeping in mind that we're bumping up against some things I don't do too often!

A simple swapping of stdout/stderr won't do anything useful. With or without the swap you can't use two different streams through a single pipeline. The first thing that comes to mind as an alternative is to use FIFOs (named pipes):

mkfifo /tmp/fifo1 /tmp/fifo2

# run the key command in the background, in a subshell to suppress
# job control messages (e.g. "[1] 12345" and "[1]+ Exit ...")
( cmd > /tmp/fifo1 2> /tmp/fifo2 & )

# concatenate tr/stderr and grep/stdout using process substitution
# (I'm assuming the summary information occurs at the end of rsync output)
cat <(cat /tmp/fifo2 | tr '[:lower:]' '[:upper:]') <(grep ... /tmp/fifo1) 

rm -f /tmp/fifo1 /tmp/fifo2

You could actually put this all on one line if so inclined!

mkfifo /tmp/fifo1 /tmp/fifo2; (cmd > /tmp/fifo1 2> /tmp/fifo2 &); cat <(cat /tmp/fifo2 | tr '[:lower:]' '[:upper:]') <(grep ... /tmp/fifo1); rm -f /tmp/fifo1 /tmp/fifo2

My sanity checks are working okay so give it a try.

edited Jan 06 '18 at 22:43

answered Jan 06 '18 at 12:17

B Layer

5,171

Thanks for your reply; I'm getting closer. I'm now using this: rsync -PaSh --stats --delete -e "-i key -p XXXX" "/source/" username@192.168.0.1:/destination/ 3>&2 2>&1 1>&3 | tr [:lower:] [:upper:] > output.txt | grep -e 'bytes received' -e 'total bytes' -e files -e 'total file size:' >> output.txt which works in converting stderr to uppercase. Unfortunately I lose the ability to append stdout to the file and instead it just appears on screen. Do I need to add 1> somewhere? – NoExpert Jan 06 '18 at 16:44
@NoExpert You want to save to file the original stdout straight from the rsync command, yes? – B Layer Jan 06 '18 at 17:02
Only one stream can go through those pipes. You can't have both the tr for the stderr and grep for the stdout in a single command. – B Layer Jan 06 '18 at 17:07
Yes, after greping what I need, so as in my example in the original above. Thank you.
EDIT: OK I've seen your updated answer now. That makes sense. I guess I'll have to live with it.
– NoExpert Jan 06 '18 at 17:08
I think you can accomplish what you want if you're willing to use named pipes (FIFOs). I'll post an update to my answer. – B Layer Jan 06 '18 at 20:06
That fifo approach is prone to deadlocks – Stéphane Chazelas Jan 07 '18 at 07:41

score 1 · Answer 2 · answered Jan 07 '18 at 07:37

To convert stderr to uppercase without changing where stdout and stderr go to:

{ rsync ... 2>&1 >&3 3>&- | perl -Mopen=locale -pe '$_=uc' >&2 3>&-; } 3>&1

That is duplicate the original stdout onto fd 3, so we can restore it for rsync.

Here using perl instead of tr as it gives better results for things like Stéphane -> STÉPHANE (STéPHANE with GNU tr) or traﬃc -> TRAFFIC (TRAﬃC with GNU tr/sed/awk).

Note that now that stderr goes through a pipe, that can affect the relative order of stdout and stderr messages as the stderr messages are now being delayed.

See

Can I configure my shell to print STDERR and STDOUT in different colors?

for some other approaches that you might find useful.

score 0 · Answer 3 · answered Jan 06 '18 at 13:01

Use below command instead of your command. I have added one extra command after redirecting the error to output.txt

I have added sed -i "s/.*/\U&/g" output.txt
As tested its works fine

rsync -PaSh --stats --delete -e "-i key -p XXXX" "/source/" username@192.168.0.1:/destination/ 2> output.txt ;sed -i  "s/.*/\U&/g" output.txt| grep -e 'bytes  received' -e 'total bytes' -e files -e 'total file size:' >> output.txt

How can I convert to uppercase stderr output but not stdout?

3 Answers3