Shell script exits early for unclear reasons

Question

I'm using the following script to backup, using Borg Backup. But I'm running into an odd problem

Part of the script below does not execute. The script follows, then the output

SCRIPT:

#!/bin/bash                                                                                                                                                  

set -ex
set -o pipefail
export SERVER="myserver"
CLEVEL=zlib,9
. /home/faheem/.keychain/${HOSTNAME}-sh

#borg init -e none faheem@$SERVER:/mnt/backup-test                                                                                                         
borg create -c 30 --compression $CLEVEL --stats faheem@ramnode:/mnt/backup-test::`hostname`-`date +%Y-%m-%d:%H.%M`  /home/faheem/test-borg                   
{ borg check faheem@$SERVER:/mnt/backup-test 2>&1 1>&3 | tr '\r' '\n' | grep -Ev "^Remote:\s*(Checking segments.*)?$" 1>&2; } 3>&1
mapfile -t testarchives < <(borg list --short faheem@$SERVER:/mnt/backup-test)
borg extract -n "faheem@$SERVER:/mnt/backup-test"::"${testarchives[-1]}"

OUTPUT:

+ set -o pipefail
+ export SERVER=ramnode
+ SERVER=ramnode
+ CLEVEL=zlib,9
+ . /home/faheem/.keychain/orwell-sh
++ SSH_AUTH_SOCK=/tmp/ssh-F7Uzg6CeQoTY/agent.5660
++ export SSH_AUTH_SOCK
++ SSH_AGENT_PID=5661
++ export SSH_AGENT_PID
++ hostname
++ date +%Y-%m-%d:%H.%M
+ borg create -c 30 --compression zlib,9 --stats faheem@ramnode:/mnt/backup-test::orwell-2016-09-16:05.04 /home/faheem/test-borg
+ borg check faheem@ramnode:/mnt/backup-test
+ tr '\r' '\n'
+ grep -Ev '^Remote:\s*(Checking segments.*)?$'

So the last two lines don't execute. But if I comment out the earlier lines, those two lines execute. Does anyone know what is going wrong?

UPDATE: If I take out grep (i.e. replace)

{ borg check faheem@$SERVER:/mnt/backup-test 2>&1 1>&3 | tr '\r' '\n' | grep -Ev "^Remote:\s*(Checking segments.*)?$" 1>&2; } 3>&1

with

{ borg check faheem@$SERVER:/mnt/backup-test 2>&1 1>&3 | tr '\r' '\n' 1>&2; } 3>&1

I get

+ set -o pipefail
+ export SERVER=ramnode
+ SERVER=ramnode
+ CLEVEL=zlib,9
+ . /home/faheem/.keychain/orwell-sh
++ SSH_AUTH_SOCK=/tmp/ssh-F7Uzg6CeQoTY/agent.5660
++ export SSH_AUTH_SOCK
++ SSH_AGENT_PID=5661
++ export SSH_AGENT_PID
++ hostname
++ date +%Y-%m-%d:%H.%M
+ borg create -c 30 --compression zlib,9 --stats faheem@ramnode:/mnt/backup-test::orwell-2016-09-17:03.18 /home/faheem/test-borg
+ borg check faheem@ramnode:/mnt/backup-test
+ tr '\r' '\n'
Remote: Checking segments 0.0%
Remote:                         
+ mapfile -t testarchives
++ borg list --short faheem@ramnode:/mnt/backup-test
+ borg extract -n faheem@ramnode:/mnt/backup-test::orwell-2016-09-17:03.18

Does the Remote: line correspond to grep failing to match the pattern and returning non-zero?

Here are a couple of relevant questions:

Can grep return true/false or are there alternative methods

and

Avoid grep returning error when input doesn't match

The latter question appears to be a similar situation, if not the same.

score 3 · Answer 1 · answered Sep 15 '16 at 23:42

3

The set -e means that if grep returns non-zero (ie the pattern is not matched) then that, combined with pipefail means that the pipe fails and causes the script to abort.

So either

remove set -e
remove pipefail
Add an || true in the grep segment to ensure that always returns zero

answered Sep 15 '16 at 23:42

Stephen Harris

44,540

1

The truthfulness of a pipeline without pipefail is the exit status of the last command. Therefore, removing pipefail doesn't help him here if the grep is not matching. – jordanm Sep 16 '16 at 01:43
Right, but if other components of the pipeline are returning non-zero then it might. – Stephen Harris Sep 16 '16 at 02:08
Hi, Stephen. The third option looks like it would be the best. I don't need grep to succeed. So, if it doesn't match a pattern, that's considered an error? Weird. What would the syntax look like, exactly? The syntax of that particular command is already quite convoluted. – Faheem Mitha Sep 16 '16 at 09:57
I would try replacing grep -Ev "^Remote:\s*(Checking segments.*)?$" 1>&2 with {grep -Ev "^Remote:\s*(Checking segments.*)?$" 1>&2 || true;} but it might need some tuning. – Stephen Harris Sep 16 '16 at 10:54
1

Or you could use sed, since grep is designed to have its exit status set according to whether the pattern is matched or not. – Wildcard Sep 16 '16 at 22:45
Thanks for the comment, @Wildcard. What would the syntax look like in this case? Would you like to write an answer? – Faheem Mitha Sep 16 '16 at 23:00
I tried { borg check faheem@$SERVER:/mnt/backup-test 2>&1 1>&3 | tr '\r' '\n' | { grep -Ev "^Remote:\s*(Checking segments.*)?$" || true } 1>&2; } 3>&1 but this gives: "line 20: syntax error: unexpected end of file". My shell knowledge is pretty sketchy, sorry. – Faheem Mitha Sep 17 '16 at 00:27

score 1 · Answer 2 · answered Sep 17 '16 at 00:37

1

grep is designed to set an exit status according to whether or not a particular pattern is found, and in some cases is even used only for this purpose and any output is suppressed. (That's the purpose of the -q switch to grep.)

sed is the "Stream EDitor" and is designed to filter a stream of text, converting it in some way (usually line by line) as it passes through.

Although there is a great deal of overlap between these two utilities, they do each have a specific purpose. In this case you are evidently not interested in whether or not the pattern is found, you simply want to filter the text stream before passing it along. That is a proper use case for sed, not for grep.

sed -E '/^Remote:\s*(Checking segments.*)?$/d' delme.txt

This is the exact equivalent of:

grep -Ev '^Remote:\s*(Checking segments.*)?$' delme.txt

...except that the grep command will give an error exit status if there are no lines besides those matching the pattern, and sed will not.

answered Sep 17 '16 at 00:37

Wildcard

36,499

Thanks for the answer. On the face of it, sed does indeed look like a better choice here. Does the $/d simply remove a line that matches the regex on the left; i.e. ^Remote:\s*(Checking segments.*)?$? I'm not familiar with sed. – Faheem Mitha Sep 17 '16 at 01:00
@FaheemMitha, yes; lines which match the regex (/.../) have the delete command applied to them. Other lines have sed's default action applied, which is to print the line (unless suppressed with -n). – Wildcard Sep 17 '16 at 01:20
Ah. Thanks for the clarification, Wildcard. – Faheem Mitha Sep 17 '16 at 01:21
@FaheemMitha you're welcome. The best free tutorial for learning sed is the Grymoire page. I encourage you to read up on it--I've found sed indispensable as a sysadmin. :) – Wildcard Sep 17 '16 at 01:22
Thanks for the link, Wildcard. That looks like basically a book. – Faheem Mitha Sep 17 '16 at 01:27
Oh, and one more thing. Does sed only operate on standard output? – Faheem Mitha Sep 17 '16 at 10:39
@FaheemMitha, you mean standard input? No, it can work on files (any number of files). – Wildcard Sep 17 '16 at 11:00
Hi Wildcard. I think I meant standard output, but maybe I mean standard input. The reason I'm asking is that I had to do gyrations with file descriptors (see above) so grep could act on standard output (or input). – Faheem Mitha Sep 17 '16 at 11:36
Let us continue this discussion in chat. – Wildcard Sep 17 '16 at 11:48

Shell script exits early for unclear reasons

2 Answers2