Elegant way to construct a pipeline based on return value and not exit code?

Question

When the status code is useless, is there anyway to construct a pipeline based on output from stdout?

I'd prefer the answer not address the use-case but the question in the scope of shell scripting. What I'm trying to do is find the most-specific package available in the repository by guessing the name based on country and language codes.

Take for instance this,

• $PACKAGE1=hunspell-en-zz
• $PACKAGE2=hunspell-en

The first guess is more appropriate but it may not exist. In this case, I want to return hunspell-en ($PACKAGE2) because the first option hunspell-en-zz ($PACKAGE1) does not exist.

pipelines of apt-cache

The command apt-cache returns success (which is defined by shell as exit-code zero) whenever the command is able to run (from the docs of apt-cache)

apt-cache returns zero on normal operation, decimal 100 on error.

That makes using the command in a pipeline more difficult. Normally, I expect the package-search equivalent of a 404 to result in an error (as would happen with curl or wget). I want to search to see if a package exists, and if not fall back to another package if it exists.

This returns nothing, as the first command returns success (so the rhs in the || never runs)

apt-cache search hunspell-en-zz || apt-cache search hunspell-en

apt-cache search with two arguments

This returns nothing, as apt-cache ANDs its arguments,

apt-cache search hunspell-en-zz hunspell-en

From the docs of apt-cache

Separate arguments can be used to specify multiple search patterns that are and'ed together.

So as one of those arguments clearly doesn't exist, this returns nothing.

The question

What is the shell idiom to handle conventions like those found in apt-cache where the return code is useless for the task? And success is determined only by the presence of output on STDOUT?

Similar to

• make find fail when nothing was found

  they both stemming from the same problem. The chosen answer there mentions find -z which sadly isn't applicable solution here and is use-case specific. There is no mention of an idiom or constructing a pipeline without using null-termination (not an option on apt-cache)

Answer 1

Create a function that takes a command and returns true iff it has some output.

r() { local x=$("$@"); [ -n "$x" ] && echo "$x"; }

( ( r echo -n ) || echo 'nada' ) | cat      # Prints 'nada'
( ( r echo -n foo ) || echo 'nada' ) | cat  # Prints 'foo'

So for this use case it 'll work like this,

r apt-cache search hunspell-en-zz || r apt-cache search hunspell-en

Answer 2

As far as I know, there is no standard way to deal with those cases where the success of a command is determined by the presence of output. You can write workarounds, though.

For example, you can save the output of the command in a variable and then check if that variable is empty or not:

output="$(command)"

if [[ -n "${output}" ]]; then
  # Code to execute if command succeded
else
  # Code to execute if command failed
fi

I think this answers the question in a general way, but if we talk about apt-cache search some solutions come to my mind.

I have a script which makes package management easier. Some of its functions are these:

search() {
  local 'package' 'packages'
  packages="$( apt-cache search '.*' | cut -d ' ' -f '1' | sort )"
  for package; do
    grep -F -i -e "${package}" <<< "${packages}"
  done
}


search_all() {
  local 'package'
  for package; do
    apt-cache search "${package}" | sort
  done
}


search_description() {
  local 'package' 'packages'
  packages="$( apt-cache search '.*' | sort )"
  for package; do
    grep -F -i -e "${package}" <<< "${packages}"
  done
}


search_names_only() {
  local 'package'
  for package; do
    apt-cache search --names-only "${package}" | sort
  done
}

These let you to do multiple searches in a single command. For example:

$ search hunspell-en-zz hunspell-en
hunspell-en-au
hunspell-en-ca
hunspell-en-gb
hunspell-en-med
hunspell-en-us
hunspell-en-za

Every function search the database in a different way, so the results may vary depending of wich function you use:

$ search gnome | wc -l
538
$ search_all gnome | wc -l
1322
$ search_description gnome | wc -l
822
$ search_names_only gnome | wc -l
550

Answer 3

I wouldn't call this elegant but I think it might do the job:

search_packages () {
    local packages=($@)
    local results=()
    for package in "${packages[@]}"; do
        results=($(apt-cache -n search "$package"))
        if [[ "${#results[@]}" -eq 0 ]]; then
            echo "$package not found."
        elif [[ "${#results[@]}" -eq 1 ]]; then
            do stuff with "$package"
        else
            echo "Warning! Found multiple packages for ${package}:"
            printf '\t-> %s\n' "${results[@]}"
        fi
    done
}

I don't have a debian machine to test on unfortunately. I've included the -n for "names-only" option of apt-cache to try and limit the search results as it looks like you are mostly sure of what you are searching.

Can be run like:

$ search_packages hunspell-en-zz hunspell-en
$ my_packages=('hunspell-en-zz' 'hunspell-en')
$ search_packages "${my_packages[@]}"

Answer 4

Muru clarified this in the comments grep will return a status of 1 if there is no input. So you can add grep . into the stream and if there is no input to match the pattern ., it'll change the status code:

( ( echo -n | grep . ) || echo 'nada' ) | cat      # prints 'nada'
( ( echo -n foo | grep . ) || echo 'nada' ) | cat  # prints 'foo'

To the use-case that looks like this. In the below, there is no -pl-pl so it falls back and returns hunspell-pl

apt-cache search hunspell-pl-pl | grep . || apt-cache search hunspell-pl

Or,

apt-cache search hunspell-en-US | grep . || apt-cache search hunspell-en

There is an -en-US so it returns hunspell-en-us.

See also,

• If you're looking for the bash-specific solution check out this answer by roaima

Answer 5

You could define a:

has_output() {
  LC_ALL=C awk '1;END{exit!NR}'
}

And then:

if cmd | has_output; then
  echo cmd did produce some output
fi

Some awk implementations may choke on NUL characters in the input.

Contrary to grep '^', the above would be guaranteed to work on an input that doesn't end in a newline character, but would add the missing newline.

To avoid that and to be portable to systems where awk chokes on NUL, you could use perl instead:

has_output() {
  perl -pe '}{exit!$.'
}

With perl, you could also define a variant that handles arbitrary files more gracefully:

has_output() {
  PERLIO=:unix perl -pe 'BEGIN{$/=\65536} END{exit!$.}'
}

That bounds the memory usage (like for files that don't have newline characters like big sparse files).

You could also create variants like:

has_at_least_one_non_empty_line() {
  LC_ALL=C awk '$0 != "" {n++};1; END{exit!n}'
}

or:

has_at_least_one_non_blank_line() {
  awk 'NF {n++};1; END{exit!n}'
}

(beware the definition of blank varies between awk implementations, some where it's limited to space and tab, some where it also includes ASCII vertical spacing characters like CR or FF, some where it considers the locale's blanks)

Ideally, on Linux, you'd want to use the splice() system call to maximize performance. I don't know of a command that would expose it but you could always use python's ctypes:

has_output() {
  python -c 'if 1:
    from ctypes import *
    import sys
    l = CDLL("libc.so.6")
    ret = 1
    while l.splice(0,0,1,0,65536,0) > 0:
      ret = 0
    sys.exit(ret)'
}

(note that either has_output's stdin or stdout (or both) has to be a pipe for splice() to work).

Answer 6

I would suggest to use very basic builtin functions of the shell:

ck_command() { [ -n $("$@") ] ; }

Here is the simplest test case:

ck_command echo 1 ; echo $?

ck_command echo ; echo $?

Then you could easily use it with the || construct you are used to:

ck_command command_1 || ck_command command_2

This simple function will work as you would like with your apt_cache behaviour whichever the number of arguments would be.