How to use output of multiple commands in last execution command while chaining commands?

Question

I have two commands, as follows:

This one gives repo names:

az acr repository list -n registry -o tsv

output looks like:

name1
name2
...

This one gives digest codes for one repo:

az acr manifest list-metadata -r ${REGISTRY} --name ${REPO} --query "[?tags[0]==null].digest" -o tsv

output looks like:

digest1
digest2
...

I want to output both repo names and digest codes.

Tried:

az acr repository list -n registry -o tsv \ 
| xargs -I% az acr manifest list-metadata -r ${REGISTRY} --n % --query "[?tags[0]==null].digest" -o tsv \
| xargs -I% echo "%: %"

Actual output:

digest_code: digest_code

Expected output:

repo_name: digestcode

Please provide some example output of these az commands, and the corresponding final output you expect. — muru, Sep 09 '22 at 09:01
@muru I have added output to both commands. Can you please check and help? — Python coder, Sep 09 '22 at 09:05
When you say “--n %”, do you mean “- - n ⁠ ⁠ %”, or do you mean “--n name1” (and “--n name2”)? — G-Man Says 'Reinstate Monica', Sep 09 '22 at 22:55
@G-ManSays'ReinstateMonica' I have updated my question. --n % is wrong, it is --name ${REPO} which will be --name name1. I am happy to see the new solution! — Python coder, Sep 10 '22 at 15:12
I *guess,* from reverse-engineering your attempted solution and the accepted answer, that you want n₁ + n₂ + … lines of output. You show *one* line of expected output. Please try to ask clearer questions. — G-Man Says 'Reinstate Monica', Sep 10 '22 at 23:32

Stéphane Chazelas · Accepted Answer · 2022-09-09T09:52:29.420

You'd need something like:

export REPO
az acr repository list -n registry -o tsv |
  while IFS= read -r REPO; do
    az acr manifest list-metadata -r "$REGISTRY" --n "$REPO" --query '[?tags[0]==null].digest' -o tsv |
      awk '{print ENVIRON["REPO"]": "$0}'
  done

Calling awk to prefix the output of each manifest command with the corresponding repo name.

Or if you need to run other commands on each repo/digest pair:

az acr repository list -n registry -o tsv |
  while IFS= read -r repo; do
    az acr manifest list-metadata -r "$REGISTRY" --n "$repo" --query '[?tags[0]==null].digest' -o tsv |
      while IFS= read -r digest; do
        other-cmd --repo "$repo" --digest "$digest"
      done
  done

With zsh, you could also do:

for repo ( ${(f)"$(az acr repository list -n registry -o tsv)"} ) {
  digests=( ${(f)"$(az acr manifest list-metadata -r $REGISTRY --n $repo --query '[?tags[0]==null].digest' -o tsv)"})
  print -rC1 -- $repo': '$^digests
}

for repo ( ${(f)"$(az acr repository list -n registry -o tsv)"} )
  for digest ( ${(f)"$(az acr manifest list-metadata -r $REGISTRY --n $repo --query '[?tags[0]==null].digest' -o tsv)"})
    other-cmd --repo $repo --digest $digest

In a Makefile, that'd look like:

target:
    az acr repository list -n registry -o tsv | \
      while IFS= read -r repo; do \
        az acr manifest list-metadata -r "$$REGISTRY" --n "$$repo" --query '[?tags[0]==null].digest' -o tsv | \
          while IFS= read -r digest; do \
            other-cmd --repo "$$repo" --digest "$$digest"; \
          done; \
      done

While that's on several lines in the Makefile, those lines are joined together, trailing \s removed and the $$s changed to $s before passing the result to sh -c, hence the need to add a few ;s to separate commands in that inline shell script.

You may want to put the code in a script instead to make it cleaner.

I am trying to run inside makefile, Is there anything I need to change in syntax to access $REPO variable? — Python coder, Sep 09 '22 at 09:37
@Pythoncoder, in Makefile, shell code has to be on one line and $'s (all $ even the one used in awk's code) must be escaped as $$. While that's perfectly doable, you'd be better of putting that code in a script and calling your script in the Makefile. — Stéphane Chazelas, Sep 09 '22 at 09:40
Thanks, it worked! If I need to run chain another az instead of awk which uses repo, digest, what changes need to make in this command? — Python coder, Sep 09 '22 at 09:46
See edit. export REPO is needed for the variable to passed in the environment of awk where it is retrieved with ENVIRON["REPO"]. — Stéphane Chazelas, Sep 09 '22 at 09:53
Thanks you so much, Stéphane. For some reason the command without environment variable works fast. I really appreciate your work and contribution to this community. — Python coder, Sep 09 '22 at 10:00

score 1 · Answer 2 · answered Sep 10 '22 at 23:28

I guess, from reverse-engineering your attempted solution and the other answer, that you want n₁ + n₂ + … lines of output, like

blue: sky
blue: berry
blue: jay
red: stop sign
red: cardinal
red: tomato
red: fire
green: grass
green: beans

Stéphane Chazelas’s answer is good, especially if you have no idea what characters might be in the repository names. However, it will not work on very old versions of Awk.

This very similar answer uses sed rather than Awk. sed is somewhat less resource-intensive and may run more quickly.

az acr repository list -n registry -o tsv |
  while IFS= read -r REPO; do
    az acr manifest list-metadata -r "$REGISTRY" --name "$REPO" \
                    --query "[?tags[0]==null].digest" -o tsv |
        sed "s/^/$REPO: /"
  done

(export not needed) This should behave the same as Stéphane’s answer unless the repository names might contain / or &. The / is a commonly-used, but arbitrary, delimiter. If you are sure that the repository names don’t contain & or, for example, |, then you can make it safer by changing the sed command to

        sed "s|^|$REPO: |"

If the repository names might contain &, that can be handled with a little more difficulty.

Note that the output will be potentially ambiguous / confusing if any of the repository names or digest codes might contain : .

If none of the repository names or digest codes might contain space, you can format the output as a nice table by piping it into column -t.

By the way, see ${variable_name} doesn’t mean what you think it does ….

How to use output of multiple commands in last execution command while chaining commands?

2 Answers2