deduplicate mostly negated awk

Question

do not repeat yourself!, lol, ok, I'm trying to see if I can consolidate my 2 github workflows awk sources into 1. I tried a few things, but for some reason I kept ending up with weird column stuff. retrospectively I might have been assigning not comparing variables. I decided not to polute the question with broken code.

awk is being run as git diff-tree -r --no-commit-id --name-status HEAD | awk -f .github/files/changed.awk, no options. The output of git diff-tree looks something like this

( cat << EOF
A       .config/plantuml/theme.puml
M       .github/workflows/main.yml
M       .github/workflows/plantuml.yml
M       README.md
A       app/gradle.lockfile
A       authn/gradle.lockfile
A       docs/README.md
A       docs/domain-model/README.md
A       docs/domain-model/user.md
A       docs/domain-model/user.puml
A       docs/domain-model/foo.puml
M       settings.gradle.kts
D       user.puml
D       foo.puml
EOF
) | awk -f file.awk

These are what I came up with.

$2 ~ /\.puml$/ &&
$2 !~ /(theme|config)\.puml$/ &&
$1 !~ /^D$/ {
  changed[++n]=$2
}
END {
  for ( i in changed ) result = result " " changed[i]
  printf "::set-output name=changed-files::%s\n", result
  printf "::warning::changed-files %s\n", result
}

outputs

::set-output name=changed-files:: docs/domain-model/user.puml docs/domain-model/foo.puml
::warning::changed-files docs/domain-model/user.puml docs/domain-model/foo.puml

$2 ~ /\.puml$/ &&
$2 !~ /(theme|config)\.puml$/ &&
$1 ~ /^D$/ {
  split($2, fn, ".")
  changed[++n]=fn[1] ".svg"
}
END {
  for ( i in changed ) result = result " " changed[i]
  printf "::set-output name=removed-files::%s\n", result
  printf "::warning::removed-files %s\n", result
}

outputs

::set-output name=removed-files:: user.svg foo.svg
::warning::removed-files  user.svg foo.svg

Here's the current workflow file if that helps. note: the change to awk -f is being made in a currently broken branch, which I'll delete and squash when finished.

the goal is to get output similar to the following. I don't care about the order of lines, and the whitespace within the list only matters so much as it can be passed to git add, git rm and plantuml on the shell.

# adding empty lines between just for readability here, comments do nott matter either
::set-output name=changed-files:: docs/domain-model/user.puml docs/domain-model/foo.puml
::warning::changed-files docs/domain-model/user.puml docs/domain-model/foo.puml
::set-output name=removed-files:: user.svg foo.svg
::warning::removed-files  user.svg foo.svg

How would you deduplicate and improve it?

Related: https://unix.stackexchange.com/a/638805/116858 – Kusalananda Mar 16 '21 at 20:04 — Kusalananda, Mar 16 '21 at 20:04

αғsнιη · Accepted Answer · 2021-03-17T07:23:23.827

with some improvements and deduplicate two parts into one.

$2 ~ /\.puml$/ && $2 !~ /(theme|config)\.puml$/ \
{ if($1 !~ /^D$/) { result1= (result1==""? "" : result1 " ") $2 } else {
      split($2, fn, ".")
      result2=(result2==""? "" : result2 " ") fn[1] ".svg"
  }
}
END {
  printf "::set-output name=files::%s\n::warning::%s\n", result1, result1
  printf "::set-output name=files::%s\n::warning::%s\n", result2, result2
}

score 0 · Answer 2 · answered Mar 18 '21 at 20:38

So, you have one input (or file, the result of git)

A       .config/plantuml/theme.puml
M       .github/workflows/main.yml
M       .github/workflows/plantuml.yml
M       README.md
A       app/gradle.lockfile
A       authn/gradle.lockfile
A       docs/README.md
A       docs/domain-model/README.md
A       docs/domain-model/user.md
A       docs/domain-model/user.puml
A       docs/domain-model/foo.puml
M       settings.gradle.kts
D       user.puml
D       foo.puml

and two awk scripts that do similar (but distinct) actions on this input.

I assume that your goal is to reduce (as possible) the actions of this scripts.

The first two lines are exactly equal:

$2 ~ /\.puml$/ &&
$2 !~ /(theme|config)\.puml$/

After that, one script takes action on $2 being not D (I assume that means deleted). The other script takes action on the complement, $2 being D. That could be coded as:

{ if ( $1 ~ /^[D]$/ ) then { print "Deleted" } else { print "Changed" } }

Or, if you want a more granular selection:

{ if ( $1 ~ /^[D]$/  ) then { print "Deleted" } 
  if ( $1 ~ /^[MA]$/ ) then { print "Changed" } }

Storing each file in one array is not really needed as what you need is an space separated list of files.

That could be done inside the loop for each input line (less memory, faster):

{ if ( $1 ~ /^[D]$/ ) then { deleted = deleted " " $2 } 
  if ( $1 ~ /^[A]$/ ) then { changed = changed " " $2 } }

Of course, the match could be an exact string instead of a regex (faster) and you also need to extract the filename without extension in the changed case:

{ if ( $1 == "D" ) then { deleted = deleted " " $2 } 
  if ( $1 == "A" ) then { sub( /\.[^.]+$/, "", $2 ) ; 
                          changed = changed " " $2 ".svg"
                        }
}

A shell script to test all ideas could be:

#!/bin/bash
a='

A       .config/plantuml/theme.puml
M       .github/workflows/main.yml
M       .github/workflows/plantuml.yml
M       README.md
A       app/gradle.lockfile
A       authn/gradle.lockfile
A       docs/README.md
A       docs/domain-model/README.md
A       docs/domain-model/user.md
A       docs/domain-model/user.puml
A       docs/domain-model/foo.puml
M       settings.gradle.kts
D       user.puml
D       foo.puml
'
printf '%s\n' "$a" | awk '
      $2  ~ /.puml$/ &&
      $2 !~ /(theme|config).puml$/ {

             if ( $1 == "D" ) { deleted = deleted " " $2 }
             if ( $1 == "A" ) { sub(/.[^.]+$/, "", $2);
                                changed = changed " " $2 ".svg"
             }
      }
      END {
             printf "::set-output name=removed-files::%s\n", deleted
             printf "::warning::removed-files %s\n", deleted
             printf "::set-output name=changed-files::%s\n", changed
             printf "::warning::changed-files %s\n", changed
      }
    '

deduplicate mostly negated awk

2 Answers2