How to extract folders, split filenames in key and value(s) and save it in an associative array?

Question

I have following script which is supposed to solve the problem from the title, but obviously it isn't working to assign the values to the keys. Is the reason an accidental error or is there a substantial mistake in the script?

The foldernames are names of gem-files like gem-file-foo-1.2.3

The key is supposed to be gem-file-foo in this example and the value(s) the version number 1.2.3 or a string of multiple version numbers if there are multiple versions of the same gem.

It doesn't output any keys with echo "${!my_gems[@]}" ... why not?

#!/bin/bash
directory=$GEM_HOME/gems
declare -A my_gems
get_gemset_versions () {
last_key=""
values=""
FIND_RESULTS=$(find $directory -maxdepth 1 -type d -regextype posix-extended -regex "^${directory}/[a-zA-Z0-9]+([-_]?[a-zA-Z0-9]+)*-[0-9]{1,3}(.[0-9]{1,3}){,3}$")
printf "%s\n" $FIND_RESULTS | sort | 
  while read -r line; do
    line=${line##*/}
KEY="${line%-*}"

VALUE="${line##*-}"

    if [[ $last_key -eq "" ]]; then
        last_key=$KEY
    fi

    if [[ $last_key -eq $KEY ]]; then
        values="$values ${VALUE}"
    else
        values="${VALUE}"
        last_key=$KEY
    fi

    my_gems[$KEY]=$values
done

echo "${!my_gems[@]}"


}
get_gemset_versions

Also, the logic with $last_key and $key to summerize equal gem-packages seems to be faulty. This is not necessarily part of the question, but it would be nice if you would point out if I am applying some faulty logics here.

Thanks

Answer 1

You have:

printf "%s\n" $FIND_RESULTS | sort | 
  while read -r line; do
    ...
    done
echo "${!my_gems[@]}"

where, regardless of the indentation, the echo is outside the pipeline. Bash runs all parts of a pipeline in subshells by default, so the assignments within the while loop aren't visible after the pipeline ends. Shellcheck.net also warns about that:

Line 32:
        my_gems[$KEY]=$values
        ^-- SC2030: Modification of my_gems is local (to subshell caused by pipeline).

Sadly it doesn't give workarounds.

In Bash, you can either enable the lastpipe option, or replace the pipe with process substitution:

shopt -s lastpipe
echo test | while read line; do
    out=$line
  done
echo "out=$out"

or

while read line; do
  out=$line
done < <(echo test)
echo "out=$out"

(lastpipe probably won't work if you try it in an interactive shell, as it's tied to job control not being enabled.)

See: Why is my variable local in one 'while read' loop, but not in another seemingly similar loop?

---

In any case, this seems a bit odd:

FIND_RESULTS=$(find ...)
printf "%s\n" $FIND_RESULTS 

find outputs filenames separated by newlines, which is fine as long as you know no filenames contain any. But here, the round-trip through the variable and the word splitting from the unquoted expansion also splits any filenames with spaces.

You could just run find ... | while ... directly. Or while ...; done < <(find...).

Also, note that you almost always want to use while IFS= read -r line; do, to prevent read from breaking leading and tailing whitespace. Well, I hope your filenames don't contain those either, but in any case.

I can't find a good reference question right now, but that's specific to IFS containing whitespace. Other leading and trailing separators are not removed with a read to just one field. E.g. IFS=": " read -r foo <<< "::foobar " leaves foo with the literal ::foobar. The colons are kept, but the trailing spaces gone.