Spaces inserted in concatenated strings despite use of braces

Question

I used the following syntax to read in a file name and modify its extensions to generate the name of a matched data file. As an example, I have ABC_1.fastq.Blockxy, XYZ_1.fastq.Block34 and I want to generate ABC_2.fastq.Block12, XYZ_2.fastq.Block34 as my new file name:

for infile in *_1.fastq.Block*
do
    #base=$(basename ${infile} _1.fastq.**)
    IFS='_'
    read -ra ADDR <<< $infile
    base=${ADDR[0]}
    IFS='.'
    read -ra ADDR <<< ${ADDR[1]}
    second_file="${base}_2.fastq.${ADDR[2]}"
    echo $second_file
done

When executed, this script prints e.g.

ABC_2 fastq Block12
XYZ_2 fastq Block34

i.e. spaces between fastq and Block12. Why am I getting spaces rather than a period between these three strings when I concatenate? I thought that using braces for variable names should have eliminated this problem.

Yes, that works in this case. However, for more complicated examples I do want to be able to use pieces of the split string to create new strings. I still don't understand why I'm getting spaces and no periods between the string elements with second_file. — Max, Aug 31 '22 at 22:51
I take it it's because IFS is still set to ".", which is making echo treat the variable as multiple arguments rather than one. You can unset IFS prior to the echo or put in quotes, echo "$second_file". — frabjous, Aug 31 '22 at 23:02

Stéphane Chazelas · Answer 1 · 2022-09-01T06:15:57.013

In

echo $second_file

Since you forgot to quote the parameter expansion, it's subject to split+glob, so with IFS=., ABC_2.fastq.Block12 is first split into, ABC_2, fastq and Block12 and each word subject to globbing, with no effect here since none of the words contain glob operators.

So 3 arguments are passed to echo which it prints space separated.

To print the contents of a variable followed by a newline character, you need:

printf '%s\n' "$var"

For more details, see:

Now, a few more comments on your code:

since bash doesn't have the equivalent of zsh's (N) glob qualifier or ksh93's ~(N) glob operator, before using a glob in a for loop (at least), you need to set the nullglob option:
```
shopt -s nullglob
for infile in *_1.fastq.Block*; do...
```
If you don't and there's no matching file, you'll loop over a literal *_1.fastq.Block*
You can set IFS for read only with: IFS=_ read -ra ADDR <<< "$infile" (see also the quotes around $infile which are needed in older versions of bash). That way, $IFS is only changed while read runs¹ and it's restored to its previous value after read returns.
IFS=. read -ra <<< "$var" is a poor method for splitting. First is only works for single line $vars which is not necessarily the case of file names, and also it's quite inefficient. That involves either storing the contents of $var into a tempfile or feed it via a pipe depending on the version of bash and/or the size of $var and then reading it one byte at a time until a newline is found.

Here, you could use the split+glob operator instead:
```
IFS=:; set -o noglob
addr=( $infile )
```
(or addr=( $infile'' ) to not ignore a trailing :.)

Or switch to better shells with proper splitting operators.

Another approach here would be to do:
```
regex='^(.*)_1\.fastq\.(Block.*)$'
if [[ $infile =~ $regex ]]; then
  outfile=${BASH_REMATCH[1]}_2.fastq.${BASH_REMATCH[2]}
  ...
```
With the caveat that regex matching only works with valid text, which again is not a guarantee for file names.

Here, you could also use standard sh parameter expansion operators:
```
new_file=${infile%_1.fastq.Block*}_2.fastq.Block${infile##*_1.fastq.Block}
```
Or the ksh93-style:
```
new_file=${infile/_1.fastq.Block/_2.fastq.Block}
```
(note the variations in behaviour among all those approaches if _1.fastq.Block occurs more than once in the file name).

^{¹ Though beware that if a trap is handled whilst read is running, the code in that trap will have the modified $IFS}

Spaces inserted in concatenated strings despite use of braces

1 Answers1