How to pass multiple variable to a for loop as a argument in a bash script?

Question

I am new to linux and i am writing a bash script... In the script, I have 2 variables(inside the varibles contents are there). I am trying to pass these 2 varibales in a same for loop and perform some actions.

But when i pass 2 varibles in a same for loop, am getting error. In the below code. to make it ease I am passing two paramerters but in actual it will vary. I am getting output of these varibales from a command

Below is my code:

FILES="2019_06/
2019_07/"
FILESIZE="100
200"

for file in `cat $FILES`; for filesize in `cat $FILESIZE`
 do
 do
 if [ -n "$file" ] && if [ -n "$filesize" ] 
  then
echo  $file

   curl -i -XPOST "http://localhost:8086/write?db=S3check&precision=s" --data-binary 'ecmwftrack,bucketpath=ecmwf-archive/'$file' size=$filesize'

fi
 done
done

Could any one pls help me to pass 2 arguments in the for loop at the same time.

Argument should pass like in the for loop

FILES=2019_06 FILESIZE=100
FILES=2019_07 FILESIZE=200

Below is the error message

Kindly help!

Below is my output

Below is my curl coomand

echo  curl -i -XPOST "http://localhost:xxxx/write?db=S3check&precision=s" --data-binary 'ecmwftrack,bucketpath=ecmwf-archive/'$files' size='$filesizes''


#!/bin/bash -x

# You said variables get their values from commands, so here 
# are stand-ins for those commands:
command_to_get_files(){
  aws s3 ls "s3://ui-dl-weather-ecmwf-ireland/ecmwf-archive/"| awk '{print $2}'  >>"$FILES"
}

command_to_get_filesizes(){
 for file in `cat $FILES`
 do
 if [ -n "$file" ]
  then
  # echo $file
   s3cmd du -r s3://ui-dl-weather-ecmwf-ireland/ecmwf-archive/$file | awk '{print $1}'>>"$FILESIZE"

 fi
 done
}

# I assume the values returned from the commands are whitespace delimited
# Therefore it is easy to use command substitution and transform the output
# of the commands into arrays:

files=( $(command_to_get_files) )

filesizes=( $(command_to_get_filesizes) )

Answer 1

Here's how I'd write it:

#!/bin/bash

while read file filesize; do
  if [[ -n "$file" && -n "$filesize" ]]; then
    # This printf is a stand-in for what your really want to do
    printf "FILES=%s FILESIZE=%s\n" "$file" "$filesize"
    # I commented out the curl invocation because I don't understand it
    #curl -i -XPOST "http://localhost:8086/write?db=S3check&precision=s" --data-binary "ecmwftrack,bucketpath=ecmwf-archive/$file" size="$filesize"
  fi
done <<'EOF'
2019_06 100
2019_07 200
EOF

It reads the two variables from the heredoc that is supplied at the end.

Answer 2

Putting multiple values in one variable, using it unquoted and relying on word splitting to separate the values again is not a good practice. Use arrays. You can retrieve values from two or more arrays "in parallel" by using the same index; this should solve your problem.

Prepare two arrays of equal size and loop like this:

#!/bin/bash

file=( "2019_06/" "2019_07/" )
filesize=( "100" "200" )

for ((i=0; i<"${#file[@]}"; i++)) do
   echo "${file[i]}" "size=${filesize[i]}"
done

Replace echo … with the desired (and adjusted) if … curl … block.

---

If FILES and FILESIZE are variables through which your script receives data and you cannot change this and you must rely on word splitting, then create arrays from the variables:

file=( $FILES )
filesize=( $FILESIZE )

The variables are not quoted, word splitting (and filename generation) occurs at this stage.

Answer 3

#!/bin/bash

# You said variables get their values from commands, so here 
# are stand-ins for those commands:

command_to_get_files(){
  echo "2019_06/
2019_07/"
}

command_to_get_filesizes(){
  echo "100
200"
}

# I assume the values returned from the commands are whitespace delimited
# Therefore it is easy to use command substitution and transform the output
# of the commands into arrays:

files=( $(command_to_get_files) )

filesizes=( $(command_to_get_filesizes) )

# Repeat this pattern for how ever many parameters you have

# Hat tip to Kamil for idea of using arrays and C-style for loop

for ((i=0; i<"${#files[@]}"; i++)) do
    # This printf is a stand-in for what your really want to do
    #printf "FILES=%s FILESIZE=%s\n" "${files[$i]%?}" "${filesizes[$i]}"
    echo curl -i -XPOST "http://localhost:8086/write?db=S3check&precision=s" --data-binary "ecmwftrack,bucketpath=ecmwf-archive/${files[$i]%?} size=${filesizes[$i]}"
done

The percent sign question mark %? chops off the / that you didn't seem to want.

Put that in a file named myscript.sh then run it:

$ chmod +x myscript.sh
$ ./myscript.sh 
FILES=2019_06 FILESIZE=100
FILES=2019_07 FILESIZE=200

Update: Output with printf replaced by echo curl ...:

$ ./myscript.sh 
curl -i -XPOST http://localhost:8086/write?db=S3check&precision=s --data-binary ecmwftrack,bucketpath=ecmwf-archive/2019_06 size=100
curl -i -XPOST http://localhost:8086/write?db=S3check&precision=s --data-binary ecmwftrack,bucketpath=ecmwf-archive/2019_07 size=200

Answer 4

Often you will want todo S3 stuff in parallel:

FILES="2019_06/
2019_07/"
FILESIZE="100
200"

doit() {
   s3cmd du -r s3://ui-dl-weather-ecmwf-ireland/ecmwf-archive/"$1" |
     awk '{print $1}'>>"$2"
}
export -f doit

parallel doit {1} {2} ::: $FILES :::+ $FILESISZE