For loop and reading a text file

Question

Why doesn't this for loop work like I expect it to?

Here is my script file (I am running zsh on a MAC computer with Ventura OS 13.2.1)

#!/bin/zsh
for user in "$(cat $1)";
do
        website="myWebSite.org/@/"
        url=$website$user
        echo "here is the url: $url"
done

here is my data file, it has four user's names, one on each line:

cat userList.txt 
user_1
user_2
user_3
user_4

I expected (want) the output to be:

myWebSite.org/@/user_1
myWebSite.org/@/user_2
myWebSite.org/@/user_3
myWebSite.org/@/user_4

Instead, here is the output I get:

./findusers userList.txt 
here is the url: myWebSite.org/@/user_1
user_2
user_3
user_4

I've googled for hours and cannot find anything even close to this type of problem. It's almost like the shell is operating on some file I don't see, I don't get why the echo command inside the for loop only executes once. Is this some version issue with stale files on my system? Any help appreciated, coding in zsh is not my day job, thanks!

Answer 1

You need to split that command substitution.

for user in $(<$1)

With $(...) unquoted, splits on characters of $IFS: space, tab, newline and nul by default¹. Here using the $(<file) Korn-like operator rather than $(cat -- $1) as an optimisation.

To split on newline (aka linefeed) only, either do the same but after IFS=$'\n' or use the f parameter expansion flag (short for ps[\n]):

for user in ${(f)"$(<$1)"}

Note the quotes to prevent IFS-splitting, and then the f flag to split on newline.

You could also use a while read loop:

while IFS= read -ru3 user; do
  ...
done 3< $1

One difference from the previous approaches is that it won't skip empty lines.

It will also skip characters after the last newline if any but those are not allowed in text files.

It avoids storing the whole file in memory but on the other hand means the file is going to be read one byte at a time as each read needs to make sure it does not read past the newline character that delimits the line.

With:

for user in "${(f@)$(<$1)}"

Or:

IFS=$'\n\n'
for user in $(<$1)

Empty lines are preserved except for trailing ones as command substitution strips all trailing newline characters.

To read all lines into an array, also considering empty lines and the non-line made of the bytes after the last newline if any and loop over it, that becomes quite awkward, you could use a helper function:

lines() {
  local ret
  reply=( "${(@f)$(cat -- "$@"; ret=$?; echo .; exit $ret)}" )
  ret=$?
  reply[-1]=( ${reply[-1]%.} )
  return $ret
}
lines myfile &&
  for line in "$reply[@]"; do
    something with "$line"
  done

Also note that echo should be avoided to output arbitrary data (though in the case of zsh, you can actually use echo -E - $data), better to use printf '%s\n' "$data" as in any other shell or print -r -- "$data" as in the Korn shell.

---

^{¹ note that contrary to other POSIX-like shells such as bash, zsh by default doesn't have that misfeature whereby the result is further subject to globbing, so you don't need set -o noglob there as you would in sh/bash/ksh for instance.}

Answer 2

You say (in comments) that you want to take each line in your input file, prepend a URL, and use it in calls to curl.

The arguably best way of doing this is to compose a configuration file for curl with several url lines on the form

url = http://some/url

This file is then passed to a single invocation of curl.

To do this:

curl --config <( sed 's|^|url = http://example.com/|' file )

Would you want to save the output from accessing each URL, maybe to some file called line.out (where line is the line read from your file), you just need to insert an output statement for each URL.

The following is using GNU sed:

curl --config <( sed 's|.*|url = http://example.com/&\noutput = &.out|' file )

Or, using awk instead:

curl --config <( awk '{ printf "url = http://example.com/%s\noutput = %s.out\n", $0, $0 }' file )

Note that these two last commands assume that we know that the lines in the input file contain simple words. If the strings contain absolute or relative pathnames, or patterns specific to the curl utility, these may have to be sanitised first.

Answer 3

I also had problems with the for loop reading files

And for that I use a while loop:

FILE="some_items.txt"
LINES=$(cat $FILE|wc -l)
INDEX=0
while [ $INDEX -lt $LINES ]
do
   LN=$((INDEX + 1))
   ITEM="$(cat $FILE|head -n $LN|tail -n 1)"
   echo "current item is $ITEM"
   INDEX=$((INDEX + 1))
done

This works on zsh, bash and sh