Append not working well in bash

Question

I noticed that the operator ">>" doesn't work well in my script and I don't know why. I have a script like this:

for file in $(ls folder)`
do
  echo $file >> text.txt
done

Into folder I got 91 elements, but only the first 87 elements were inserted into file.txt. I can't figure out what is wrong with this code, can anybody help me to understand, please?

EDIT

The script I wrote up there is very simplified, but I understood that it don't give a clear picture of the situation. So, here more details:

Into my folder I got 91 csv files that contains each one two columns: name and value. For every file I need to control if this value is greater than 2.500 and if is not 0.000. If one of this 2 condition is true, I append the file name and its value to a txt file that contains my discarded files, otherwise I append the file name and its value in a csv file that contains my chosen files. The code I use to control the value works well, but when I use >> to append the results in the txt or csv file, the last four results aren't appended and I can't understand why.

for file in $(ls folder)
do
     value=$(cat path/$file | awk -F, '{print $2}')
     discard=$(awk -v num1="$value" 'BEGIN { if (num1 > 2.500) print 1; else if (num1 == 0.000) print 0; else print 2 }')
 if [[ $discard -eq 1 || $discard -eq 0 ]]
 then
      echo ""$file" has value="$value"" >> path/discard.txt
      rm path/"$file"
 else
      echo ""$file",$value" >> path/selected.csv
      rm path/"$file"
 fi

done

This is a more complete version of my script.

EDIT 2

I corrected my script to fix the issue you've find in it. Still same problem. To be more clear, the files in my folder are csv files automatically generated by a program and they contains only a row with 2 columns: an ID and a float value. They all are very similar, so the problem isn't in there, also because I can see from terminal the script recognizes them and processes them well. I still don't know why append doesn't put the last 4 four files into the txt file.

for file in folder/*
do
     value=$(cat "$file" | awk -F, '{print $2}')
     discard=$(awk -v num1="$value" 'BEGIN { if (num1 > 2.500) print 1; else if (num1 == 0.000) print 0; else print 2 }')
 if [[ "$discard" -eq 1 || "$discard" -eq 0 ]]
 then
      echo ""$file" has value="$value"" >> path/discard.txt
      rm -- "$file"
 else
      echo ""$file",$value" >> path/selected.csv
      rm -- "$file"
 fi

done

Reducing the number of files, it works perfectly by the way.

Answer 1

There are two main issues here:

1. Never do for file in $(ls). This is also known as bash pitfall number 1. First, it is fragile and, as you have seen, breaks on even slightly strange file names. What's worse, it isn't needed at all, it just makes the code more fragile and more complicated than it needs to be. You can just do for file in *.

2. Always quote your variables. This is doubly important when dealing with file names that can contain any character except \0. and /. If you don't quote a variable, it will be expanded by the shell, so if it contains a space, the shell will see two values instead of one. For example:

To illustrate, consider a directory with these two files:

$ touch 'a file2' 'file1 *'
$ ls -l
total 0
-rw-r--r-- 1 terdon terdon 0 Sep 22 17:39 'file1 *'
-rw-r--r-- 1 terdon terdon 0 Sep 22 17:39 'a file2'

Now, if we try your original code, we get:

$ for file in $(ls); do echo $file; done
a
file2
file1
a file2
file1 a file2 file1 *

This is because, since you are not quoting, the shell receives a, file2, and then file1 and then *. So it treats each of them separately and at the end runs echo * which prints everything in the directory again. However, if you do it the right way, it all works:

$ for file in *; do echo "$file"; done
a file2
file1 *

Here's what should be (test it first, I cannot see your data) a working version of your script:

for file in folder/*
do
     value=$(awk -F, '{print $2}' "$file")
     discard=$(awk -v num1="$value" 'BEGIN { if (num1 > 2.500) print 1; else if (num1 == 0.000) print 0; else print 2 }')
 if [[ $discard -eq 1 || $discard -eq 0 ]]
 then
      printf '"%s" has value="%s"' "$file" "$value" >> path/discard.txt
      rm -- "$file"
 else
      printf '"%s",%s' "$file" "$value" >> path/selected.csv
      rm -- "$file"
 fi

done

Answer 2

I just tried using a CSV file instead of a TXT file for the discarded elements. Now the script works well.

for file in folder/*
do
     value=$(cat "$file" | awk -F, '{print $2}')
     discard=$(awk -v num1="$value" 'BEGIN { if (num1 > 2.500) print 1; else if (num1 == 0.000) print 0; else print 2 }')
 if [[ "$discard" -eq 1 || "$discard" -eq 0 ]]
 then
      echo ""$file" has value="$value"" >> path/discard.csv
      rm -- "$file"
 else
      echo ""$file",$value" >> path/selected.csv
      rm -- "$file"
 fi

done

Thank you to all for the help and the advices!