For loop - Bash isn't reading my variables

Question

I am trying to make a little bash script, that calls a command on each of my files in have in a file, found with the find command.

I want to be able to keep track of where the script stopped (it tends to crash) so i can take back from there. I managed to read my file, get the lines,... But currently i'm stuck at the for loop. I want to do a C style for loop, starting at the the last line i stopped at, increment by one, and do it as long as i'm smaller than the number of lines. I got this :

#!/bin/bash
LINES=$(wc -l < file.txt)
LASTLINE=$(grep -P '### Stop marker ###' file.txt | wc -l)
STARTFROM=$(($LINES - $LASTLINE))
for ((i = STARTFROM; i < LINES; i++));
do
    echo "we are processing file number $i"
    file=sed -n $i'p' file.txt
    ocrmypdf [some stuff] -input $file
done

Here is an excerpt of what my file.txt looks like inside

./input_folder/hard_blurry.pdf
./input_folder/l_ordre_malte.pdf
### Stop marker ###
./input_folder/single_page.pdf
./input_folder/very_hard.pdf

When i run this i get... nothing. Bash doesn't enter the loop at all. I tried setting ints directly, and it worked, which tells me that the vars are bein read as string.

I tried all these ways to write my var :

for ((i = STARTFROM; i < LINES; i++));
for ((i = $((STARTFROM)); i < $((LINES)); i++));
for ((i = $(echo STARTFROM); i < $(echo LINES); i++));

and nothing works. I'm suprised that no erros are thrown as well. My Os is ubuntu 20.0.4

Its content is the path to the files i want to work with.

Any ideas ? Thanks

Answer 1

LASTLINE=$(grep -P '### Stop marker ###' file.txt | wc -l)

This will tell you how many lines match that pattern, but not where they are. If there's one marker in the file, this returns 1. You'd need to use something like grep -n (--line-number) to get the line numbers.

file=sed -n $i'p' file.txt

That should probably be file=$(sed ...), i.e. with a command substitution to capture the output of sed. But, if you do that in a loop, you read the whole file for each iteration of the loop, a silly waste, and will take ages if the file is long.

That's the sort of thing the question Why is using a shell loop to process text considered bad practice? that was linked here earlier refers to. Note that the bad practice is processing, modifying text. Running commands based on some data in a file using the shell is just fine; the shell exists to run commands.

So, just loop over the file once and detect the stop marker in the shell:

#!/bin/bash
i=0
while IFS= read -r line; do 
    if [[ $line == '### Stop marker ###' ]]; then
        break;
    fi
    i=$((i + 1))
    echo "line $i, do some stuff with '$line'"
done < file.txt

([[ .. ]] is a Ksh-ism. It could be replaced with case in a POSIX shell.)

Or, have some external text processing tool deal with the stop marker and have the shell just run the commands:

#!/bin/sh
i=0
< file.txt sed -n -e '/### Stop marker ###/q' -e p |
while IFS= read -r line; do
    i=$((i + 1))
    echo "line $i, do some stuff with '$line'"
done

If you really wanted to do a for (i = 0; i < end; i++) style loop, you could read the whole file into an array first, but unless you need random access to the lines, that's completely unnecessary. Streaming through the file is far more natural.

Answer 2

you can treid like this seq to get range of var:

#!/bin/bash
LINES=1
LASTLINE=10
for i in $(seq $LINES $LASTLINE )
do
 echo $i
done

output:

1
2
3
4
5
6
7
8
9
10