1

I'm trying to grep through some logs at the url below to look for a specific username. However I'm getting no results, it just runs without stopping.

grepfor="username"
urls=("https://tgstation13.org/parsed-logs/terry/data/logs/2019/01")
while [ ${#urls[@]} -ne 0 ]
do
    content="$(curl -s "$url[0]")"
    echo "$content" | grep "$grepfor"
    delete=($urls[0])
    add=(`echo "$content" | grep -Po '(?<=href=")[^"]*'`)
    urls=( "${urls[@]/$delete}" "{$add[@]}" )
done
Kusalananda
  • 333,661
Shardj
  • 140
  • Don't correct code in the question. It invalidates the answers. – Kusalananda Feb 07 '19 at 19:57
  • I've figured out my next fuckup, the links I'm adding are relative, i need to prepend urls[0] to the start of each. Also the url is missing a / at the end and I'm not following redirects – Shardj Feb 07 '19 at 20:00

1 Answers1

2

Use "${urls[0]}" for the first element of the urls array, not $urls[0].

To delete the first element of urls and to add the add array to the end, use

urls=( "${urls[@]:1}" "${add[@]}" )

Always quote every expansion, even ${#urls[@]}.

I haven't looked too closely at your curls and greps, but use

printf '%s\n' "$content"

if you want to be sure that you preserve backslashes in the data.

Related:

Kusalananda
  • 333,661
  • Thanks, I'll implement those changes. However even if I change $grepfor to a string which is on the first url I can't get it to grep out that line – Shardj Feb 07 '19 at 19:51
  • @Shard Well, the curl call previous to that line will look for an url ending in [0], so fix that first. Also $url is unset there (it should be "${urls[0]}"). – Kusalananda Feb 07 '19 at 19:52
  • Ahh thanks that was a big fuck up, I've implemented the changed code into the question. The code is now terminating near immediately. I assume new urls found from the first link aren't correctly being added to $urls – Shardj Feb 07 '19 at 19:56