For loop to iterate over directory tree extracting results from files of the same name

Question

I have a series of directories, all with list.txt in the same format, and I wish to put the results into a single file. I am looking to write a script that will iteratively move through each directory tree, extract a specific column from the list.txt file without surrounding text using the grep/awk pipeline below and write the outputs of each to the same file.

    grep 'bar[0-9]' file.txt | awk '{print $1}'

I have attempted the following but I am not sure exactly where my loops in the script are going wrong.

#!/bin/bash
##Extract ligands from toplist and concatenate to file
for i in /home/ubuntu/Project/working/library_*/Results/list.txt
do
    grep 'bar[0-9]' i | awk '{print $1}' | cat ../output.txt i
done

The directory tree is as follows:

.
├── library_1-200
│   ├── Results
│   │   ├── complex
│   │   ├── sorted.txt
│   │   └── list.txt
│   ├── files
│   │   ├── output
│   │   └── txt
│   └── summary.txt
├── library_201-400
│   ├── Results
│   │   ├── complex
│   │   ├── sorted.txt
│   │   └── list.txt
│   ├── files
│   │   ├── output
│   │   └── txt
│   └── summary.txt
├── library_401-600
│   ├── Results
│   │   ├── complex
│   │   ├── sorted.txt
│   │   └── list.txt
│   ├── files
│   │   ├── output
│   │   └── txt
│   └── summary.txt
└── library_601-800
    ├── Results
    │   ├── complex
    │   ├── sorted.txt
    │   └── list.txt
    ├── files
    │   ├── output
    │   └── txt
    └── summary.txt

Sample of list.txt, where I just want the Name values put into output.txt

Name    Score
bar65    -7.8 
bar74    -7.5 
bar14    -7.5 
bar43    -7.4 
bar94    -7.4 
bar16    -7.4 
bar12    -7.3 
bar25    -7.3 
bar65    -7.3 
bar76    -7.3 
bar24    -7.3 
bar13    -7.3 
bar58    -7.2 
bar68    -7.2 
bar28    -7.2 

Solution was to put "$i" where I previously had just i and to modify to | cat >> ../output.txt

Answer 1

You are using i, instead of this use $i in grep command.

And you said that you want to put all of them into single file then the last command should be:

cat >> /home/ubuntu/Project/working/output.txt

Or just:

>> /home/ubuntu/Project/working/output.txt

Answer 2

Apart from correcting some small typos in your original code (using "$i" in place of i and redirecting the output to the output file rather than trying to output its contents), if you don't have many thousands of these list.txt files:

awk '/^bar[0-9]/ { print $1 }' /home/ubuntu/Project/working/library_*/Results/list.txt >output.txt

This is using awk to extract the first column of all lines that start with the string bar followed by a digit. It does this for all files matching the patten /home/ubuntu/Project/working/library_*/Results/list.txt. The extracted data is redirected to output.txt.

The loop becomes necessary when the filename globbing pattern /home/ubuntu/Project/working/library_*/Results/list.txt expands to too many names:

for pathname in /home/ubuntu/Project/working/library_*/Results/list.txt; do
    awk '/^bar/ { print $1 }' "$pathname"
done >output.txt

Note that it's more efficient to redirect the output of the loop than of each individual awk call. Also note that awk easily does the job of grep to detect the wanted lines and that cat is not needed.

If you need the first column from all lines except the first (as in your example data), you can change the condition in the awk code from /^bar[0-9]/ to FNR > 1.