4

I am trying to do the following:

  1. Loop through a list of files
  2. Edit each file and output a "trimmed" file
  3. Horizontally join each trimmed file in turn to a pre-made master file

I have steps 1 & 2 working, but can't get 3 to work.

Example

MASTER FILE:

Col1   Col2
A       1
B       1
C       2

TRIMMED FILE for file S1:

S1.Col3    S1.Col4
0            1
1            1
1            1

OUTPUT I WANT after S1 & S2 have been joined to MASTER:

Col1   Col2   S1.Col3   S1.Col4   S2.Col3   S2.Col4
A       1       0         1          0        1
B       1       1         1          1        0
C       2       1         1          0        0

After each loop, I've tried running something like:

paste MASTER.txt S1.txt > MASTER.txt

However, instead of the above, I get a file with only the last S file's data. I tried running this without outputting and outside of the loop, everything works fine. Thanks in advance for any answers.

For completion, here is a very simplified breakdown of my for loop.

for FILE in FILELIST
do
    cut -f4,6 $FILE > ${FILE}_trimmed.txt
    paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt
done
Gaius Augustus
  • 414
  • hint: the > MASTER.txt clobbers the file before paste can get to its contents. – iruvar Dec 22 '15 at 04:54
  • I see what you're saying, I think. I initially tried paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt but that obviously doesn't work if I'm looping through files. I just end up with the final S file joined to the MASTER. So, still unsure how to get the result I'm looking for. I'm used to working in R, where I could possibly store as a temp variable and then output the file at the end. Unsure if that is plausible here or a good way of doing this... – Gaius Augustus Dec 22 '15 at 05:24
  • 1
    simplistically, change paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt to paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt && mv MASTER2.txt MASTER.txt – iruvar Dec 22 '15 at 05:28
  • I see. That's perfect. I've created an answer, but will delete if you would prefer to explain it more thoroughly. – Gaius Augustus Dec 22 '15 at 05:39
  • No problems, +1 as well – iruvar Dec 22 '15 at 05:43
  • Related: http://unix.stackexchange.com/q/189473/38906 – cuonglm Dec 22 '15 at 06:56

1 Answers1

2

Thank you to 1_CR , who answered this in the comments.

The > MASTER.txt is truncating the file before the paste actually completes the horizontal join. The answer is to output to a temporary file (e.g. MASTER2.txt) and then rename it (with the mv command) to the proper file name.

My Old Code:

paste MASTER.txt ${FILE}_trimmed.txt > MASTER.txt

Fixed Code:

paste MASTER.txt ${FILE}_trimmed.txt > MASTER2.txt && mv MASTER2.txt MASTER.txt
Gaius Augustus
  • 414