3

I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC.

The below shell script will copy the files into TEST1 and TEST2 directory in machineA..

#!/bin/bash
set -e

readonly TEST1=/data01/test1
readonly TEST2=/data02/test2
readonly SERVER_LOCATION=(machineB machineC)
readonly FILE_LOCATION=/data/snapshot

dir1=$(ssh -o "StrictHostKeyChecking no" david@${SERVER_LOCATION[0]} ls -dt1 "$FILE_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${SERVER_LOCATION[1]} ls -dt1 "$FILE_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    rm -rf $TEST1/*
    rm -rf $TEST2/*
    for el in $test1_partition
    do
        scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$el"_200003_5.data $TEST1/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$el"_200003_5.data $TEST1/.
    done
    for sl in $test2_partition
    do    
        scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$sl"_200003_5.data $TEST2/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$sl"_200003_5.data $TEST2/.
    done
fi

Is there a way to run process parallelly in the loop of a bash script

Currently it copies the file from machineB and machineC into machineA TEST1 directory first, and if it is done, then only it will go and copy the files from machineB and machineC into machineA TEST2 directory.. Is there any way I transfer the files both in TEST1 and TEST2 directory simultaneously?

I am running Ubuntu 12.04

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
arsenal
  • 3,093

3 Answers3

2

In addition to sending them to the background, use the wait built in to wait for all background processes to finish before continuing.

for el in $test1_partition
do
    (scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$el"_200003_5.data $TEST1/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$el"_200003_5.data $TEST1/.) &
    WAITPID="$WAITPID $!"
done
for sl in $test2_partition
do    
    (scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$sl"_200003_5.data $TEST2/. || scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$sl"_200003_5.data $TEST2/.) &
    WAITPID="$WAITPID $!"
done
wait $WAITPID
echo "All files done copying."
bahamat
  • 39,666
  • 4
  • 75
  • 104
  • Thanks for the suggestion. What will happen in case of any errors? Will I be able to identify those if I go with your suggestion? In general if there is any error, I would like to show them on the console and stopped executing the script so that's why I was using set -e – arsenal Feb 16 '14 at 04:39
  • Handle the error in the subshell. I.e., whatever you put after the || is what handles the error. – bahamat Feb 16 '14 at 05:31
  • Suppose for whatever reason if the files are not present in both the machine, then what will happen in that case, it will throw an error right? – arsenal Feb 16 '14 at 05:34
  • Yes, but you'll still need to handle the error in the subshell. It won't be surfaced to the parent process. – bahamat Feb 16 '14 at 05:42
  • hmmm.. Can you provide an example how would I do this in this example? I would like to clearly indicate to parent process about this error.. – arsenal Feb 16 '14 at 05:48
1

You can do:

scp david@${SERVER_LOCATION[0]}:$dir1/pp_monthly_9800_"$el"_200003_5.data $TEST1/ ||
scp david@${SERVER_LOCATION[1]}:$dir2/pp_monthly_9800_"$el"_200003_5.data $TEST1/ &

The & at the end will detach the command, returning control to the script.

If you put this in a loop, this will run all the bodies in parallel. To wait for all these background jobs to finish, use

wait
Gilles 'SO- stop being evil'
  • 829,060
Abdul
  • 321
  • Thanks. In this, after running the script, control will come out of the script and script will end successfully but the file transfer will happen in the background? Is there any way to wait for the script to finish but the file transfer should happened in the background? I also need to see whether there was any exception happened during the file transfer process or not.. If the control comes out of the script, then I am assuming shell script will always show success status. – arsenal Feb 15 '14 at 15:13
  • This isn't going to work: different files are being copied to the two servers ($test1_partition vs $test2_partition). Also you're going to have a hard time detecting errors. – Gilles 'SO- stop being evil' Feb 15 '14 at 23:21
1

You should take a look to pssh project, there's a pscp command that run parallel scp

Gilles Quénot
  • 33,867