How to call a service URL from bash shell script in parallel?

Question

I have a service which I am calling from another application. Below is my service URL which I am calling -

http://www.betaservice.domain.host.com/web/hasChanged?ver=0

I need to do some load test on my above service URL in multithreaded way instead of calling sequentially one by one.

Is there any way from bash shell script, I can put a load on my above service URL by calling it in multithreaded way? I can have 60-70 threads calling above URL in parallel very fast if possible?

Answer 1

I wouldn't call it multithreading as such but you could simply launch 70 jobs in the background:

for i in {1..70}; do 
   wget http://www.betaservice.domain.host.com/web/hasChanged?ver=0 2>/dev/null &
done

That will result in 70 wget processes running at once. You can also do something more sophisticated like this little script:

#!/usr/bin/env bash

## The time (in minutes) the script will run for. Change 10
## to whatever you want.
end=$(date -d "10 minutes" +%s);

## Run until the desired time has passed.
while [ $(date +%s) -lt "$end" ]; do 
    ## Launch a new wget process if there are
    ## less than 70 running. This assumes there
    ## are no other active wget processes.
    if [ $(pgrep -c wget) -lt 70 ]; then
        wget http://www.betaservice.domain.host.com/web/hasChanged?ver=0 2>/dev/null &
    fi
done

Answer 2

Try ab, you get a nice statistic too:

ab -n 10000 -c 70 http://www.betaservice.domain.host.com/web/hasChanged?ver=0

This call will do 10000 requests with concurrency of 70 parallel queries.

Answer 3

You could try installing GNU parallel. You could get some GNU parallel examples from here.

Testing

I installed gnu-parallel from source in my machine and I could get it to work.

You could install it from source from here. I have a redhat system and so I downloaded the fedora package and then ran the .configure, make and make install to get the parallel installed in my system.

Now, after the successful installation, I created a directory checking and ran the below command.

seq 10 | parallel -n0  wget http://www.betaservice.domain.host.com/web/hasChanged?ver=0

As expected the above command downloaded me 10 copies of the web page. You could set the number that you wish with seq.

For more information on how to run the same command in parallel, you could verify the examples provided by gnu-parallel from here. From the example page,

If you want to run the same command with the same arguments 10 times in parallel you can do:

seq 10 | parallel -n0 my_command my_args

EDIT

Now, to take advantage of the parallel execution, you could use the command as,

 seq 70 | parallel -j70 wget http://www.betaservice.domain.host.com/web/hasChanged?ver=0

The -j option is something that could specify the total jobs that can be executed in parallel based on the total CPU cores.