Is there something wrong with my script or is Bash much slower than Python?

Question

I was testing the speed of Bash and Python by running a loop 1 billion times.

$ cat python.py
#!/bin/python
# python v3.5
i=0;
while i<=1000000000:
    i=i+1;

Bash code:

$ cat bash2.sh
#!/bin/bash
# bash v4.3
i=0
while [[ $i -le 1000000000 ]]
do
let i++
done

Using the time command I found out that the Python code takes just 48 seconds to finish while the Bash code took over 1 hour before I killed the script.

Why is this so? I expected that Bash would be faster. Is there something wrong with my script or is Bash really much slower with this script?

Answer 1

Shell loops are slow and bash's are the slowest. Shells aren't meant to do heavy work in loops. Shells are meant to launch a few external, optimized processes on batches of data.

---

Anyway, I was curious how shell loops compare so I made a little benchmark:

#!/bin/bash

export IT=$((10**6))

echo POSIX:
for sh in dash bash ksh zsh; do
    TIMEFORMAT="%RR %UU %SS $sh"
    time $sh -c 'i=0; while [ "$IT" -gt "$i" ]; do i=$((i+1)); done'
done


echo C-LIKE:
for sh in bash ksh zsh; do
    TIMEFORMAT="%RR %UU %SS $sh"
    time $sh -c 'for ((i=0;i<IT;i++)); do :; done'
done

G=$((10**9))
TIMEFORMAT="%RR %UU %SS 1000*C"
echo 'int main(){ int i,sum; for(i=0;i<IT;i++) sum+=i; printf("%d\n", sum); return 0; }' |
   gcc -include stdio.h -O3 -x c -DIT=$G - 
time ./a.out

( Details:

• CPU: Intel(R) Core(TM) i5 CPU M 430 @ 2.27GHz
• ksh: version sh (AT&T Research) 93u+ 2012-08-01
• bash: GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
• zsh: zsh 5.2 (x86_64-unknown-linux-gnu)
• dash: 0.5.7-4ubuntu1

)

The (abbreviated) results (time per iteration) are:

POSIX:
5.8 µs  dash
8.5 µs ksh
14.6 µs zsh
22.6 µs bash

C-LIKE:
2.7 µs ksh
5.8 µs zsh
11.7 µs bash

C:
0.4 ns C

From the results:

If you want a slightly faster shell loop, then if you have the [[ syntax and you want a fast shell loop, you're in an advanced shell and you have the C-like for loop too. Use the C like for loop, then. They can be about 2 times as fast as while [-loops in the same shell.

• ksh has the fastest for ( loop at about 2.7µs per iteration
• dash has the fastest while [ loop at about 5.8µs per iteration

C for loops can be 3-4 decimal orders of magnitude faster. (I heard the Torvalds love C).

The optimized C for loop is 56500 times faster than bash's while [ loop (the slowest shell loop) and 6750 times faster than ksh's for ( loop (the fastest shell loop).

---

Again, the slowness of shells shouldn't matter much though, because the typical pattern with shells is to offload to a few processes of external, optimized programs.

With this pattern, shells often make it much easier to write scripts with performance superior to python scripts (last time I checked, creating process pipelines in python was rather clumsy).

Another thing to consider is startup time.

time python3 -c ' '

takes 30 to 40 ms on my PC whereas shells take around 3ms. If you launch a lot of scripts, this quickly adds up and you can do very very much in the extra 27-37 ms that python takes just to start. Small scripts can be finished several times over in that time frame.

(NodeJs is probably the worst scripting runtime in this department as it takes about 100ms just to start (even though once it has started, you'd be hard pressed to find a better performer among scripting languages)).

Answer 2

This is a known bug in bash; see the man page and search for "BUGS":

BUGS
       It's too big and too slow.

;)

---

For an excellent primer on the conceptual differences between shell scripting and other programming languages, I highly recommend reading:

• Why is using a shell loop to process text considered bad practice?

The most pertinent excerpts:

Shells are a higher level language. One may say it's not even a language. They're before all command line interpreters. The job is done by those commands you run and the shell is only meant to orchestrate them.

...

IOW, in shells, especially to process text, you invoke as few utilities as possible and have them cooperate to the task, not run thousands of tools in sequence waiting for each one to start, run, clean up before running the next one.

...

As said earlier, running one command has a cost. A huge cost if that command is not builtin, but even if they are builtin, the cost is big.

And shells have not been designed to run like that, they have no pretension to being performant programming languages. They are not, they're just command line interpreters. So, little optimisation has been done on this front.

---

Don't use big loops in shell scripting.

Answer 3

I did a bit of testing, and on my system ran the following--none made the order of magnitude speedup that would be needed to be competitive, but you can make it faster:

Test 1: 18.233s

#!/bin/bash
i=0
while [[ $i -le 4000000 ]]
do
    let i++
done

test2: 20.45s

#!/bin/bash
i=0
while [[ $i -le 4000000 ]]
do 
    i=$(($i+1))
done

test3: 17.64s

#!/bin/bash
i=0
while [[ $i -le 4000000 ]]; do let i++; done

test4: 26.69s

#!/bin/bash
i=0
while [ $i -le 4000000 ]; do let i++; done

test5: 12.79s

#!/bin/bash
export LC_ALL=C

for ((i=0; i != 4000000; i++)) { 
:
}

The important part in this last one is the export LC_ALL=C. I've found that many bash operations end up significantly faster if this is used, in particular any regex function. It also shows an undocumented for syntax to use the {} and the : as a no-op.

Answer 4

A shell is efficient if you use it for what it has been designed for (though efficiency is rarely what you look for in a shell).

A shell is a command-line interpreter, it is designed to run commands and have them cooperate to a task.

If you want to count to 1000000000, you invoke a (one) command to count, like seq, bc, awk or python/perl... Running 1000000000 [[...]] commands and 1000000000 let commands is bound to be terribly inefficient, especially with bash which is the slowest shell of all.

In that regard, a shell will be a lot faster:

$ time sh -c 'seq 100000000' > /dev/null
sh -c 'seq 100000000' > /dev/null  0.77s user 0.03s system 99% cpu 0.805 total
$ time python -c 'i=0
> while i <= 100000000: i=i+1'
python -c 'i=0 while i <= 100000000: i=i+1'  12.12s user 0.00s system 99% cpu 12.127 total

Though of course, most of the job is done by the commands that the shell invokes, as it should be.

Now, you could of course do the same with python:

python -c '
import os
os.dup2(os.open("/dev/null", os.O_WRONLY), 1);
os.execlp("seq", "seq", "100000000")'

But that's not really how you'd do things in python as python is primarily a programming language, not a command line interpreter.

Note that you could do:

python -c 'import os; os.system("seq 100000000 > /dev/null")'

But, python would actually be calling a shell to interpret that command line!

Answer 5

Answer: Bash is much slower than Python.

One little example is in blog post Performance of several languages.

Answer 6

Nothing is wrong (except your expectations) as python is really rather fast for non-compiled language, see https://wiki.python.org/moin/PythonSpeed

Answer 7

Aside the comments, you could optimize the code a little, e.g.

#!/bin/bash
for (( i = 0; i <= 1000000000; i++ ))
do
: # null command
done

This code should take a bit less time.

But obviously not fast enough to be actually usable.

Answer 8

I've noticed a dramatic difference in bash from the use of logically equivalent "while" and "until" expressions:

time (i=0 ; while ((i<900000)) ; do  i=$((i+1)) ; done )

real    0m5.339s
user    0m5.324s
sys 0m0.000s

time (i=0 ; until ((i=900000)) ; do  i=$((i+1)) ; done )

real    0m0.000s
user    0m0.000s
sys 0m0.000s

Not that it really bears tremendous relevance to the question, other than that perhaps sometimes small differences make a big difference, even though we'd expect they'd be equivalent.