Shell Script Performance, Part 3

Real cases/scenarios are some time hard to come by and definitely you want to be prepared when you are in such a situation. It is good that you can create such scenario yourself. In my Part 2 regarding shell script performance, I asked my audiences to come up with the fastest way to create 1000 files. of course you do not want to wait for minutes or hours for just 1000 files. What if you want to test on 10,000 or even 100,000 files ?

With that many files in your directory, you can practice your shell scripting by changing the prefix, suffix, extension, padding with zeros, ..., the amount of things you can script is endless.

In Part 1, we understand that process creation is a very expensive task and we should avoid running too many commands, especially within a loop. So I am not going to use the traditional approach like this one:
count=1; while [ $count -le 100 ]; do touch file-$count.txt; count=`expr $count + 1`; done

Below shows you various ways to 'skin a cat':

1. create1.sh - use 'seq' to generate 1 to 1000 number to be used in a for loop and each loop create a new file
2. create2.sh - use Bash shell built-in capability to do brace expansion
3. create3.sh - use 'seq -f' to generate the file name and supply that to touch to create all the files. [Thanks to comment by pjz]
4. create.tcl - a Tcl implementation
5. create.py - a Python implementation

And the run time:

1. create1.sh - 2m36.452s
2. create2.sh - 0m2.371s
3. create3.sh - 0m2.075s
4. create.tcl - 0m7.831s
5. create.py - 0m2.403s

As you can see, the less commands you used the shortest the run time. In this case, pjz won!. In create2.sh, I introduced Bash shell brace expansion which is a very useful feature in doing command expansion. See this blog in how I create 48 'devices' (c0d0t0s0, ...) in a single command line.

create1.sh

$ cat create1.sh
#! /bin/sh

if [ $# -ne 3 ]; then
        echo "Usage: $0 <prefix> <start#> <end#>"
        exit 1
fi

prefix=$1
start=$2
end=$3

for i in `seq -w $2 $3`
do
        touch $prefix-$i.txt
done

$ time ./create1.sh file 1 1000

real    2m36.452s
user    0m4.260s
sys     0m24.288s

$ rm -f file-*

create2.sh

$ cat create2.sh
#! /bin/sh

if [ $# -ne 1 ]; then
        echo "Usage: $0 <prefix>"
        exit 1
fi

prefix=$1
n="0,1,2,3,4,5,6,7,8,9"
eval touch $prefix-0{$n}{$n}{$n}.txt
mv $prefix-0000.txt $prefix-1000.txt

$ time ./create2.sh file

real    0m2.371s
user    0m0.031s
sys     0m0.856s

$ rm -f file-*

create3.sh

$ cat ./create3.sh
#! /bin/sh


if [ $# -ne 3 ]; then
        echo "Usage: $0 <prefix> <start#> <end#>"
        exit 1
fi
prefix=$1
start=$2
end=$3

touch `seq -f "$prefix-%04g.txt" $start $end`

$ time ./create3.sh file 1 1000

real    0m2.075s
user    0m0.108s
sys     0m0.746s

$ rm -f file-*

create.tcl

$ cat create.tcl
#! /cygdrive/c/Tcl8.4.19/bin/tclsh

if { $argc != 3 } {
        puts stderr "Usage: $argv0 <prefix> <start#> <end#>"
        exit 1
}

set prefix [lindex $argv 0]
set start  [lindex $argv 1]
set end    [lindex $argv 2]
set pad    [string length $end]
set format "%s-%0${pad}d.txt"

for { set i $start } { $i <= $end } { incr i } {
        set fname [format $format $prefix $i]
        set fp [open $fname w]
        close $fp
}

$ time ./create.tcl file 1 1000

real    0m7.831s
user    0m0.000s
sys     0m0.031s

$ rm -f file-*

create.py

$ cat create.py
#! /usr/bin/python

import sys

if len(sys.argv) != 4:
        sys.stderr.write('Usage: %s <prefix> <start#> <end#>' % (sys.argv[0]))
        exit(1)

prefix = sys.argv[1]
start = int(sys.argv[2])
end = int(sys.argv[3])
pad = len(sys.argv[3])
format = '%s-%0' + str(pad) + 'd.txt'
for i in xrange(start, end+1):
        fname = format % (prefix, i)
        fp = open(fname, 'w')
        fp.close()

$ time ./create.py file 1 1000

real    0m2.403s
user    0m0.031s
sys     0m0.951s

$ rm -f file-*