Saturday, March 05, 2011

Feel Like A Scripting Guy Again!

I have not been scripting for quite a while and feel like I don't quality as a "Scripting Guy in the Lion City" anymore. Luckily my colleage (now is ex-colleague, 'cos I will be starting my new job next week) approached me to develop script to check hardware error messages from the /var/adm/messages in Solaris. Now I am as happy as a sandboy, or I should say "As Happy As A Scripting Guy!"

Anyway, here is my last script in the company.

If this script is going to run periodically via cron, we cannot just 'grep' certain pattern from the file. What if the /var/adm/messages has not been rotated for days or even weeks, it will be pretty inefficient to simply scan through the entire file. BTW, I have seen /var/adm/messages as big as few GB. Also, problem fixed previously will also appear in /var/adm/messages if it is not rotated.

The only way is to 'grep' only the content from where it left off in the previous run. It is easy to keep track of the current file size and store this information somewhere for the next poll to consume. The challenge is to 'seek' to the previous location and output only the content between the two polls. However, I am not able to find any UNIX commands that allow me to 'cat' file based on byte range.

In order to accomplish this, some high level scripting languages are used. Perl is selected because it is available since Solaris 8. Below one-liner is able to 'cat' only the updated contents in /var/adm/messages.

POS=$prev_pos perl -e 'open(F,"/var/adm/messages");seek(F,$ENV{"POS"},0); while($line=<F>){print $line;}'

Who say shell script is slow and inefficient! Always challenge yourself for a better solution.

Here is the complete script doing all the checking

#! /usr/bin/ksh
#
# Check on all the faults
#       hardware, svm, vx, df, /var/adm/messages, ifconfig, psrinfo
# Exit 1 if there is an error
#



PATH=/usr/bin:/usr/sbin:/usr/platform/`arch -k`/sbin:/usr/sfw/bin:/usr/local/bin
export PATH
LD_LIBRARY_PATH=/usr/lib:usr/platform/`arch -k`/lib:/usr/sfw/lib:/usr/local/lib
export LD_LIBRARY_PATH


Update()
{
    # if you do not want output, simply comment out the 'echo' command
    # remember to leave the colon (no-ops) command there
    echo "$1,\c"
    :
}


RC=0


#
# prtdiag(1M) - display system diagnostic information
#
# Exit Status
# 0 No failures or errors are detected in the system.
# 1 Failures or errors are detected in the system.
# 2 An internal prtdiag error occurred, for example, out of memory.
#
prtdiag > /dev/null 2>&1
[ $? -ne 0 ] && ((RC+=1)) && Update prtdiag



#
# fmadm(1M) - fault management configuration tool
#
# Exit Status
# 0 Successful completion.
# 1 An error occurred. Errors include a failure to communicate with fmd or 
#   insufficient privileges to perform the requested operation.
# 2 Invalid command-line options were specified.
#
osver=`uname -r`
if [ "${osver#*.}" -ge 10 ]; then
    fmadm faulty > /dev/null 2>&1
    [ $? -ne 0 ] && ((RC+=1)) && Update fmadm
fi



#
# df(1M) - displays number of free disk blocks and free files
#
# Exit Status
# 0  Successful completion.
# >0 An error occurred.
#
df -l > /dev/null 2>&1
[ $? -ne 0 ] && ((RC+=1)) && Update df



#
# ifconfig(1M) - configure network interface parameters
#
# Interface flag:
# FAILED
#     The  interface  has  failed.  New  addresses  cannot  be
#     created  on this interface. If this interface is part of
#     an IP network multipathing group, a failover will  occur
#     to another interface in the group, if possible
#
ifconfig -a | grep FAILED > /dev/null 2>&1
[ $? -eq 0 ] && ((RC+=1)) && Update ifconfig



#
# check svm via metadb and metastat
#
pkginfo SUNWmdu > /dev/null 2>&1
if [ $? -eq 0 ]; then
    metaset=`metaset`
    metadb > /dev/null 2>&1
    if [ $? -eq 0 -a ! -z $metaset ]; then

        # output should not have these characters from metadb, see metadb -i man page
        # d - replica does not have an associated device ID.
        # r - replica does not have device relocation information
        # W - replica has device write errors
        # M - replica had problem with master blocks
        # D - replica had problem with data blocks
        # F - replica had format problems
        # S - replica is too small to hold current database
        # R - replica had device read errors
        # B - tagged data associated with the replica is not valid
        metadb -i | sed -n '2,$s/[1-9][0-9].*//p' | egrep '[drWMDFSRB]' > /dev/null 2>&1
        rc1=$?
        #
        metastat | awk '$1=="State:" && $2!="Okay" {exit(1)}'
        rc2=$?
        if [ $rc1 -ne 0 -o $rc2 -ne 0 ]; then
            ((RC+=1))
            Update svm
        fi
    fi
fi



#
# psrinfo(1M) - displays information about processors
# Exit Status
# 0  Successful completion.
# >0 An error occurred.
#
psrinfo > /dev/null 2>&1
rc1=$?
psrinfo | awk '$2!="on-line"{exit(1)}'
rc2=$?
if [ $rc1 -ne 0 -o $rc2 -ne 0 ]; then
    ((RC+=1))
    Update psrinfo
fi



#
# VX
#
pkginfo VRTSvxfs VRTSvxvm > /dev/null 2>&1
if [ $? -eq 0 ]; then
    vxdisk list 2>&1 | grep -i fail > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        ((RC+=1))
        Update vx
    fi
fi



#
# 'grep' from /var/adm/messages
# need to seek to the previous position, instead of scan through the whole file again
#
hidden=${0##*/}
hidden="/tmp/.${hidden%%.*}_var-adm-messages"
prev_pos=0
if [ -f $hidden ]; then
    prev_pos=`cat $hidden`
fi
# if previous position is less than current file size, likely file has been recycled
filesize=`ls -l /var/adm/messages | awk '{print $5}'`
if [ $prev_pos -gt $filesize ]; then
    prev_pos=0
fi
#
perl -e 'open(F,"/var/adm/messages");seek(F,'$prev_pos',0); while($line=<F>){print $line;}' | \
    egrep -i 'ECC error|PS[0-9] has FAILED|Link Down|reboot after panic|file system full|by NR list: I/O error|fmd.*SEVERITY: Critical' > /dev/null 2>&1
if [ $? -eq 0 ]; then
    ((RC+=1))
    Update /var/adm/messages
fi
ls -l /var/adm/messages | awk '{print $5}' > $hidden



if [ $RC -gt 0 ]; then
    exit 1
fi

Labels: , ,

Friday, July 16, 2010

One Million Files in A Directory Under Veritas File System

Recently my backup ran really really slow with a throughput of 50KBps. With such a low throughput, I was not able to finish it within the backup window. After some investigations, I realised that there is a directory comprises of close to 1 million files. It literally takes minutes to do a "ls -l".

Although file system allows you have that many files, it will be extremely inefficient for other downstream activities.

It will be a challenge to locate the problematic directory if you were to use shell script. Other scripting languages such as Perl will be more appropriate. With find2perl utility that comes with standard perl installation, you can get it to generate equilvant find command in perl

$ find2perl /usr/include -type d
#! /usr/bin/perl -w
    eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
        if 0; #$running_under_some_shell

use strict;
use File::Find ();

# Set the variable $File::Find::dont_use_nlink if you're using AFS,
# since AFS cheats.

# for the convenience of &wanted calls, including -eval statements:
use vars qw/*name *dir *prune/;
*name   = *File::Find::name;
*dir    = *File::Find::dir;
*prune  = *File::Find::prune;

sub wanted;



# Traverse desired filesystems
File::Find::find({wanted => \&wanted}, '/usr/include');
exit;


sub wanted {
    my ($dev,$ino,$mode,$nlink,$uid,$gid);

    (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) &&
    -d _
    && print("$name\n");
}
With this skeleton code, I am able to modify it to help to locate sub-directory with the most items
#! /usr/bin/perl


use strict;
use File::Find ();
use Cwd 'abs_path';


# Set the variable $File::Find::dont_use_nlink if you're using AFS,
# since AFS cheats.

# for the convenience of &wanted calls, including -eval statements:
use vars qw/*name *dir *prune/;
*name   = *File::Find::name;
*dir    = *File::Find::dir;
*prune  = *File::Find::prune;


my $max = 0;
my $maxpath;


sub wanted;


my $searchdir;
if ( $#ARGV == -1 ) {
        $searchdir=".";
} else {
        $searchdir=$ARGV[0];
}



# Traverse desired filesystems
File::Find::find({wanted => \&wanted}, $searchdir);
print $maxpath, ' ', $max, "\n";
exit;


sub wanted {
        my ($dev,$ino,$mode,$nlink,$uid,$gid);

        my $file;
        my $count=0;
        if ( (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) && -d _ )
{
                opendir(DIR, $_);
                while (defined($file=readdir(DIR))) {
                        ++$count;
                }
                closedir(DIR);
                if ( $count > $max ) {
                        $max = $count;
                        $maxpath = abs_path($_);
                }
        }
}
If I run on my Ubuntu 10.04 Netbook Edition /usr/share directory, it will tell me /usr/share/foomatic/db/source/printer has 3258 files (include . and ..)
$ ./files-in-directory-max.pl /usr/share
/usr/share/foomatic/db/source/printer 3258

So, what will be the maximum number of files in a directory under vxfs ? I found this using Google:
Recommended maximum number of files in a single Veritas File System (VxFS) directory is 100,000

Labels:

Thursday, March 25, 2010

File Uploading, A Perl+Python Approach

I created a download server some time ago and always wanted to come up with something similar for uploading.

My initial approach avoided the whole issue by running the download server from the opposite end, but this approach is not very flexible 'cos user requires Python to be installed in their desktop.

My second approach was to develop a CGI program (upload.cgi) based on the established Perl CGI.pm module. The below CGI script is launched via a running Apache web server. However, we have to depend on the existence of Apache web server.

#! /usr/bin/perl

use CGI;
use CGI::Carp qw ( fatalsToBrowser );
use File::Basename;

if ( $ENV{"REQUEST_METHOD"} eq "GET" ) {
        print "Content-type: text/html\n\n";
        print "<html><head><title>Upload</title></head><body>";
        printf("<form action=%s method=POST ENCTYPE=multipart/form-data>", $ENV{'SCRIPT_NAME'});
        print "<input type=file name=upfile> <input type=submit value=Upload></form>";
        print "</body></html>";
} else {
        my $safe_filename_characters = "a-zA-Z0-9_.-";
        my $upload_dir = "/export/software/upload";

        my $query = new CGI;
        my $fname = $query->param("upfile");


        $fname =~ s/\\/\//g;
        my ( $name, $path, $extension ) = fileparse ( $fname, '\..*' );
        $filename = $name . $extension;
        $filename =~ tr/ /_/;
        $filename =~ s/[^$safe_filename_characters]//g;


        my $upload_filehandle = $query->upload("upfile");

        open ( UPLOADFILE, ">$upload_dir/$filename" ) or die "$!";
        binmode UPLOADFILE;

        print "Content-type: text/html\n\n";
        print "<html><head><title>Upload</title></head><body>";

        while ( <$upload_filehandle> )
        {
        print "...";
         print UPLOADFILE;
        }

        close UPLOADFILE;

        printf("Your file is in /export/software/upload/%s (%s)", $filename, $fname);
        print "</body></html>";
}

My third approach was to combine Perl and Python in a single Python script. Python comes with a simple and workable CGIHTTPServer module that allows me to run a self-contained web server that can server CGI script. With the below script (uploadserver), it will dynamically create a random cgi-bin directory with the Perl upload.cgi script. An index.html is also dynamically created that can do auto-redirection to the cgi-bin/upload.cgi. With this script, I can launch the upload server anywhere and able to upload file via a web browser.

#! /usr/local/bin/python


import BaseHTTPServer
import CGIHTTPServer
import tempfile
import os
import random
import platform


#
# standalone CGI perl script for uploading.
# escaped all the backslashes
#
perlscript="""#! /usr/bin/perl


use CGI;
use CGI::Carp qw ( fatalsToBrowser );
use File::Basename;


if ( $ENV{"REQUEST_METHOD"} eq "GET" ) {
        print "Content-type: text/html\\n\\n";
        print "<html><head><title>Upload</title></head><body>";
        printf("<form action=%s method=POST ENCTYPE=multipart/form-data>", $ENV{'SCRIPT_NAME'});
        print "<input type=file name=upfile> <input type=submit value=Upload></form>";
        print "</body></html>";
} else {
        my $safe_filename_characters = "a-zA-Z0-9_.-";
        my $upload_dir = ".";
        my $query = new CGI;
        my $fname = $query->param("upfile");


        #
        # convert backslash to forward flash, remove un-safe characters in
filename
        #
        $fname =~ s/\\\\/\\//g;
        my ( $name, $path, $extension ) = fileparse ( $fname, '\\..*' );
        $filename = $name . $extension;
        $filename =~ tr/ /_/;
        $filename =~ s/[^$safe_filename_characters]//g;


        my $upload_filehandle = $query->upload("upfile");
        open ( UPLOADFILE, ">$upload_dir/$filename" ) or die "$!";
        binmode UPLOADFILE;
        while ( <$upload_filehandle> )
        {
                print UPLOADFILE;
        }
        close UPLOADFILE;

        print "Content-type: text/html\\n\\n";
        print "<html><head><title>Upload</title></head><body>";
        printf("Your file '%s' has been successfully uploaded", $filename,);
        print "</body></html>";
}
"""


if not os.path.isfile('index.html'):
        try:
                tempdir = tempfile.mkdtemp(dir='.')[2:]
                os.chmod(tempdir, 0755)
                uploadcgi = tempdir + '/upload.cgi'
                cgidir = '/' + uploadcgi

                # write index.html
                fp = open('index.html', 'w')
                refresh = '<html><head><meta http-equiv="REFRESH" content="0;url=/%s"></head><body></body></html>' % uploadcgi
                fp.write(refresh)
                fp.close()

                # create cgi script
                fp = open(uploadcgi, 'w')
                fp.write(perlscript)
                fp.close()
                os.chmod(uploadcgi, 0755)

                # change the internal settings
                CGIHTTPServer.CGIHTTPRequestHandler.cgi_directories = [cgidir]
                CGIHTTPServer.nobody = os.getuid()

                port = random.randint(50000,60000)
                url = "http://%s:%d/" % (platform.node(), port)
                httpd = BaseHTTPServer.HTTPServer(('',port), CGIHTTPServer.CGIHTTPRequestHandler)
                print "Ask user to visit this URL:\n\t%s" % url
                httpd.serve_forever()

        except:
                # clean up
                os.remove('index.html')
                os.remove(uploadcgi)
                os.rmdir(tempdir)

else:
                print 'Error. index.html exists'

# ./uploadserver
Ask user to visit this URL:
        http://user-PC:52233/

When you point your web server to the able URL, it will auto redirect.

Labels: , , ,

Thursday, February 25, 2010

Finding the Latest Modified Time In A Mount Point

User wanted to decommission a particular mount point but needed to ensure no files/directories was updated recently. So the question is how can we find out exactly the latest modified time in this mount point.

My initial approach was to do a couple of find /mount/point -mtime -... -ls to roughly locate the last modified time. I used the divide and conquer approach, eg, starts with 100 days, if no file return change to 200 days, else 50 days, ... This approach can only give me a rough estimate. If I can list the timestamp in the ISO 8601 format YYYYMMDDTHHMMSS, I can just simply sort it and the latest modified time will be the last record. I know find is not able to do that, but find2perl can convert a find command to Perl code. With Perl, I can modify it to output to the format that I want.

$ /usr/perl5/5.8.4/bin/find2perl /mount/point -ls
#! /usr/perl5/5.8.4/bin/perl -w
    eval 'exec /usr/perl5/5.8.4/bin/perl -S $0 ${1+"$@"}'
        if 0; #$running_under_some_shell

use strict;
use File::Find ();

# Set the variable $File::Find::dont_use_nlink if you're using AFS,
# since AFS cheats.

# for the convenience of &wanted calls, including -eval statements:
use vars qw/*name *dir *prune/;
*name   = *File::Find::name;
*dir    = *File::Find::dir;
*prune  = *File::Find::prune;

sub wanted;
sub ls ();


my @rwx = qw(--- --x -w- -wx r-- r-x rw- rwx);
my @moname = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);

my (%uid, %user);
while (my ($name, $pw, $uid) = getpwent) {
    $user{$uid} = $name unless exists $user{$uid};
}

my (%gid, %group);
while (my ($name, $pw, $gid) = getgrent) {
    $group{$gid} = $name unless exists $group{$gid};
}


# Traverse desired filesystems
File::Find::find({wanted => \&wanted}, '/mount/point');
exit;


sub wanted {
    my ($dev,$ino,$mode,$nlink,$uid,$gid);

    (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) &&
    ls;
}


sub sizemm {
    my $rdev = shift;
    sprintf("%3d, %3d", ($rdev >> 8) & 0xff, $rdev & 0xff);
}

sub ls () {
    my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
        $atime,$mtime,$ctime,$blksize,$blocks) = lstat(_);
    my $pname = $name;

    $blocks
        or $blocks = int(($size + 1023) / 1024);

    my $perms = $rwx[$mode & 7];
    $mode >>= 3;
    $perms = $rwx[$mode & 7] . $perms;
    $mode >>= 3;
    $perms = $rwx[$mode & 7] . $perms;
    substr($perms, 2, 1) =~ tr/-x/Ss/ if -u _;
    substr($perms, 5, 1) =~ tr/-x/Ss/ if -g _;
    substr($perms, 8, 1) =~ tr/-x/Tt/ if -k _;
    if    (-f _) { $perms = '-' . $perms; }
    elsif (-d _) { $perms = 'd' . $perms; }
    elsif (-l _) { $perms = 'l' . $perms; $pname .= ' ->>' . readlink($_); }
    elsif (-c _) { $perms = 'c' . $perms; $size = sizemm($rdev); }
    elsif (-b _) { $perms = 'b' . $perms; $size = sizemm($rdev); }
    elsif (-p _) { $perms = 'p' . $perms; }
    elsif (-S _) { $perms = 's' . $perms; }
    else         { $perms = '?' . $perms; }

    my $user = $user{$uid} || $uid;
    my $group = $group{$gid} || $gid;

    my ($sec,$min,$hour,$mday,$mon,$timeyear) = localtime($mtime);
    if (-M _ > 365.25 / 2) {
        $timeyear += 1900;
    } else {
        $timeyear = sprintf("%02d:%02d", $hour, $min);
    }

    printf "%5lu %4ld %-10s %3d %-8s %-8s %8s %s %2d %5s %s\n",
            $ino,
                 $blocks,
                      $perms,
                            $nlink,
                                $user,
                                     $group,
                                          $size,
                                              $moname[$mon],
                                                 $mday,
                                                     $timeyear,
                                                         $pname;
    1;
}

Just change the original printf to this one:

printf("%04d%02d%02dT%02d%02d%02d %d %s\n", $timeyear, $mon+1, $mday, $hour, $min, $sec, $mtime, $pname);

There is a lot of stuff not needed from the find2perl output. We can trim it down to just suit our need, which is to locate the latest modified time. My friend found this on the Internet that do exactly this task. It is very efficient because the script will keep track of the latest time stamp.

#!/bin/perl
use File::Find;
@ARGV = ('.') unless @ARGV;
my ($age, $name);
sub youngest {
    return if defined $age && $age > (stat($_))[9];
    $age = (stat(_))[9];
    $name = $File::Find::name;
}
find(\&youngest, @ARGV);
print "$name " . scalar(localtime($age)) . "\n";

Labels:

Saturday, September 26, 2009

Date Format, N Days Ago

In UNIX shell script, it is very difficult to manipulate date/time using standard UNIX commands. Modern programming languages such as Perl, Python, Tcl, ..., are designed to be a general programming/scripting language that can handle this type of task with ease.

Below shows shell functions that wrap around these general scripting languages to determine the date format that is N days ago.

ago_perl()
{
        # localtime return:
        #       Seconds past the minute
        #       Minutes past the hour
        #       Hours past midnight
        #       Day of the month
        #       Months past the start of the year
        #       Number of years since 1900
        #       Number of days since the start of the week (Sunday)
        #       Number of days since the start of the year
        #       Whether or not daylight savings is active
        days=${1:-1}
        perl -e "@t=localtime(time-86400*$days);\
                printf('%4d-%02d-%02d', @t[5]+1900, @t[4]+1, @t[3])"

}
ago_tcl()
{
        days=${1:-1}
        echo "puts [clock format [expr [clock seconds] - $days*86400] \
                -format {%Y-%m-%d}]" | tclsh
}
ago_python()
{
        days=${1:-1}
        python -c "import time, datetime
t=datetime.datetime.fromtimestamp(time.time()-86400*$days)
print '%4d-%02d-%02d' % (t.year,t.month,t.day)"
}

Here is how you can use any of these functions

$ ago=`ago_perl 45`

$ echo "45 days ago is $ago"
45 days ago is 2009-08-12

Labels: , , ,

Tuesday, April 07, 2009

Think Big

Suppose you have some command output consists of hexadecimal that require to be converted to decimal number, likely you will use bc to work it out
$ bc
ibase=16
ABCDEF
11259375
abcdef
syntax error on line 3, teletype
0089
137
^D

So if you want to make use of it in shell script, you need to do the following. Bear in mind that bc only accept uppercase hex number.

$ echo "ibase=16; ABCDEF" | bc
11259375

$ echo "ibase=16; abcdef" | bc
syntax error on line 1, teletype

$ echo "ibase=16; abcdef" | tr '[a-z]' '[A-Z]' | bc
syntax error on line 1, teletype
1123455

$ echo "abcdef" | tr '[a-z]' '[A-Z]' | xargs -I{} echo 'ibase=16;{}' | bc
11259375

This may work fine for small data set. However, if you need to loop through thousands of lines to do hex to dec convert, it becomes a performance problem. Below I am going to show you what are the alternatives.

Traditional shell script way. You can see how slow it is when we throw it with large data set.

$ cat hex.input
drive1 0089
drive2 0a2f
drive3 1FFE
drive4 980B
drive5 0011780c

$ cat hex2dec-sh.sh
#! /bin/sh


if [ $# -ne 1 ]; then
        echo "Usage: $0 <hex-input>"
        exit 1
fi


cat $1 | while read drive hex
do
        dec=`echo $hex | tr '[a-z]' '[A-Z]' | xargs -I{} echo "ibase=16; {}" | bc`
        echo "Drive=$drive, Hex=$hex, Dec=$dec"
done

$ time ./hex2dec-sh.sh hex.input
Drive=drive1, Hex=0089, Dec=137
Drive=drive2, Hex=0a2f, Dec=2607
Drive=drive3, Hex=1FFE, Dec=8190
Drive=drive4, Hex=980B, Dec=38923
Drive=drive5, Hex=0011780c, Dec=1144844

real    0m0.042s
user    0m0.011s
sys     0m0.065s

$ for i in `perl -e '$,=" "; print 1..1000'`
do
    cat hex.input
done > hex.big

$ wc -l hex.big
    5000 hex.big

$ time ./hex2dec-sh.sh hex.big > /dev/null

real    0m32.141s
user    0m9.131s
sys     0m56.458s

What if I have a few million lines to convert? You will need a lot of coffee breaks. The alternative is to use some high level scripting languages, such as Perl, Python, Tcl, ... In fact you can crank out your own function in AWK to do this kind of thing. Here I will show you a Perl one-liner and the AWK way.

Perl one-liner:

$ time perl -ne 'chomp();@l=split(/\s+/);print "Drive=",$l[0]," Hex=",$l[1]," Dec=",hex($l[1]),"\n"' < hex.input
Drive=drive1 Hex=0089 Dec=137
Drive=drive2 Hex=0a2f Dec=2607
Drive=drive3 Hex=1FFE Dec=8190
Drive=drive4 Hex=980B Dec=38923
Drive=drive5 Hex=0011780c Dec=1144844

real    0m0.008s
user    0m0.003s
sys     0m0.005s

$ time perl -ne 'chomp();@l=split(/\s+/);print "Drive=",$l[0]," Hex=",$l[1]," Dec=",hex($l[1]),"\n"' < hex.big  > /dev/null

real    0m0.044s
user    0m0.038s
sys     0m0.005s

AWK way:

$ cat hex2dec-awk.sh
#! /bin/sh


if [ $# -ne 1 ]; then
        echo "Usage: $0 <hex-input>"
        exit 1
fi


nawk '
function hex2dec(hex, h, i, factor, n, sum) {
        n = length(hex)
        factor = 1
        sum = 0
        for ( i=n ; i>0 ; --i ) {
                h = substr(hex, i, 1)
                if ( h == "a" || h == "A" ) { h=10 }
                if ( h == "b" || h == "B" ) { h=11 }
                if ( h == "b" || h == "B" ) { h=11 }
                if ( h == "c" || h == "C" ) { h=12 }
                if ( h == "d" || h == "D" ) { h=13 }
                if ( h == "e" || h == "E" ) { h=14 }
                if ( h == "f" || h == "F" ) { h=15 }
                sum += factor * h
                factor *= 16
        }
        return sum
}
{
        printf("Drive=%s, Hex=%s, Dec=%s\n", $1, $2, hex2dec($2))
}' $1

$ time ./hex2dec-awk.sh hex.input
Drive=drive1, Hex=0089, Dec=137
Drive=drive2, Hex=0a2f, Dec=2607
Drive=drive3, Hex=1FFE, Dec=8190
Drive=drive4, Hex=980B, Dec=38923
Drive=drive5, Hex=0011780c, Dec=1144844

real    0m0.008s
user    0m0.002s
sys     0m0.006s

$ time ./hex2dec-awk.sh hex.big > /dev/null 2>&1

real    0m0.100s
user    0m0.093s
sys     0m0.006s

For 5000 lines, you can reduce it from 32 seconds run time down to sub second. I am sure you can see the performance differences in the 3 implementations. Next time, think big! Bigger data size.

Labels: , , ,

Friday, September 05, 2008

Comparing Timestamp Between Two Files

If you need to compare modified time (mtime) of two files, you can use the shell construct:
if [ "myfile1" -nt "myfile2" ]
However, this is only available in Korn and Bash shells. Also it compares only the modified time. If you are used to programming in Bourne shell, you will have to rely on external utilities like Perl/Python/Tcl or others. If you need to compare creation time (ctime), access time (atime) or even compare local file against remote file, below shell function will definitely help you. The "epoch" function will try to return the epoch atime/ctime/mtime of the input file using either Perl, Python or Tcl. It is highly unlikely that any modern UNIX operating system will not have any of these utilities installed.
epoch()
{
 which perl > /dev/null 2>&1
 if [ $? -eq 0 ]; then
  epoch_in_perl $@
  return $?
 fi
 which python > /dev/null 2>&1
 if [ $? -eq 0 ]; then
  epoch_in_python $@
  return $?
 fi
 which tclsh > /dev/null 2>&1
 if [ $? -eq 0 ]; then
  epoch_in_tclsh $@
  return $?
 fi
}

epoch_in_perl()
{
 if [ $# -ne 2 ]; then
  echo "Usage: epoch atime|mtime|ctime file" 1>&2
  return 1
 fi
 if [ ! -e "$2" ]; then
  echo "Error. \"$2\" does not exist" 1>&2
  return 1
 fi
  
 case "$1" in
 atime)
  perl -e '@a=stat("'$2'");print $a[8]'
  ;;
 mtime)
  perl -e '@a=stat("'$2'");print $a[9]'
  ;;
 ctime)
  perl -e '@a=stat("'$2'");print $a[10]'
  ;;
 *)
  echo "Usage: epoch atime|mtime|ctime file" 1>&2
  return 1
  ;;
 esac
}


epoch_in_python()
{
 if [ $# -ne 2 ]; then
  echo "Usage: epoch atime|mtime|ctime file" 1>&2
  return 1
 fi
 if [ ! -e "$2" ]; then
  echo "Error. \"$2\" does not exist" 1>&2
  return 1
 fi
  
 case "$1" in
 atime)
  python -c 'import os.path;print int(os.path.getatime("'$2'"))'
  ;;
 mtime)
  python -c 'import os.path;print int(os.path.getmtime("'$2'"))'
  ;;
 ctime)
  python -c 'import os.path;print int(os.path.getctime("'$2'"))'
  ;;
 *)
  echo "Usage: epoch atime|mtime|ctime file" 1>&2
  return 1
  ;;
 esac
}

epoch_in_tclsh()
{
 if [ $# -ne 2 ]; then
  echo "Usage: epoch atime|mtime|ctime file" 1>&2
  return 1
 fi
 if [ ! -e "$2" ]; then
  echo "Error. \"$2\" does not exist" 1>&2
  return 1
 fi
  
 case "$1" in
 atime)
  echo "file stat \"$2\" a;puts [set a(atime)]" | tclsh
  ;;
 mtime)
  echo "file stat \"$2\" a;puts [set a(mtime)]" | tclsh
  ;;
 ctime)
  echo "file stat \"$2\" a;puts [set a(ctime)]" | tclsh
  ;;
 *)
  echo "Usage: epoch atime|mtime|ctime file" 1>&2
  return 1
  ;;
 esac
}

Here is how you can use this function:

$ epoch mtime myfile1
1201206566

$ epoch atime myfile2
1220593896

$ if [ `epoch ctime myfile1` -lt `epoch ctime myfile2 ]; then
    dosomething
  fi

$ if [ `epoch mtime myfile1` -gt `ssh chihung@remote epoch myfile1` ]; then
    dosomething
  fi

Labels: , , ,

Tuesday, July 29, 2008

How To Parse Timestamp

Whenever I am 'free' (am I always free? I am not going to tell you :-), I will visit UNIX.com shell programming forum and find some challenging tasks to solve. One of the questions posted today is: how to calculate the time difference between two timestamps.

Very often people "echo" date format to log file at the beginning and end in a shell script and the output will be stored in a log file. Suppose you are given the below output, how do you work out the elapsed time:
Oracle Q03 Begin Hot BACKUP Time: 07/23/08 18:35:46
Oracle Q03 End Hot BACKUP Time: 07/24/08 14:18:15

I know it is trivial to convert the timestamps to epoch in Tcl. How about other scripting languages such as Perl and Python

Let's find out.
Perl way:

#! /usr/bin/perl

use Time::Local;

while (<>) {
 if ( /(\d\d)\/(\d\d)\/(\d\d) (\d\d):(\d\d):(\d\d)$/ ) {
  $yr=$3+2000; $mth=$1; $day=$2;
  $hh=$4; $mm=$5; $ss=$6;

  if (/Begin Hot BACKUP/) {
   $t0=timelocal($ss,$mm,$hh,$day,$mth,$year);
  }
  if (/End Hot BACKUP/) {
   $t1=timelocal($ss,$mm,$hh,$day,$mth,$year);
  }
 }
}
print $t1-$t0,"\n";

Python way:

#! /usr/bin/python

import sys
import re
import datetime
import time

for newline in sys.stdin.readlines():
 line=newline.rstrip()
 t=re.findall('(\d\d)/(\d\d)/(\d\d) (\d\d):(\d\d):(\d\d)$',line)
 ts=t[0]
 if t:
  if 'Begin Hot BACKUP' in line:
   _t=datetime.datetime(int(ts[2])+2000, int(ts[0]), int(ts[1]), int(ts[3]), int(ts[4]), int(ts[5]))
   t0=int(time.mktime(_t.timetuple()))
  if 'End Hot BACKUP' in line:
   _t=datetime.datetime(int(ts[2])+2000, int(ts[0]), int(ts[1]), int(ts[3]), int(ts[4]), int(ts[5]))
   t1=int(time.mktime(_t.timetuple()))

print t1-t0

Tcl:

#! /usr/bin/tclsh

while { [gets stdin line] >= 0 } {
 if { [regexp {(\d\d/\d\d/\d\d \d\d:\d\d:\d\d)$} $line x ts] } {
  if { [string match {*Begin Hot BACKUP*} $line] } {
   set t0 [clock scan $ts]
  }
  if { [string match {*End Hot BACKUP*} $line] } {
   set t1 [clock scan $ts]
  }
 }
}
puts [expr $t1-$t0]

UNIX shell script way:

#! /bin/sh

: NO WAY!! Let me know if you have the solution simpler than above

In fact, you may want to consider working out the elapsed time in your original shell script. Here is the skeleton you may want to adopt:

#! /bin/sh
...
starttime=`perl -e 'print time()'`
...
# do you stuff
...
endtime=`perl -e 'print time()'`
...
echo "Elapsed time in seconds = `expr $endtime - $starttime`"

BTW, if you want sample codes as a reference, you should visit PLEAC (Programming Language Examples Alike Cookbook)

Labels: , ,

Thursday, April 17, 2008

Bastille tried to use $GLOBAL_BIN{'ping6'} but it does not exist

A colleague of mine was having problem running Bastille to harden his RedHat Linux server. He keeps getting the following error message and cannot find any solution on the web.
ERROR: Bastille tried to use $GLOBAL_BIN{'ping6'} but it does not exist.

I heard about Bastille but have no clue how it works. Anyway, here is how I managed to provide a workaround to avoid the above error message by explicitly define the fullpath of ping6.

bastille is actually a shell script that eventually launch a Perl script to run through the hardening steps. At the beginning of the bastille script, it defines PATH and PERL5LIB. PERL5LIB is the place to explore all the other bastille perl modules (.pm). Search through all the PM files for string "GLOBAL_BIN" and I found the following files that contain the string: API.pm, API.PM.sweth and OSX_API.PM

$ which ping6
/bin/ping6


$ cat /usr/sbin/bastille
...
...
export PATH="/bin:/usr/bin:/usr/sbin:$PATH"
export PERL5LIB="$PERL5LIB:/opt/sec_mgmt/bastille/lib:/usr/lib/perl5/site_perl:/usr/lib/Bastille:/usr/lib/perl5/site_perl/5.6.0/i386-linux"
...
...

$ cd /usr/lib/Bastille
$ grep GLOBAL_BIN *
API.pm:       %GLOBAL_BIN %GLOBAL_DIR %GLOBAL_FILE
API.pm:    my %map = ("BIN"   => \%GLOBAL_BIN,
API.pm:     # type relates to the map above, type bin will map to GLOBAL_BIN
API.pm:    foreach my $hashref (\%GLOBAL_BIN,\%GLOBAL_DIR,\%GLOBAL_FILE,\%GLOBAL_BFILE,\%GLOBAL_BDIR) {
API.pm:  my %map = ("BIN"   => \%GLOBAL_BIN,
API.pm:    # i.e. Bastille tried to use $GLOBAL_BIN{'cp'} but it does not exist.
API.pm.sweth:   %GLOBAL_BIN %GLOBAL_DIR %GLOBAL_FILE %GLOBAL_MISC
API.pm.sweth:         $GLOBAL_BIN{$DISTRO_FILE} = $PATH_TEMP;
API.pm.sweth:         `$GLOBAL_BIN{"cp"} -p $file $backup_file`;
API.pm.sweth:         $return=`$GLOBAL_BIN{"mknod"} $prefix $file $suffix`;
OSX_API.pm:    $GLOBAL_BIN{"md5sum"}="/usr/bin/md5sum";

In the OSX_API.pm, it is likely a PM meant for MacOSX. It also hard-coded the md5sum fullpath in the GLOBAL_BIN associative array. Hey, likely I can do the same for ping6 in the API.pm. My instinct tells me that API.pm is the generic API module and likely to be the entry point for bastille.

I explicitly define the full path of ping6 right after the package definition as shown below. Guess what, bastille runs without any error. I won't say the problem is solved, but at least I provided a workaround.

$ cat API.pm
...
...
package Bastille::API;
$GLOBAL_BIN{'ping6'}="/bin/ping6";

...
...

Labels: ,

Wednesday, March 26, 2008

Tcl Is As Fast As Perl

Netflixprize came with a utility check_format.pl to check the submission format before you upload your predictions.

The Perl utility took 11.437 seconds to check my submit file with a total of 2,834,601 lines. So how is Tcl compare with Perl in this "race". It took me about a couple of minutues to crank out a similar utility with almost similar functionalities. Guess what, Tcl is slightly faster than Perl with a run time of 11.128 seconds, that is 0.309 second faster.

My version of check_format:

#! /usr/local/bin/tclsh
if { $argc != 1 } {
    puts stderr "Usage: $argv0 <file>"
    exit 1
}
array set m {
1 53
10 10
...
...
9998 5
9999 3
}
set f [lindex $argv 0]
if { ![file exists $f] } {
    puts stdeer "Error. $f does not exist"
    exit 2
}
set fp [open $argv r]
set lineno 1
while { [gets $fp line] >= 0 } {
    if { [string match {*:} $line] } {
        set movie_id [string range $line 0 end-1]
        if { ![string is integer $movie_id] } {
            puts "Error. Movie_id=$movie_id not integer"
            exit 3
        }
        if { [info exists count] && $count != $m($mid) } {
            puts "Error. Movie_id=$mid, count=$count, should be $m($mid)"
            exit 4
        }

        set mid $movie_id
        set count 0
        set required $m($mid)
    } else {
        incr count
        if { $line < 1.0 || $line > 5.0 } {
            puts "Error. LINE#=$lineno, Movie id=$mid, count=$count"
            exit 5
        }
    }
    incr lineno
}
close $fp
puts "OK!"

See Tcl and Perl in action:

$ perl -v

This is perl, v5.8.4 built for i86pc-solaris-64int
(with 28 registered patches, see perl -V for more detail)

Copyright 1987-2004, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'.  If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.

$ tclsh
% set tcl_patchLevel
8.4.16
% parray tcl_platform
tcl_platform(byteOrder) = littleEndian
tcl_platform(machine)   = i86pc
tcl_platform(os)        = SunOS
tcl_platform(osVersion) = 5.10
tcl_platform(platform)  = unix
tcl_platform(threaded)  = 1
tcl_platform(user)      = chihung
tcl_platform(wordSize)  = 8
% set tcl_patchLevel
8.4.16
% exit

$ wc -l netflix-submission.txt
  2834601 2834601 11379859 netflix-submission.txt

$ time ../../check_format.pl netflix-submission.txt
OK!

real    0m11.437s
user    0m11.417s
sys     0m0.017s

$ time ./check_format.tcl netflix-submission.txt
OK!

real    0m11.128s
user    0m11.093s
sys     0m0.029s

Tcl has changed a lot and the statement of "Everything is string" is not really true. Internally it has more than one representation. As Donal Fellows once mentioned "Tcl_Obj's are like storks. They have two legs, the internal representation and the string representation. They can stand on either leg, or on both." See the typedef struct Tcl_Obj in the tcl.h (8.4):

typedef struct Tcl_Obj {
    int refCount;               /* When 0 the object will be freed. */
    char *bytes;                /* This points to the first byte of the
                                 * object's string representation. The array
                                 * must be followed by a null byte (i.e., at
                                 * offset length) but may also contain
                                 * embedded null characters. The array's
                                 * storage is allocated by ckalloc. NULL
                                 * means the string rep is invalid and must
                                 * be regenerated from the internal rep.
                                 * Clients should use Tcl_GetStringFromObj
                                 * or Tcl_GetString to get a pointer to the
                                 * byte array as a readonly value. */
    int length;                 /* The number of bytes at *bytes, not
                                 * including the terminating null. */
    Tcl_ObjType *typePtr;       /* Denotes the object's type. Always
                                 * corresponds to the type of the object's
                                 * internal rep. NULL indicates the object
                                 * has no internal rep (has no type). */
    union {                     /* The internal representation: */
        long longValue;         /*   - an long integer value */
        double doubleValue;     /*   - a double-precision floating value */
        VOID *otherValuePtr;    /*   - another, type-specific value */
        Tcl_WideInt wideValue;  /*   - a long long value */
        struct {                /*   - internal rep as two pointers */
            VOID *ptr1;
            VOID *ptr2;
        } twoPtrValue;
    } internalRep;
} Tcl_Obj;

Labels: , ,

Wednesday, April 18, 2007

clock scan, the perl way

Recently I was helping someone to find out those directories (in yyyymmdd- naming convention) that is 2 weeks old. It can be easily done in Tcl with clock scan. However, not every operating system comes with Tcl. As far as I know Tclsh8.3 is only available in Solaris 9 onwards, but not in Solaris 8.

If I were to do it in shell script, it will be very tough. In this case, I chose the Perl way for my implementation because Perl is available since Solaris 8.

$ cat clockscan.pl
#! /usr/bin/perl


$narg=$#ARGV+1;
if ( $narg < 2 || $narg > 3 ) {
die "Usage: $0   [ago]\n";
}


$num=$ARGV[0];

# unit
$_=lc($ARGV[1]);
if ( /^second[s]?$/ ) {
        $factor=1;
} elsif ( /^minute[s]?$/ ) {
        $factor=60;
} elsif ( /^hour[s]?$/ ) {
        $factor=60*60;
} elsif ( /^day[s]?$/ ) {
        $factor=24*60*60;
} elsif ( /^week[s]?$/ ) {
        $factor=7*24*60*60;
} elsif ( /^month[s]?$/ ) {
        $factor=30*24*60*60;
} elsif ( /^year[s]?$/ ) {
        $factor=365*24*60*60;
} else {
        die "Error $unit unknown";
}

# ago
$_=lc($ARGV[2]);
$sign=1;
if ( /^ago$/ ) {
        $sign=-1;
}


# count
$diff=$sign*$factor*$ARGV[0];
$,=" ";
$now=time();


($sec,$min,$hr,$day,$mth,$year)=localtime($now+$diff);
printf("%04d-%02d-%02d %02d:%02d:%02d\n",$year+1900,$mth+1,$day,$hr,$min,$sec);


$ ./clockscan.pl 2 weeks ago
2007-04-03 18:04:07

$ ./clockscan.pl -2 weeks
2007-04-03 18:04:12

$ ./clockscan.pl 1 year ago
2006-04-17 18:04:17

$ ./clockscan.pl 2 months
2007-06-16 18:04:22

Labels: , , ,

Saturday, April 14, 2007

mkdir, the limit

A friend of mine hit the limit of having no more than 32765 sub-directories. In Solaris 8, it has been documented: "Too many links cause an attempt was made to create more than the maximum number of hard links (LINK_MAX, by default 32767) to a file. Because each subdirectory is a link to its parent directory, the same error results from trying to create too many subdirectories"

So the question is: Is there anything we can do. I don't think so because this is a built in limit in Solaris. What one can do is to ask ourself why we need that many sub-directories and can we change that flat directory structure to a hierarchical. It is pretty obvious that too many files/directories in folder will have a performance issue when an application interacts with it.

Let do an experiment just to convince ourself that indeed there is a limit:

$ uname -a
SunOS myhost 5.9 Generic_118558-11 sun4u sparc SUNW,Sun-Fire-V240

$ psrinfo -v
Status of processor 0 as of: 04/14/2007 12:16:12
  Processor has been on-line since 10/16/2006 09:10:39.
  The sparcv9 processor operates at 1002 MHz,
        and has a sparcv9 floating point processor.
Status of processor 1 as of: 04/14/2007 12:16:12
  Processor has been on-line since 10/16/2006 09:10:38.
  The sparcv9 processor operates at 1002 MHz,
        and has a sparcv9 floating point processor.

$ mkdir test1

$ cd test1

$ time for i in `perl -e '$,=" ";print 1..32768'`
 do
 mkdir $i
 done
mkdir: Failed to make directory "32766"; Too many links
mkdir: Failed to make directory "32767"; Too many links
mkdir: Failed to make directory "32768"; Too many links

real    2m52.716s
user    0m40.830s
sys     2m7.210s

$ cd ..

$ perl
$dir="test1";
($x,$x,$x,$nlink)=stat($dir);
print $nlink,"\n";
 
32767

As we can see, the maximum number of sub-directories one can create will be 32765. The total links is 32767 because for every directory we create, it creates two links, one for itself (.) and the other one is the parent directory (..)

Let sidetrack a little bit and take a look from the performance angle. It seems to take almost 3 minutes to create 32765 sub-directories. Can it be faster? Let see what interpreted languages like Tcl and Perl can offer.

The perl way:

$ perl -v

This is perl, v5.6.1 built for sun4-solaris-64int
(with 48 registered patches, see perl -V for more detail)

Copyright 1987-2001, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'.  If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.

$ mkdir test2

$ cd test2

$ time $ perl
$|=1;
$start=1;
$end=32768;
for ($i=$start; $i<=$end; $i++) {
    mkdir($i);
}
 
real    0m21.098s
user    0m0.180s
sys     0m7.740s
Wow, we are talking about 8 times speed up. How about my favourite Tcl. The Tcl way:
$ cat a.tcl
#! /usr/sfw/bin/tclsh8.3

for { set i 1 } { $i <= 32768 } { incr i } {
        file mkdir $i
}

$ mkdir test3

$ cd test3

$ time ../a.tcl
can't create directory "32766": too many links
    while executing
"file mkdir $i"
    ("for" body line 2)
    invoked from within
"for { set i 1 } { $i <= 32768 } { incr i } {
        file mkdir $i
}
"
    (file "../a.tcl" line 3)

real    0m20.796s
user    0m0.900s
sys     0m8.040s
Wow, I am impressed that Tcl 8.3 (latest is 8.4) is as good as Perl 5.6.1 (latest is 5.8.8)

BTW, to remove that many sub-directories (under test1, test2, test3) 3x32767, it took

$ time /bin/rm -rf test1 test2 test3

real    0m35.425s
user    0m1.210s
sys     0m19.740s
Why perl and Tcl can perform better than the shell script ? Obviously, there is no fork or exec of processes in Perl and Tcl because they have the "mkdir" function call built-in into their interpreter. So how much overhead we are talking about, let's do another experiement.
$ truss mkdir newdir 2>&1 | wc -l
      48

$ truss mkdir u
execve("/usr/bin/mkdir", 0xFFBFFCF4, 0xFFBFFD00)  argc = 2
resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
resolvepath("/usr/bin/mkdir", "/usr/bin/mkdir", 1023) = 14
stat("/usr/bin/mkdir", 0xFFBFFAC8)              = 0
open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
stat("/usr/lib/libgen.so.1", 0xFFBFF5D0)        = 0
resolvepath("/usr/lib/libgen.so.1", "/usr/lib/libgen.so.1", 1023) = 20
open("/usr/lib/libgen.so.1", O_RDONLY)          = 3
mmap(0x00010000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_ALIGN, 3, 0) = 0xFF3A0000
mmap(0x00010000, 98304, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFF380000
mmap(0xFF380000, 22677, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF380000
mmap(0xFF396000, 2343, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 24576) = 0xFF396000
munmap(0xFF386000, 65536)                       = 0
memcntl(0xFF380000, 6304, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3)                                        = 0
stat("/usr/lib/libc.so.1", 0xFFBFF5D0)          = 0
resolvepath("/usr/lib/libc.so.1", "/usr/lib/libc.so.1", 1023) = 18
open("/usr/lib/libc.so.1", O_RDONLY)            = 3
mmap(0xFF3A0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF3A0000
mmap(0x00010000, 802816, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFF280000
mmap(0xFF280000, 701788, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF280000
mmap(0xFF33C000, 24664, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 704512) = 0xFF33C000
munmap(0xFF32C000, 65536)                       = 0
memcntl(0xFF280000, 117372, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3)                                        = 0
stat("/usr/lib/libdl.so.1", 0xFFBFF5D0)         = 0
resolvepath("/usr/lib/libdl.so.1", "/usr/lib/libdl.so.1", 1023) = 19
open("/usr/lib/libdl.so.1", O_RDONLY)           = 3
mmap(0xFF3A0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF3A0000
mmap(0x00002000, 8192, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFF3FA000
mmap(0xFF3FA000, 1894, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF3FA000
close(3)                                        = 0
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFF370000
stat("/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1", 0xFFBFF2E0) = 0
resolvepath("/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1", "/usr/platform/sun4u-us3/lib/libc_psr.so.1", 1023) = 41
open("/usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1", O_RDONLY) = 3
mmap(0xFF3A0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF3A0000
close(3)                                        = 0
getustack(0xFFBFF914)
getrlimit(RLIMIT_STACK, 0xFFBFF90C)             = 0
getcontext(0xFFBFF748)
setustack(0xFF343A5C)
brk(0x000245E8)                                 = 0
brk(0x000265E8)                                 = 0
umask(0)                                        = 077
umask(077)                                      = 0
mkdir("u", 0777)                                = 0
_exit(0)

Ok, we now know that for every 'mkdir', it calls 48 system functions. So there will be 32,765 x 48 = 1,572,720 system calls in order to create that many directories. In Tcl/Perl, it only takes 6 system calls to create a single directory and that is 8 times less system calls. This tally with our initial speed up calculation.

9364/1:         read(0, " f i l e   m k d i r   t".., 4096)     = 13
9364/1:         stat("t", 0xFFBFE910)                           Err#2 ENOENT
9364/1:         umask(0)                                        = 077
9364/1:         umask(077)                                      = 0
9364/1:         mkdir("t", 0700)                                = 0
9364/1:         write(1, " %  ", 2)                             = 2

Labels: , , , ,

Thursday, June 01, 2006

Reliable Computing, scripting language perspective

I am happy that I attended the Sun HPC Consortium Singapore 2006. This was held in conjunction with GridAsia 2006.

One of the presentations (Reliable Computing and Petascale systems, by Ruud van der Pas, Sun Microsystems) interested me to explore further, from the viewpoint of scripting languages. Although his presentation material is not available, I managed to find a similar paper that he presented last year in Australia.
The Future Of Floating-Point Computing Is In Reliable Computing, in Australian Partnership for Advanced Computing Conference and Exhibition on Advanced Computing, Grid Applications and eResearch, 26-30 September 2005.

The equation in question is:
F(a,b) = (333.75-a^2)*b^6 + a^2*(11*a^2*b^2-121*b^4-2) + 5.5*b^8 + a/(2*b)
a = 77617.0; b = 33096.0

The answer (using 32-bit precision): 1.172604. Question: How Good Is This Result?

In thr APAC 2005 paper, it said:

"If we bump up the precision and the numbers match, we're okay" Not Necessarily .......

% ./rump.exe
Computed Results
32-bit precision: 1.172604
64-bit precision: 1.1726039400531786
128-bit precision: 1.1726039400531786949244406059733592


% ./rump.exe
Computed Results
32-bit precision: 1.172604
64-bit precision: 1.1726039400531786
128-bit precision: 1.1726039400531786949244406059733592
Correct Results
32-bit precision: -0.82739603
64-bit precision: -0.8273960599468213
128-bit precision: -0.8273960599468213050755593940266408

Okay, let's look at how scripting languages handle the above equation on my notebook running Cygwin:

Perl (5.8.7)
[ans=1.17260394005318] NOT CORRECT

$ cat rump.pl
#! /usr/bin/perl

$a=77617.0;
$b=33096.0;
$a2=$a*$a;
$b2=$b*$b;
$b4=$b2*$b2;
$b6=$b4*$b2;
$b8=$b6*$b2;
$ans=(333.75-$a2)*$b6 + $a2*(11*$a2*$b2-121*$b4-2) + 5.5*$b8 + $a/(2*$b);
print $ans;

$ perl rump.pl
1.17260394005318

Tcl (8.4.1)
[ans=1.17260394005] NOT CORRECT

$ cat rump.tcl
#! /usr/bin/tclsh

proc f { a b } {
        set a2 [expr {$a*$a}]
        set b2 [expr {$b*$b}]
        set b4 [expr {$b2*$b2}]
        set b6 [expr {$b4*$b2}]
        set b8 [expr {$b6*$b2}]
        set ans [expr {(333.75-$a2)*$b6 + $a2*(11*$a2*$b2-121*$b4-2) +                 5.5*$b8 + $a/(2*$b)}]
        return $ans
}
puts [f 77617.0 33096.0]

$ ./rump.tcl
1.17260394005

Apparently all these calculations are based on IEEE floating point standard. If we were to calculate it long hand, it looks like this:

(333.75 - a^2)*b^6=
-7917110903377385049079188237280149504.00

a^2*(11a^2*b^2-121*b^4-2)=
       -437291576312021946464244793346.00

5.5*b^8=
 7917111340668961361101134701524942848.00

a/(2*b)=
                                     1.17260394005317863185
-------------------------------------------------------------------------------
F(a,b)=
                                    -0.8273960599468213681411650954798162919991


-7917110903377385049079188237280149504.00 +
-437291576312021946464244793346.00 +
7917111340668961361101134701524942848.00 = 2

Can we do it correctly with scripting languages (if we consider "bc" also) ? Let see

bc (1.06) [ANS=-.8273960599468213681411650954798162919991] CORRECT

$ cat rump.bc
scale=40
a=77617.0
b=33096.0
(333.75-a^2)*b^6+a^2*(11*a^2*b^2-121*b^4-2)+5.5*b^8+a/(2*b)

$ bc -l < rump.bc
-.8273960599468213681411650954798162919991
Perl (5.8.7) (same script as above, but run with Bignum module)
[ans=-0.827396059946821368141165095479816291999] CORRECT
$ cat rump.pl
#! /usr/bin/perl

$a=77617.0;
$b=33096.0;
$a2=$a*$a;
$b2=$b*$b;
$b4=$b2*$b2;
$b6=$b4*$b2;
$b8=$b6*$b2;
$ans=(333.75-$a2)*$b6 + $a2*(11*$a2*$b2-121*$b4-2) + 5.5*$b8 + $a/(2*$b);
print $ans;

$ perl -Mbignum rump.pl
-0.827396059946821368141165095479816291999
Tcl (math::bigfloat)
[ans=-0.827396059946821368141165095479816291999] CORRECT
The tricky part is how much precision one has to set to get the correct answer. Also, the implementation is very clumsy.
package require math::bigfloat
namespace import math::bigfloat::*

set precision 77
set a [fromdouble 77617.0 $precision]
set b [fromdouble 33096.0 $precision]

set c333_75 [fromdouble 333.75 $precision]
set c11 [fromdouble 11.0 $precision]
set c121 [fromdouble 121.0 $precision]
set c2 [fromdouble 2.0 $precision]
set c5_5 [fromdouble 5.5 $precision]

set a2 [mul $a $a]
set b2 [mul $b $b]
set b4 [mul $b2 $b2]
set b6 [mul $b4 $b2]
set b8 [mul $b6 $b2]

set term1 [mul [sub $c333_75 $a2] $b6]
set term2 [mul $a2 [sub [sub [mul [mul $c11 $a2] $b2] [mul $c121 $b4]] $c2]]
set term3 [mul $c5_5 $b8]
set term4 [div $a [mul $c2 $b]]

set ans [add [add [add $term1 $term2] $term3] $term4]
puts [tostr $ans]
Tcl (Mpexpr)
[ans=-0.827396059946821368141165095479816291999] CORRECT
Tcl module (Mpexr ) is able to do multiple precision arithmatic. Below is Tcl 8.1.1 with Mpexr on Solaris, because I have problem compiling that with the latest Tcl8.4 on Cygwin
$ cat rump.tcl
package require Mpexpr
proc f { a b } {
        set a2 [mpexpr {$a*$a}]
        set b2 [mpexpr {$b*$b}]
        set b4 [mpexpr {$b2*$b2}]
        set b6 [mpexpr {$b4*$b2}]
        set b8 [mpexpr {$b6*$b2}]
        set ans [mpexpr {(333.75-$a2)*$b6 + $a2*(11*$a2*$b2-121*$b4-2) +                 5.5*$b8 + $a/(2*$b)}]
        return $ans
}
set mp_precision 40
puts [f 77617.0 33096.0]

$ tclsh rump.tcl
-0.827396059946821368141165095479816291999

Conclusion

Scripting languages can be deployed in the area of reliable computing, but one has to be mindful in choosing the right modules for the job. BTW, I am very impressed by Perl's Bignum implementation which hides all the technicality from the user. The same perl script can run with or without Bignum module.

Labels: , , , ,