The AWK Way

Today I was given the task of converting few hundred files (743 to be exact) into CSV format. The filename is prefixed with hostname with a fix suffix and the content contains all the local user names. The task is to put them in rows with hostname in the 1st column and usernames in the 2nd column onwards. One more requirement is to exclude a few users in the output. My initial solution is very much unix shell script-based. Although this is an one-off 'throw-away' solution, it is pretty inefficient because there is a lot of process creation within a for loop. It took 1 min 39.453 sec. After some thoughts, I reckoned it is possible to do it efficiently in just AWK. With the help of some of the built-in variables like FILENAME / NR / FNR, we can process all the input files within a single AWK code. The below code works in Cygwin. The runtime for the AWK code is 2.797 sec, that's 35 times faster !

$ ls *txt
host1_root.txt  host2_root.txt  host3_root.txt  host4_root.txt

$ paste *txt
usera   usere   userm   userx
userb   userx   userx   userw
userc   userf   usern   usery
userd   userg   usero   userz
userdx  usery   userp
userdy  userh   userx
        userz   userq
        useri   userqx
        userj   userr
        userk   userz
        userl   users
        userx   usert
                usery

$ cat a.awk
#! /usr/bin/awk -f


BEGIN {
        suffix="_root.txt"
        len=length(suffix)
}
#
# print CR if first line in input file except first file
FNR==1 && NR>1 {
        printf("\n")
}
#
# print hostname
FNR==1 {
        host=substr(FILENAME, 0, length(FILENAME)-len)
        printf("%s", host)
}
#
# print users, but exclude certain users
$0 !~ /^(userx|usery|userz)$/ {
        printf(",%s", $0)
}


$ ./a.awk *.txt
host1,usera,userb,userc,userd,userdx,userdy
host2,usere,userf,userg,userh,useri,userj,userk,userl
host3,userm,usern,usero,userp,userq,userqx,userr,users,usert
host4,userw

Finding Newer Files

Although find provides flags to locate files newer than certain days (-mtime -2), it is not fine grain enough to allow user to specify based on user-defined timestamp. Below script will touch a temporary file with specific timestamp and find will make use of it as a reference by using the "-newer" flag. FYI, this has been tested on Solaris

#! /bin/ksh

if [ $# -ne 2 ]; then
        echo "Usage: $0 directory time"
        echo "       time - a decimal number of the form: [[CC]YY]MMDDhhmm[.SS]"
        exit 1
fi
directory=$1
timestamp=$2


if [ ! -d $directory ]; then
        echo "Error. Directory $directory does not exist"
        exit 2
fi


tmpfile="/tmp/${0##*/}-$$"
touch -t $timestamp $tmpfile > /dev/null 2>&1
if [ $? -ne 0 ]; then
        echo "Error. Incorrect time format [[CC]YY]MMDDhhmm[.SS]"
        exit 2
fi


trap "rm -f $tmpfile; exit 0" EXIT


find $directory -newer $tmpfile -local -mount -type f

What Is Personal Leadership Brand

If you have 10 minutes to spare, you should watch this video from Harvard Business in YouTube

Thumbnail Your RRDtool Graphs

If you have lots of RRDtool graphs you need to go through everyday, you may want to consider creating thumbnail dynamically from the RRD files. As CPU and memory utilisations are plotted with same scale (100%), we can put them together for ease of viewing and analysis.

A simple CGI script is able to do the job. Basically it ignores x- & y-grids/ticks/labels and legends

echo "Content-tyep: image/png"
echo ""
/usr/bin/rrdtool graph - \
        --start -${2:-1d} \
        --imgformat PNG \
        --lower-limit 0 --upper-limit 100 --rigid \
        --width=125 --height=30 \
        --x-grid none \
        --y-grid none \
        --title "$host" \
        --color=BACK#ffffff \
        --color=CANVAS#ffeee5 \
        --color=GRID#7F7F7F \
        --color=MGRID#B8B8B8 \
        --color=ARROW#FF0000 \
        DEF:usr=$rrd_cpu:usr:AVERAGE \
        DEF:sys=$rrd_cpu:sys:AVERAGE \
        DEF:wio=$rrd_cpu:wio:AVERAGE \
        DEF:idle=$rrd_cpu:idle:AVERAGE \
        DEF:mtotal=$rrd_mem:total:AVERAGE \
        DEF:mused=$rrd_mem:used:AVERAGE \
        CDEF:gbtotal=mtotal,1024,/ \
        CDEF:gbused=mused,1024,/ \
        CDEF:gbfree=mtotal,mused,-,1024,/ \
        CDEF:percent=gbfree,gbtotal,/,100,* \
        AREA:usr#dddd00:\
        STACK:sys#dd0000:\
        STACK:wio#ff8a60:\
        STACK:idle#e2e2f2:\
        LINE1:percent#0000ff: \
        GPRINT:gbfree:LAST:"Mem\:%5.1lf/" \
        GPRINT:gbtotal:LAST:"%5.1lfGB"

New DTrace Resources

Some new resources from Sun on DTrace

• DTrace Toolkit
• DTraceQuick StartGuide, Observing Native andWeb Applications in Production
• Observing Applications with DTrace (video)

Scripting Hustle

I was given this compliment:
不愧是万中无一的scripting guy
+ learn a lifetime skill of scripting for $10

Watch this movie clip (fast forward to 08:40) to find out the resemblance

Too Big An Audit Log To Handle

Have you ever encounter Solaris audit log that has not been rotated for months ? It is just far too big for some of the down stream programs to handle. Click here for a sample of the output from praudit -l. Here is a sample script to 'chop' them into individual files based on day. With this simple script, now you can handle audit log with ease.

IFS_orig=$IFS
cd /var/audit
praudit -l 2009052803069.20091009095022.myhost | while read line
do
        IFS=","
        set -- $line
        d=$7
        IFS=$IFS_orig
        set -- $d
        ymd=$1
        echo $line >> $ymd.txt
done

Number of Days in a Month

If you have to loop through everyday in a month to generate daily summary, you may want to know how many days in a particular month. It may not be as straightforward as you think because you need to take into account of leap year, short months, and so on. Here is a simple trick that can help you on this. Run the cal command to display the calendar and pipe it to awk to pick up the last number from the last non-empty line.

$ cal
    October 2009
Su Mo Tu We Th Fr Sa
             1  2  3
 4  5  6  7  8  9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

$ cal | awk 'NF>0{n=$NF}END{print n}'
31

$ cal 2 2012
    February 2012
Su Mo Tu We Th Fr Sa
          1  2  3  4
 5  6  7  8  9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29

$ cal 2 2012 | awk 'NF>0{n=$NF}END{print n}'
29

Date Format, N Days Ago

In UNIX shell script, it is very difficult to manipulate date/time using standard UNIX commands. Modern programming languages such as Perl, Python, Tcl, ..., are designed to be a general programming/scripting language that can handle this type of task with ease.

Below shows shell functions that wrap around these general scripting languages to determine the date format that is N days ago.

ago_perl()
{
        # localtime return:
        #       Seconds past the minute
        #       Minutes past the hour
        #       Hours past midnight
        #       Day of the month
        #       Months past the start of the year
        #       Number of years since 1900
        #       Number of days since the start of the week (Sunday)
        #       Number of days since the start of the year
        #       Whether or not daylight savings is active
        days=${1:-1}
        perl -e "@t=localtime(time-86400*$days);\
                printf('%4d-%02d-%02d', @t[5]+1900, @t[4]+1, @t[3])"

}
ago_tcl()
{
        days=${1:-1}
        echo "puts [clock format [expr [clock seconds] - $days*86400] \
                -format {%Y-%m-%d}]" | tclsh
}
ago_python()
{
        days=${1:-1}
        python -c "import time, datetime
t=datetime.datetime.fromtimestamp(time.time()-86400*$days)
print '%4d-%02d-%02d' % (t.year,t.month,t.day)"
}

Here is how you can use any of these functions

$ ago=`ago_perl 45`

$ echo "45 days ago is $ago"
45 days ago is 2009-08-12

Monitoring of LTO Tape Drives via FC Switch

I was trying to monitor the throughput of individual LTO tape drive but not getting any useful information from the tape library system. The only way to overcome this is to monitor it via the FC switch. In my case, it is a Brocade switch. Online SNMP MIBs for Brocade provides a very detailed description of the OIDs (swFCPortRxWords:1.3.6.1.4.1.1588.2.1.1.1.6.2.1.12 and swFCPortTxWords: 1.3.6.1.4.1.1588.2.1.1.1.6.2.1.11).

Due to the fact that the counter reset so frequent, I need to poll the FC switch every minute. Data is stored in RRDtool format.

Below shows two ways to visualise the LTO throughput. Line grpah shows individual drive throughput. Area/Stack graph shows the overall throughput of the backup: