Delete Old Files

Wanna to delete those old files with a certain extension in some directory, there you go...

#! /bin/sh

if [ $# -ne 3 ]; then
        echo "Usage: $0 <directory> <days old> <extension>"
        exit 1
fi


PATH=/usr/bin:/bin
export PATH


dir="$1"
old="$2"
ext="$3"


if [ ! -d "$dir" ]; then
        echo "Error. $dir does not exist"
        exit 2
fi


find "$dir" -type f -name "*.$ext" -mtime "+$old" -exec rm -i {} \;

If you plan to run that in a cron environment, you will need to remove the "-i" option and discard all the standard output and error to avoid cron from sending you an email. I also include checking to ensure the "old" is an integer greater than 0. The modified script will look like this.

#! /bin/sh

if [ $# -ne 3 ]; then
        echo "Usage: $0 <directory> <days old> <extension>"
        exit 1
fi


PATH=/usr/bin:/bin
export PATH


dir="$1"
old="$2"
ext="$3"



if [ ! -d "$dir" ]; then
        echo "Error. $dir does not exist"
        exit 2
fi
expr $old / 1 > /dev/null 2>&1 && [ $old -gt 0 ]
if [ $? -ne 0 ]; then
        echo "Error. $old is not a number > 0"
        exit 3
fi
 


find "$dir" -type f -name "*.$ext" -mtime "+$old" -exec rm {} \; > /dev/null 2>&1

In Linux, you can specify "-maxdepth" to control descend at most levels. In our case, we need to have an equivalent of "-maxdepth 1" in Solaris. In this blog, it mentioned using "-prune" to simulate this effect. However, I cannot get it working for me.

Alternatively, you can use egrep and xargs to achieve this:

find "$dir" -type f -name "*.$ext" -mtime "+$old" | egrep "^$dir/[^/]+$" | xargs rm

Archive Your System Activity Reports in Solaris 10

If you want to archive your system activity report (sar) in Solaris 10, you need to do the following:

1. Enable sar. By default, sar is not enabled in Solaris.

   $ su -
   
   # svcs sar
   STATE          STIME    FMRI
   disabled       Apr_12   svc:/system/sar:default
   
   # svcadm enable sar
   

2. Setup cron job for "sys" account. The default the "sys" user crontab file is commented out. In this example, we set up sar with 5 minute interval.

   # crontab -l sys
   #ident  "@(#)sys        1.5     92/07/14 SMI"   /* SVr4.0 1.2   */
   #
   # The sys crontab should be used to do performance collection. See cron
   # and performance manual pages for details on startup.
   #
   # 0 * * * 0-6 /usr/lib/sa/sa1
   # 20,40 8-17 * * 1-5 /usr/lib/sa/sa1
   # 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A
   0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/lib/sa/sa1
   

3. Create a script to copy sar data file to an archive directory and rename it with date information. Ensure this script will be executed just after the last sampling interval, in my case, 23:59.

   $ su -
   
   # cat /usr/lib/sa/sacp
   #! /bin/sh
   
   PATH=/bin:/usr/bin
   export PATH
   DIR=/var/adm/sa
   ARCHIVE="$DIR/archive"
   [ ! -d "$ARCHIVE" ] && mkdir $ARCHIVE
   TARGET="$ARCHIVE/sa`date '+%Y-%m-%d'`"
   cp $DIR/sa`date '+%d'` $TARGET
   bzip2 $TARGET
   
   # chown root:bin /usr/lib/sa/sacp
   
   # chmod 555 /usr/lib/sa/sacp
   
   # crontab -l sys
   #ident  "@(#)sys        1.5     92/07/14 SMI"   /* SVr4.0 1.2   */
   #
   # The sys crontab should be used to do performance collection. See cron
   # and performance manual pages for details on startup.
   #
   # 0 * * * 0-6 /usr/lib/sa/sa1
   # 20,40 8-17 * * 1-5 /usr/lib/sa/sa1
   # 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A
   0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/lib/sa/sa1
   59 23 * * * /usr/lib/sa/sacp
   

RSS Feed from Government Sites ? One only

I was asked to write a feed aggregration for a project, so I tried to aggregrate the government related sites (150+) as an examples. Guess what, all of those sites that are available do not have RSS or Atom feed provided in their HTML header, execpt the official government site.

Tcl program was written using packages such as http and tDOM to do this exploratory work. BTW, ActiveTcl from ActiveState has a lot of these extensions pre-compiled. I extracted all the govt related sites from their "A-Z Government List" and looped through all these sites to see whether they provide feed in their header. A snippet of the Tcl using XPath syntax to locate the feed link is:

set result [$root selectNode {//link[@type="application/rss+xml" or @type="application/atom+xml"]}]
if { [llength $result] > 0 } {
 puts "Yes - $url"
} else {
 puts "No  - $url"
}

So the big question is: Why there isn't any feed ? May be it is embedded in the html body instead of head, or there is intellectual property in the feed that is too valuable to expose to the rest of the world.

Anyway, I will let you to figure that out.

Timestamp your monitoring output

Some of the system / application monitoring commands (eg, vmstat, mpstat, iostat, ...) allow user to continuously monitor based on certain time interval. However, they do not have option to 'timestamp' the sampled data.

I used to do that with Tcl with 'fileevent readable' to trap the output and prepend the 'clock format.

It is possible to do all these with nawk (You cannot use awk in Solaris because it does not support "system" execution).

Run this and have the result append to the log file

vmstat 1 | nawk '{system("date +%Y-%m-%dT%H:%M:%S|tr -d \"\n\"");printf(" = %s\n",$0)}' | tee -a vmstat.log

and you will get this

2007-05-11T13:46:01 =  kthr      memory            page            disk          faults      cpu
2007-05-11T13:46:01 =  r b w   swap  free  re  mf pi po fr de sr f0 m1 m1 m1   in   sy   cs us sy id
2007-05-11T13:46:01 =  0 0 0 474432 58664   0   2  2  1  1  0  0  0  0 0  0  418   51  115  0  1 99
2007-05-11T13:46:02 =  0 0 0 318976  9368  68 600 150 0  0  0  0  0  8 0  2  481 1050  273  3  9 88
2007-05-11T13:46:03 =  0 0 0 318976  9368  24 190 40  0  0  0  0  0  2 0  0  437  438  207  1  5 94
2007-05-11T13:46:04 =  0 0 0 318976  9368  25 188 31  0  0  0  0  0  2 0  1  427  408  222  2  4 94

With the timestamp, you are able to plot it with Gnuplot with x-axis configured as time scale. Suppose we want to plot the kernal threads of runable and blocked.

set title 'Kthr - r and b'
set ylabel 'No. of Kernel Threads'
set xdata time
set timefmt '%Y-%m-%dT%H:%M:%S'
plot 'vmstat.log' using 1:3 title 'kthr-r' with linespoints, \
                  using 1:3 title 'kthr-b' with linespoints

Happy timestamp-ing :-)

ZFS Adaptive Replacement Cache (ARC) relinquishes memory

One of my customers has a SunFire X4500 (a.k.a Thumper) serving as a NAS for all the Galaxy type of servers running Windows 2003 via Samba.

All servers' CPU/Memory/Disks/Network are monitored using SNMP. Something strange happens in the X4500 server 'cos the memory utilisation keep going up. Although the server has 16GB memory, it is definitely not a good sign for a production server. This seems to indicate some form of memory leak in the system. However, we are not delopying any application except Samba which comes with Solaris 10 anyway. I don't think Samba or Solaris itself has some serious memory leaking issue, so where does the memory go ?

After some searching in the Internet, I found the answer to the problem. ZFS Best Practices Guide explained that "The ZFS adaptive replacement cache (ARC) tries to use most of a system's available memory to cache file system data. The default is to use all of physical memory except 1 Gbyte. As memory pressure increases, the ARC relinquishes memory.". So, how true is this ? Let's find out.

I am lucky to have an identical X4500 for testing to verify that the Solaris kernel indeed do free up memory. I wrote a script to create a 200MB file (dd if=/dev/urandom of=/zfs/`perl -e 'print time()'` bs=1024 count=204800 > /dev/null 2>&1) in the ZFS file systems and run that for every minute as a cron job. SAR (system activity reporter) is enabled and configured to collect SAR info every 5 minutes.

sar -r of the freemem (average pages available to user processes) is plotted using Gnuplot. BTW, the size of a page in Solaris 10 x86 is 4096 (run pagesize to get the system page size). As you can see, it relinquishes memory when it is about 2GB freemem.

If we continue to run the test for hours, the freemem looks like this. BTW, the CPU utilisation is about 5% (0% usr, 5% sys) throughout.

Web Scraping, the Tcl way

My office's resource booking system does not provide a birdeye view of the availability of resource (meeting rooms). This seems to be another good opportunity to web scrap it with Tcl (with tDOM) to reformat the data into something more user friendly.

HTML data is read and DOM tree is built (set doc [dom parse -html $data]). Once we have the DOM, it is pretty easy to locate the information using XPath

This is a snippet of the HTML code:

<table>
<tr class='DashNavCurDashArea'>
<td align=left valign=top height=60><b>3</b>
<br><font size=1>
<a Title='boardroom: user1' href="display_event.asp?Pkey=13931"><img src=image/R.gif border=0 alt='boardroom: user1'>
10:00-12:00
</font></a><br>
<font size=1>
<a Title='room3: user2' href="display_event.asp?Pkey=13948"><img src=image/R.gif border=0 alt='room3: user2'>
13:30-15:00
</font></a><br>
<font size=1>
<a Title='boardroom: user3' href="display_event.asp?Pkey=13938"><img src=image/R.gif border=0 alt='boardroom: user3'>
16:30-18:00
</font></a><br>
...

With tDOM, you can locate all the resources (or nodes) booked today and then loop through them to extract the username and timeslot.

set todayNode [$root selectNode "//table/tr\[@class='DashNavCurDashArea'\]/td\[b/text()=\"$today\"]"]
foreach node [$todayNode selectNode {font/a/img[@src="image/R.gif"]}] {
 foreach { r u } [split [$node getAttribute alt] {:}] {}
 set room [string trim $r]
 set user [string trim $u]
 set time [string trim [[$node nextSibling] nodeValue]]
}

Since the resource booking timeslot interval is 10 minute, I create a HTML table with 144 (24*6) columns to represent each and every interval in a day. If a particular resource at a particular interval is taken, the table cell will be filled by a 1x1 pixel image (in red, but resize to 5x10). Also, they will be hyperlinked to itself (#) with attribute title set to the username so that the username will be displayed when mouse over it.

Before:

After: