Wednesday, April 30, 2008

VirtualBox -> Ubuntu 8.04 -> LINPACK Benchmark

After watching this video, I have an urge to run the LINPACK benchmark on my new notebook with Ubuntu as guest OS in VirtualBox.

To run LINPACK benchmark, you need to install MPICH (A Portable Implementation of MPI, Message Passing Interface). Installation of MPICH is really easy in Ubuntu and you do not need to go through the compilation phase. Simply launch the Synaptic Package Manager (located at System->Administration) and select "mpich 1.2.7-8 , MPI parallel computing system implementation" package to install. It is dead simple.

LINPACK benchmark does not come with binary and therefore you need to compile it on your target system. You will need to download these dependencies - BLAS (Basic Linear Algebra Subprogram), ATLAS (Automatically Tuned Linear Algebra Software). Configure LINPACK to point to the BLAS (Basic Linear Algebra Subprogram) and ATLAS (Automatically Tuned Linear Algebra Software) library by setting the "MPdir", "MPinc", "MPlib", "LAdir", "LAinc" and "LAlib" variables in the Make file specific to your architecture.

To run the benchmark, run mpirun -np 1 xhpl. Remember to configure the HPL.dat to fit your environment. I am running off 1 CPU because I get VirtualBox to allocate 1 CPU worth of compute power (I only have 2 cores). I did not finish the entire run because the heat generated on my laptop is "too hot for me to handle". Anyway, this is the screenshot.

This is the video that I watched
An Overview of High Performance Computing and Challenges for the Future

Labels: ,

Tuesday, April 29, 2008

Sun Grid Engine for Rendering

In the past, I used to submit animation scene files to SGE (Sun Grid Engine) to render as individual job per frame. It is very hard to manage because we have to deal with lots of SGE job IDs for a scene.

Do you know that in SGE you can treat that as an array job and let SGE to handle the scheduling of individual sub-task. Eg. if you have a scene file to render with frame 1 to 256, you can submit as qsub -t 1-256:1 render.sh scenefile.mb. SGE will only assign one SGE job ID. This give you the flexibility to modify the entire job with a single SGE job ID, eg. alter the job dependency, change the priority, hold the job in queue, remove from queue. With this, you do not have to keep track in terms of individual frame. In SGE man page, -t option is described as:

Submits a so called Array Job, i.e. an array of identical tasks
being differentiated only by an index number and being treated
by Sun Grid Engine almost like a series of jobs. The option
argument to -t specifies the number of array job tasks and the
index number which will be associated with the tasks. The index
numbers will be exported to the job tasks via the environment
variable SGE_TASK_ID.

In essence,

  • Scene file == SGE Job ID / Name (set in JOB_ID / JOB_NAME environment variable)
  • Frame in scene file == SGE Task ID (set in SGE_TASK_ID environment variable)

When a job is scheduled to run on a execution node, SGE_TASK_ID environment variable will be set based on the task id in the array job. In this case, the SGE_TASK_ID will be your frame number.

You may also want to couple the submission command (qsub) with "-P" (for project), "-A" (for accounting) and "-N" (for job name) to help you streamline other activities such as accounting, monitoring, ...

The images generated from the rendering can easily consists of hundred of files with a total file size of MB or GB. It would be too troublesome to download individual images and 'brandwidth unfriendly' if they are not compressed. I explored a number of ways to effectively accomplish this:

  1. Write a "epilog" script in the SGE queue configuration (qconf -mq all.q) so that it will run at the end of each job. However, once the array job is run in the queue, we have no control which task will complete first. We cannot assume the last frame will be rendered last. The script has to be smart enough to know that your current task is indeed the last task in the queue. In my newly revamped cluster, I ensure the SGE job name is unique and therefore I can easily identify the scene file from qstat -xml
    njob=`SGE_ROOT=/gridware/sge /gridware/sge/bin/lx24-amd64/qstat -xml | grep -c "<JB_name>$SGE_JOB_NAME</JB_name>"`
    if [ $njob == 1 ]; then
     cd /san/renderImages
     tar cf $SGE_JOB_NAME.tar ./$SGE_JOB_NAME
     gzip $SGE_JOB_NAME.tar
    fi
    
  2. Another method is to take advantage of SGE job dependency. First job to do the rendering and the second job to do the compression. The dependency is that second job will only start if first job finishes.
  3. Run a cron job in the server where the storage is directly attached. The script need to ensure it will not compress if the job is still running. Since the script will list out all the directories in reverse chronological order, we can terminate the loop once the directory has been processed before.
    #! /bin/sh
    
    
    dir="/san/renderImages"
    
    cd $dir
    for i in `ls -1td scenefile*`
    do
            # still running
            SGE_ROOT=/gridware/sge /gridware/sge/bin/lx24-amd64/qstat -xml | grep "<JB_name>$i</JB_name>" > /dev/null
            if [ $? -eq 0 ]; then
                    continue
            fi
    
    
            # previously compressed; we can break from here
            tarfile="$i.tar"
            if [ -f "$tarfile.gz" ]; then
                    break
            fi
    
            # compress now
            tar cf $tarfile ./$i
            gzip $tarfile
    
    done
    

I opted for the last method because I can do the 'tar' and 'gzip' in the server with direct attached storage, no network traffic incurred. Also, it has no additional SGE job unless the second method. The first and second methods will have to run the compression in the execution node and it has to read and write over the network.

BTW, the output of qstat -xml looks like this:

<?xml version='1.0'?>
<job_info  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <queue_info>
    <job_list state="running">
      <JB_job_number>572</JB_job_number>
      <JAT_prio>0.55500</JAT_prio>
      <JB_name>scenefile_001</JB_name>
      <JB_owner>renderer</JB_owner>
      <state>r</state>
      <JAT_start_time>2008-04-29T10:48:11</JAT_start_time>
      <queue_name>all.q@l1</queue_name>
      <slots>1</slots>
      <tasks>9</tasks>
    </job_list>
    <job_list state="running">
      <JB_job_number>570</JB_job_number>
      <JAT_prio>0.55500</JAT_prio>
      <JB_name>scenefile_002</JB_name>
      <JB_owner>renderer</JB_owner>
      <state>r</state>
      <JAT_start_time>2008-04-29T10:47:16</JAT_start_time>
      <queue_name>all.q@l10</queue_name>
      <slots>1</slots>
      <tasks>68</tasks>
    </job_list>
 ...

Labels: ,

Saturday, April 26, 2008

Windows Maximum Path Length

I am currently running some grid jobs on my newly revamped Linux cluster running CentOS 5.1 64 bit. Somehow my customer's program generated files deep in a directory with at least 4 level. It created a lot of problem when I tried to copy to Windows via Filezilla. Those failed FTP transfer are having a path length longer than 256.

To avoid all these downstream activities in Windows, I 'move' all these files to there respective base directory and delete all these directories. In UNIX, you can achieve that with 'find'

find . -type f -exec mv {} . \;
find . -type d -depth rmdir {} \;

First find is to locate all the files (no directory) and execute a 'mv' to move the file to current directory. Second find is to locate all the directories, with the trick that we locate directory with depth first. With this, we are able to delete the deeper directory in the hierarchy. BTW, the second find will generate error 'cos the find also locate itself (".") and rmdir will try to remove it. It will fail 'cos you are not allow to remove itself or non-empty directory. Although it will not do any harm, you may want to avoid error by putting find in a loop and skip the current directory

for i in `find . -type d -depth`
do
    [ "$i" = "." ] || rmdir $i
done

Labels:

Friday, April 25, 2008

Ubuntu on my old notebook

Downloaded the latest Ubuntu 8.04 this morning and installed on my old Toshiba notebook. Now my notebook has Windows XP, Fedora Core 8 and Ubuntu 8.04. Actually I wanted to reserve a partition to install Solaris, but it just refused to recognise my keyboard. How can one install an operating system without a keyboard.

Labels: ,

Thursday, April 24, 2008

Hadoop Summit and Data-Intensive Computing Symposium Videos and Slides

Lots of videos and slides related to scalability and performance, definitely worth your time.

Hadoop Summit and Data-Intensive Computing Symposium Videos and Slides

Hadoop Summit - March 25, 2008 - The Hadoop Summit brought together leaders from the Hadoop developer and user communities for the first time. Apache Hadoop, an open-source distributed computing project of the Apache Software Foundation, is a distributed file system and parallel execution environment that enables its users to process massive amounts of data. Slides from the presentations are available below, and video will be coming soon.

  1. Hadoop Overview: Doug Cutting / Eric Baldeschwieler, Yahoo! - Slides - Video
  2. Pig: Chris Olston, Yahoo! - Slides
  3. JAQL: Kevin Beyer, IBM - Slides - Video
  4. DryadLINQ: Michael Isard, Microsoft - Slides - Video
  5. Monitoring Hadoop using X-Trace: Andy Konwinski, UC Berkeley - Slides - Video
  6. Zookeeper: Ben Reed, Yahoo! - Slides - Video
  7. Hbase: Michael Stack, Powerset - Slides - Video
  8. Hbase at Rapleaf: Bryan Duxbury, Rapleaf - Slides - Video
  9. Hive: Joydeep Sen Sarma / Ashish Thusoo, Facebook - Slides - Video
  10. GrepTheWeb- Hadoop on AWS: Jinesh Varia, Amazon - Slides - Video
  11. Building Ground Models of Southern California: Steve Schlosser / David O'Hallaron, Intel / CMU - Slides - Video
  12. Online search for engineering design content: Mike Haley, Autodesk - Slides - Video
  13. Yahoo – Webmap: Christian Kunz, Yahoo! - Slides - Video
  14. Natural language Processing: Jimmy Lin, U of Maryland / Christophe Bisciglia, Google - Slides - Video
  15. Panel on future directions: Sameer Paranjpye, Sanjay Radia, Owen O’Malley (Yahoo), Chad Walters (Powerset), Jeff Eastman (Mahout) - Video


Data-Intensive Computing Symposium - March 26, 2008 - Hosted by Yahoo! and the CCC, the Data-Intensive Computing Symposium brought together experts in system design, programming, parallel algorithms, data management, scientific applications, and information-based applications to better understand existing capabilities in the development and application of large-scale computing systems, and to explore future opportunities. Slides from the presentations are available below, and video will be coming soon.

  1. Data-Intensive Scalable Computing: Randy Bryant, Carnegie Mellon - Slides - Video
  2. Text Information Management: Challenges and Opportunities: ChengXiang Zhai, University of Illinois at Urbana-Champaign - Slides - Video
  3. Clouds and ManyCore: The Revolution: Dan Reed, Microsoft Research - Slides - Video
  4. Computational Paradigms for Genomic Medicine: Jill Mesirov, Broad Institute of MIT and Harvard - Video
  5. Simplicity and Complexity in Data Systems at Scale: Garth Gibson, Carnegie Mellon - Slides - Video
  6. Handling Large Datasets at Google: Current Systems and Future Directions: Jeff Dean, Google - Slides - Video
  7. Algorithmic Perspectives on Large-Scale Social Network Data: Jon Kleinberg, Cornell - Slides - Video
  8. Mining the Web Graph: Marc Najork, Microsoft Research - Slides - Video
  9. "What" Goes Around: Joe Hellerstein, U.C. Berkeley - Video
  10. Sherpa: Hosted Data Serving: Raghu Ramakrishnan, Yahoo! Research - Slides - Video
  11. Scientific Applications of Large Databases: Alex Szalay, Johns Hopkins - Slides - Video
  12. Data-Rich Computing: Where It's At : Phil Gibbons, Intel Research - Slides - Video
  13. NSF Plans for Supporting Data Intensive Computing: Jeannette Wing, NSF - Slides - Video
  14. The Google/IBM data center: Christophe Bisciglia, Google - Video
  15. The Computing Community Consortium: Stimulating Bigger Thinking: Ed Lazowska, University of Washington and CCC - Slides

Labels:

Friday, April 18, 2008

ZFS Rocks

I am a great fan of ZFS. With Sun Fire X4500, you will see the benefit of having ZFS to virtualise all the disks and have the best protection without having to worry about lost of data like silent data corruption. BTW, I blogged about ZFS and X4500 a lot in the past. With ZFS and OpenSolaris, Sun Microsystems is trying to make Solaris as the platform for storage.

Those interested to implement ZFS should read this ZFS Best Practice Guide. As for most of my implementations, I prefer to do it this way,

  • use 2 disks (from different controller) to be mirrored with SVM (Solaris Volume Manager)
  • the rest of the 46 disks in one zpool
  • raidz (raid-5 equivalent) group of 6 disks across all the controllers
  • strip across 7 raidz groups
  • 4 disks as spare
  • you will get 16TB usable space in a 24TB raw box
  • weekly zfs scrub in crontab to examine all data and verify checksum
BTW, Sun Fire X4500 has 6 controllers and the below is my disk layout.


If you configure probably, you don't even have to open up the box to replace any hard disk for the life time of the Sun Fire X4500, hopefully.

Below videos will definitely show off ZFS:
ZFS is Smashing Baby

CSI:Munich

Labels: , ,

Thursday, April 17, 2008

Bastille tried to use $GLOBAL_BIN{'ping6'} but it does not exist

A colleague of mine was having problem running Bastille to harden his RedHat Linux server. He keeps getting the following error message and cannot find any solution on the web.
ERROR: Bastille tried to use $GLOBAL_BIN{'ping6'} but it does not exist.

I heard about Bastille but have no clue how it works. Anyway, here is how I managed to provide a workaround to avoid the above error message by explicitly define the fullpath of ping6.

bastille is actually a shell script that eventually launch a Perl script to run through the hardening steps. At the beginning of the bastille script, it defines PATH and PERL5LIB. PERL5LIB is the place to explore all the other bastille perl modules (.pm). Search through all the PM files for string "GLOBAL_BIN" and I found the following files that contain the string: API.pm, API.PM.sweth and OSX_API.PM

$ which ping6
/bin/ping6


$ cat /usr/sbin/bastille
...
...
export PATH="/bin:/usr/bin:/usr/sbin:$PATH"
export PERL5LIB="$PERL5LIB:/opt/sec_mgmt/bastille/lib:/usr/lib/perl5/site_perl:/usr/lib/Bastille:/usr/lib/perl5/site_perl/5.6.0/i386-linux"
...
...

$ cd /usr/lib/Bastille
$ grep GLOBAL_BIN *
API.pm:       %GLOBAL_BIN %GLOBAL_DIR %GLOBAL_FILE
API.pm:    my %map = ("BIN"   => \%GLOBAL_BIN,
API.pm:     # type relates to the map above, type bin will map to GLOBAL_BIN
API.pm:    foreach my $hashref (\%GLOBAL_BIN,\%GLOBAL_DIR,\%GLOBAL_FILE,\%GLOBAL_BFILE,\%GLOBAL_BDIR) {
API.pm:  my %map = ("BIN"   => \%GLOBAL_BIN,
API.pm:    # i.e. Bastille tried to use $GLOBAL_BIN{'cp'} but it does not exist.
API.pm.sweth:   %GLOBAL_BIN %GLOBAL_DIR %GLOBAL_FILE %GLOBAL_MISC
API.pm.sweth:         $GLOBAL_BIN{$DISTRO_FILE} = $PATH_TEMP;
API.pm.sweth:         `$GLOBAL_BIN{"cp"} -p $file $backup_file`;
API.pm.sweth:         $return=`$GLOBAL_BIN{"mknod"} $prefix $file $suffix`;
OSX_API.pm:    $GLOBAL_BIN{"md5sum"}="/usr/bin/md5sum";

In the OSX_API.pm, it is likely a PM meant for MacOSX. It also hard-coded the md5sum fullpath in the GLOBAL_BIN associative array. Hey, likely I can do the same for ping6 in the API.pm. My instinct tells me that API.pm is the generic API module and likely to be the entry point for bastille.

I explicitly define the full path of ping6 right after the package definition as shown below. Guess what, bastille runs without any error. I won't say the problem is solved, but at least I provided a workaround.

$ cat API.pm
...
...
package Bastille::API;
$GLOBAL_BIN{'ping6'}="/bin/ping6";

...
...

Labels: ,

Wednesday, April 16, 2008

/var/log/messages and Security, Part 2

This morning I blogged about the how to find out what ip addresses the hackers are trying to ssh into your server. In order to be more proactive, I implemented this auto-include script to run as a cron job so that I can automatically include those ip addresses into the /etc/hosts.deny if they are not already there.
#! /bin/sh
# automatically include the top 10 ip addressess that deny ssh login logged in lastb


for i in `lastb -i | awk '$2~/^ssh:/{++s[$3]}END{for(i in s){print i,s[i]}}' | sort -n -k 2 -r | head | cut -f1 -d" "`
do
        egrep ":$i$|:$i,|,$i,|,$i$" /etc/hosts.deny > /dev/null 2>&1 || echo "sshd:$i" >> /etc/hosts.deny
done

Labels: , ,

/var/log/messages and Security

Yesterday my colleague was asking me how to summarise the /var/log/messages in Linux. Below is a typical output of /var/log/messages:
Apr 13 04:06:41 myserver syslogd 1.4.1: restart.
Apr 13 04:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed 2 Celsius to 31 Celsius since last report
Apr 13 13:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 29 Celsius since last report
Apr 15 04:02:42 myserver webalizer: gethostby*.getanswer: asked for "194.216.80.218.in-addr.arpa IN PTR", got type "A"
Apr 15 04:37:23 myserver auditd[1899]: Audit daemon rotating log files
Apr 15 12:00:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed 2 Celsius to 31 Celsius since last report
Apr 15 14:00:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 29 Celsius since last report
Apr 15 14:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 27 Celsius since last report
Apr 15 14:30:23 myserver smartd[2483]: Device: /dev/sdb, Temperature changed -5 Celsius to 24 Celsius since last report
Apr 15 16:30:23 myserver smartd[2483]: Device: /dev/sdb, Temperature changed 3 Celsius to 27 Celsius since last report

Basically the fields are month, day, time, hostname, process-name, message. We can summarise the process count base on date and time. Also, we can launch the script just before mid-night to summarise that day's messages and have it email to your account. Include the below script in your crontab to run at 23:59 everyday

#! /bin/sh


month=`date '+%b'`
day=`date '+%d'`

# cron at 23:59 and sleep for 1 min in order to collect a complete messages for the day
sleep 60

awk -v m=$month -v d=$day '
$1==m && $2==d {
        split($3,t,":")
        split($5,p,"[:\(\[]")
        ind=sprintf("%s %s:00:00",p[1],t[1])
        ++s[ind]
}
END {
        for(i in s) {
                print i, s[i]
        }
}' /var/log/messages 2>/dev/null | sort | \
mailx -s "Summary of /var/log/messages" yourname@yourcompany.com

You should get an email everyday in this format:

auditd 04:00:00 1
smartd 12:00:00 1
smartd 14:00:00 3
smartd 16:00:00 1
smartd 17:00:00 2
smartd 19:00:00 1
smartd 23:00:00 1
webalizer 04:00:00 1

After running this script, my colleague realised that there were lots of sshd processes with deny access. In Linux, you can track down the bad login using lastb command. However, this is not setup by default and you need to touch /var/log/btmp to activate that. Once that is done, you will get output from lastb like below: (Note, ip and usernames are masked for secuity reason)

# lastb | head
xxuser    ssh:notty    xxx.166.139.yyy   Wed Apr 16 07:50 - 07:50  (00:00)
rxxot     ssh:notty    xxx.124.185.yyy   Wed Apr 16 04:14 - 04:14  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)

What we can do is to find out the top 10 'hacker' ip addresses so that we can include that in the /etc/hosts.deny file to block off the bad guys. We can also find out what username hackers like to try and ensure these accounts do not exist or protected with strong password.

# lastb | awk '/Apr/{++s[$3]}END{for(i in s){print i,s[i]}}' | sort -n -k 2 -r | head -10
xxx.191.254.yyy 1178
xxx.208.50.yyy 1106
xxx.108.131.yyy 1043
xxx.29.152.yyy 957
xxx.220.217.yyy 295
xxx.68.36.yyy 295
xxx.196.13.yyy 295
xxx.194.55.yyy 295
xxx.118.166.yyy 295
xxx.130.201.yyy 265
# lastb | awk '{++s[$1]}END{for(i in s){print i,s[i]}}' | sort -n -k 2 -r | head -10
root 19230
admin 3704
test 3290
guest 1442
webmaste 1146
user 1014
oracle 894
info 716
postgres 618
ftpuser 604

Labels: , ,

Tuesday, April 15, 2008

VirtualBox

Recently I brought a new Compaq Presario B1230 notebook with 4GB memory, but Windows Vista Home Premium can only see 3GB. Anyway, it is more than sufficient for me to try out VirtualBox. VirtualBox was developed by innotek but acquired by Sun Microsystems in Feb 2008.

It is pretty easy to use and configure. I managed to install Fedora Core 8 in just under half an hour. Below shows virtualbox in action:



Labels:

Quote

This was taken during my recent overseas trip and I can relate this wisdom with my recent encounter with one of our sales folk. Sorry, I am not going to tell you the story :-(

Anyway, the words on the stone is as follow:

BIMAL K. BACHHAWAT BLOCK

"ONE NEED NOT BE ASHAMED TO
EXPRESS IGNORANCE. IT IS HIDING
OF ONE'S IGNORANCE THAT ONE
SHOULD BE ASHAMED OF."


THESE WERE THE WORDS OF PROFESSOR BIMAL K.
BACHHAWAT (AUGUST 16, 1925 - SEPTEMBER 23, 1996) AN
OUTSTANDING SCIENTIST, INSTITUTION BUILDER, NURTURER OF
YOUNG TALENT AND ABOVE ALL A HUMAN BEING PAREXCELLENCE.

DEDICATED ON 22ND SEPTEMBER, 1997

PROFESSOR A BHAI MANSINGH
DIRECTOR SOUTH CAMPUS
PROFESSOR V.R. MEHTA
VICE CHANCELLOR


UNIVERSITY OF DELHI, DELHI

Labels:

Saturday, April 12, 2008

Rack on a Desk

Have you ever see a rack on a desk. A Sun Fire X2100M2 and Sun Fire X4600 are rack mounted in this mini 19" rack. Amusing, isn't it.


Labels:

Thursday, April 10, 2008

Number Padded With Zeros

While my colleague is doing the documentation, I am pretty free at this moment and therefore I can blog a bit.

In my previous blog, I mentioned that I am currently working on a 48-node grid cluster. The servers' naming convention is like "servername" followed by 2 digit number ranging from 01 to 48. The previous vendor created a bash shell to start daemon on all the nodes and it looked like this:

for ((i=1;i<10;++i))
do
 ssh root@servername0$i /usr/local/bin/some-daemon
done

for ((i=10;i<48;++i))
do
 ssh root@servername$i /usr/local/bin/some-daemon
done

As you can see, the above code just repeat itself. It is pretty easy to ensure the number are padded with zero and the final script does not have to repeat itself again.

for ((i=1;i<48;++i))
do
 i2=`echo $i | awk '{printf("%02d",$1)}'`
 ssh root@servername$i2 /usr/local/bin/some-daemon
done

If you are fan of Bourne shell, you can do like this. Also, I introduced a new function seq that automatically pad the number with zero

seq()
{
awk 'END{for(i='$1';i<='$2';++i){printf("%02d ",i)}}' /dev/null
}
for i2 in `seq 1 48`
do
 ssh root@servername$i2 /usr/local/bin/some-daemon
done

The seq function basically take advantage of the shell so that I the first and second arguments to the function concatenates to the entire awk code. If you were to break it down, it is something like this:

  • awk 'END{for(i='
  • $1
  • ';i<='
  • $2
  • ';++i){printf("%02d ",i)}}' /dev/null
Those in red will be interrupted by the shell before they are passed to the awk. Make sure you do not include any space in between.

Labels:

Tcl Automates Interactive Installation

I am currently working overseas for a 48-node grid cluster installation and one of the applications installation requires to be done interactively. It would be a pain in the ass to do that 48 times, right?

In the Expect (a Tcl tool/extension for automating interactive applications) source tarball, there is an example called "multixterm" which allow the user to send a single input to multiple terminals. Since all the 48 nodes are able to be ssh in from the grid master without password, I was able to launch multixterm to do my installation interactively.

Here is a sample screen dump example that I used to launch 4 xterms in the localhost.

In case you would like to see who the multixterm works, I am copying the original expect source here. Here is a good example showing you how to install and launch Expect in Debian Linux.

#!/depot/path/expectk
#
# NAME
# multixterm - drive multiple xterms separately or together
#
# SYNOPSIS
# multixterm [-xa "xterm args"]
#     [-xc "command"]
#     [-xd "directory"]
#     [-xf "file"]
#     [-xn "xterm names"]
#     [-xv] (enable verbose mode)
#     [-xh] or [-x?] (help)
#     [xterm names or user-defined args...]
#
# DESCRIPTION
# Multixterm creates multiple xterms that can be driven together
# or separately.
#
# In its simplest form, multixterm is run with no arguments and
# commands are interactively entered in the first entry field.
# Press return (or click the "new xterm" button) to create a new
# xterm running that command.
#
# Keystrokes in the "stdin window" are redirected to all xterms
# started by multixterm.  xterms may be driven separately simply
# by focusing on them.
#
# The stdin window must have the focus for keystrokes to be sent
# to the xterms.  When it has the focus, the color changes to
# aquamarine.  As characters are entered, the color changes to
# green for a second.  This provides feedback since characters
# are not echoed in the stdin window.
#
# Typing in the stdin window while holding down the alt or meta
# keys sends an escape character before the typed characters.
# This provides support for programs such as emacs.
#
# ARGUMENTS
# The optional -xa argument indicates arguments to pass to
# xterm.
#
# The optional -xc argument indicates a command to be run in
# each named xterm (see -xn).  With no -xc argument, the command
# is the current shell.
#
# The optional -xd argument indicates a directory to search for
# files that will appear in the Files menu.  By default, the
# directory is: ~/lib/multixterm
#
# The optional -xf argument indicates a file to be read at
# startup.  See FILES below for more info.
#
# The optional -xn argument indicates a name for each xterm.
# This name will also be substituted for any %n in the command
# argument (see -xc).
#
# The optional -xv flag puts multixterm into a verbose mode
# where it will describe some of the things it is doing
# internally.  The verbose output is not intended to be
# understandable to anyone but the author.
#
# Less common options may be changed by the startup file (see
# FILES below).
#
# All the usual X and wish flags are supported (i.e., -display,
# -name).  There are so many of them that to avoid colliding and
# make them easy to remember, all the multixterm flags begin
# with -x.
#
# If any arguments do not match the flags above, the remainder
# of the command line is made available for user processing.  By
# default, the remainder is used as a list of xterm names in the
# style of -xn. The default behavior may be changed using the
# .multixtermrc file (see DOT FILE below).
#
# EXAMPLE COMMAND LINE ARGUMENTS
# The following command line starts up two xterms using ssh to
# the hosts bud and dexter.
#
#  multixterm -xc "ssh %n" bud dexter
#
# FILES
# Command files may be used to drive or initialize multixterm.
# The File menu may be used to invoke other files.  If files
# exist in the command file directory (see -xd above), they will
# appear in the File menu.  Files may also be loaded by using
# File->Open.  Any filename is acceptable but the File->Open
# browser defaults to files with a .mxt suffix.
#
# Files are written in Tcl and may change any variables or
# invoke any procedures.  The primary variables of interest are
# 'xtermCmd' which identifies the command (see -xc) and
# 'xtermNames' which is a list of names (see -xn).  The
# procedure xtermStartAll, starts xterms for each name in the
# list.  Other variables and procedures may be discovered by
# examining multixterm itself.
#
# EXAMPLE FILE
# The following file does the same thing as the earlier example
# command line:
#
#  # start two xterms connected to bud and dexter
#  set xtermCmd "ssh %n"
#  set xtermNames {bud dexter}
#  xtermStartAll
#
# DOT FILE
# At startup, multixterm reads ~/.multixtermrc if present.  This
# is similar to the command files (see FILES above) except that
# .multixtermrc may not call xtermStartAll.  Instead it is
# called implicitly, similar to the way that it is implicit in
# the command line use of -xn.
#
# The following example .multixtermrc file makes every xterm run
# ssh to the hosts named on the command line.
#
#  set xtermCmd "ssh %n"
#
# Then multixterm could be called simply:
#
#  multixterm bud dexter
#
# If any command-line argument does not match a multixterm flag,
# the remainder of the command line is made available to
# .multixtermrc in the argv variable.  If argv is non-empty when
# .multixtermrc returns, it is assigned to xtermNames unless
# xtermNames is non-empty in which case, the content of argv is
# ignored.
#
# Commands from .multixtermrc are evaluated early in the
# initialization of multixterm.  Anything that must be done late
# in the initialization (such as adding additional bindings to
# the user interface) may be done by putting the commands inside
# a procedure called "initLate".
#
# MENUS
# Except as otherwise noted, the menus are self-explanatory.
# Some of the menus have dashed lines as the first entry.
# Clicking on the dashed lines will "tear off" the menus.
#
# USAGE SUGGESTION - ALIASES AND COMMAND FILES
# Aliases may be used to store lengthy command-line invocations.
# Command files can be also be used to store such invocations
# as well as providing a convenient way to share configurations.
#
# Tcl is a general-purpose language.  Thus multixterm command
# files can be extremely flexible, such as loading hostnames
# from other programs or files that may change from day-to-day.
# In addition, command files can be used for other purposes.
# For example, command files may be used to prepared common
# canned interaction sequences.  For example, the command to
# send the same string to all xterms is:
#
#     xtermSend "a particularly long string"
#
# The File menu (torn-off) makes canned sequences particularly
# convenient.  Interactions could also be bound to a mouse
# button, keystroke, or added to a menu via the .multixtermrc
# file.
#
# USAGE SUGGESTION - HANDLING MANY XTERMS BY TILING
# The following .multixtermrc causes tiny xterms to tile across
# and down the screen.  (You may have to adjust the parameters
# for your screen.)  This can be very helpful when dealing with
# large numbers of xterms.
#
#     set yPos 0
#     set xPos 0
#
#     trace variable xtermArgs r traceArgs
#
#     proc traceArgs {args} {
#         global xPos yPos
#         set ::xtermArgs "-geometry 80x12+$xPos+$yPos -font 6x10"
#         if {$xPos} {
#      set xPos 0
#      incr yPos 145
#      if {$yPos > 800} {set yPos 0}
#         } else {
#      set xPos 500
#         }
#     }
#
# The xtermArgs variable in the code above is the variable
# corresponding to the -xa argument.
#
# xterms can be also be created directly.  The following command
# file creates three xterms overlapped horizontally:
#
#     set xPos 0
#
#     foreach name {bud dexter hotdog} {
#         set ::xtermArgs "-geometry 80x12+$xPos+0 -font 6x10"
#         set ::xtermNames $name
#         xtermStartAll
#         incr xPos 300
#     }
#
# USAGE SUGGESTION - SELECTING HOSTS BY NICKNAME
# The following .multixtermrc shows an example of changing the
# default handling of the arguments from hostnames to a filename
# containing hostnames:
#
#  set xtermNames [exec cat $argv]
#
# The following is a variation, retrieving the host names from
# the yp database:
#
#  set xtermNames [exec ypcat $argv]
#
# The following hardcodes two sets of hosts, so that you can
# call multixterm with either "cluster1" or "cluster2":
#
#  switch $argv {
#      cluster1 {
#   set xtermNames "bud dexter"
#      }
#      cluster2 {
#   set xtermNames "frank hotdog weiner"
#      }
#  }
#
# COMPARE/CONTRAST
# It is worth comparing multixterm to xkibitz.  Multixterm
# connects a separate process to each xterm.  xkibitz connects
# the same process to each xterm.
#
# LIMITATIONS
# Multixterm provides no way to remotely control scrollbars,
# resize, and most other window system related functions.
#
# Multixterm can only control new xterms that multixterm itself
# has started.
#
# As a convenience, the File menu shows a limited number of
# files.  To show all the files, use File->Open.
#
# FILES
# $DOTDIR/.multixtermrc   initial command file
# ~/.multixtermrc         fallback command file
# ~/lib/multixterm/       default command file directory
#
# BUGS
# If multixterm is killed using an uncatchable kill, the xterms
# are not killed.  This appears to be a bug in xterm itself.
#
# Send/expect sequences can be done in multixterm command files.
# However, due to the richness of the possibilities, to document
# it properly would take more time than the author has at present.
#
# REQUIREMENTS
# Requires Expect 5.36.0 or later.
# Requires Tk 8.3.3 or later.
#
# VERSION
#! $::versionString
# The latest version of multixterm is available from
# http://expect.nist.gov/example/multixterm .  If your version of Expect
# and Tk are too old (see REQUIREMENTS above), download a new version of
# Expect from http://expect.nist.gov
#
# DATE
#! $::versionDate
#
# AUTHOR
# Don Libes <don@libes.com>
#
# LICENSE
# Multixterm is in the public domain; however the author would
# appreciate acknowledgement if multixterm or parts of it or ideas from
# it are used.

######################################################################
# user-settable things - override them in the ~/.multixtermrc file
#    or via command-line options
######################################################################

set palette       #d8d8ff   ;# lavender
set colorTyping   green
set colorFocusIn  aquamarine

set xtermNames    {}
set xtermCmd      $env(SHELL)
set xtermArgs     ""
set cmdDir   ~/lib/multixterm
set inputLabel    "stdin window"

set fileMenuMax   30     ;# max number of files shown in File menu
set tearoffMenuMin 2     ;# min number of files needed to enable the File
    ;# menu to be torn off

proc initLate {} {}      ;# anything that must be done late in initialization
    ;# such as adding/modifying bindings, may be done by
    ;# redefining this

######################################################################
# end of user-settable things
######################################################################

######################################################################
# sanity checking
######################################################################

set versionString 1.8
set versionDate "2004/06/29"

package require Tcl
catch {package require Tk} ;# early versions of Tk had no package
package require Expect

proc exit1 {msg} {
    puts "multixterm: $msg"
    exit 1
}

exp_version -exit 5.36

proc tkBad {} {
    exit1 "requires Tk 8.3.3 or later but you are using Tk $::tk_patchLevel."
}

if {$tk_version < 8.3} {
    tkBad
} elseif {$tk_version == 8.3} {
    if {[lindex [split $tk_patchLevel .] 2] < 3} tkBad
}

######################################################################
# process args - has to be done first to get things like -xv working ASAP
######################################################################

# set up verbose mechanism early

set verbose 0
proc verbose {msg} {
    if {$::verbose} {
 if {[info level] > 1} {
     set proc [lindex [info level -1] 0]
 } else {
     set proc main
 }
 puts "$proc: $msg"
    }
}

# read a single argument from the command line
proc arg_read1 {var args} {
    if {0 == [llength $args]} {
 set argname -$var
    } else {
 set argname $args
    }

    upvar argv argv
    upvar $var v

    verbose "$argname"
    if {[llength $argv] < 2} {
 exit1 "$argname requires an argument"
    }

    set v [lindex $argv 1]
    verbose "set $var $v"
    set argv [lrange $argv 2 end]
}

proc xtermUsage {{msg {}}} {
    if {![string equal $msg ""]} {
 puts "multixtermrc: $msg"
    }
    puts {usage: multixterm [flags] ... where flags are:
 [-xa "xterm args"]
 [-xc "command"]
 [-xd "directory"]
 [-xf "file"]
 [-xn "xterm names"]
 [-xv] (enable verbose mode)
 [-xh] or [-x?] (help)
 [xterm names or user-defined args...]}
    exit
}

while {[llength $argv]} {
    set flag [lindex $argv 0]
    switch -- $flag -x? - -xh {
 xtermUsage
    } -xc {
 arg_read1 xtermCmd -xc
    } -xn {
 arg_read1 xtermNames -xn
    } -xa {
 arg_read1 xtermArgs -xa
    } -xf {
 arg_read1 cmdFile -xf
 if {![file exists $cmdFile]} {
     exit1 "can't read $cmdFile"
 }
    } -xd {
 arg_read1 cmdDir -xd
 if {![file exists $cmdDir]} {
     exit1 "can't read $cmdDir"
 }
    } -xv {
 set argv [lrange $argv 1 end]
 set verbose 1
 puts "main: verbose on"
    } default {
 verbose "remaining args: $argv"
 break ;# let user handle remaining args later
    }
}

######################################################################
# determine and load rc file -  has to be done now so that widgets
#  can be affected
######################################################################

# if user has no $DOTDIR, fall back to home directory
if {![info exists env(DOTDIR)]} {
    set env(DOTDIR) ~
}
# catch bogus DOTDIR, otherwise glob will lose the bogus directory
# and it won't appear in the error msg
if {[catch {glob $env(DOTDIR)} dotdir]} {
    exit1 "$env(DOTDIR)/.multixtermrc can't be found because $env(DOTDIR) doesn't exist or can't be read"
} 
set rcFile $dotdir/.multixtermrc

set fileTypes {
    {{Multixterm Files} *.mxt}
    {{All Files} *}
}

proc openFile {{fn {}}} {
    verbose "opening $fn"
    if {[string equal $fn ""]} {
 set fn [tk_getOpenFile \
      -initialdir $::cmdDir \
      -filetypes $::fileTypes \
      -title "multixterm file"]
 if {[string match $fn ""]} return
    }
    uplevel #0 source [list $fn]
    verbose "xtermNames = \"$::xtermNames\""
    verbose "xtermCmd = $::xtermCmd"
}

if {[file exists $rcFile]} {
    openFile $rcFile
} else {
    verbose "$rcFile: not found"
}

if {![string equal "" $argv]} {
    if {[string equal $xtermNames ""]} {
 set xtermNames $argv
    }
}

######################################################################
# Describe and initialize some important globals
######################################################################

# ::activeList and ::activeArray both track which xterms to send
# (common) keystrokes to.  Each element in activeArray is connected to
# the active menu.  The list version is just a convenience making the
# send function easier/faster.

set activeList {}

# ::names is an array of xterm names indexed by process spawn ids.

set names(x) ""
unset names(x)

# ::xtermSid is an array of xterm spawn ids indexed by process spawn ids.
# ::xtermPid is an array of xterm pids indexed by process spawn id.

######################################################################
# create an xterm and establish connections
######################################################################

proc xtermStart {cmd name} {
    verbose "starting new xterm running $cmd with name $name"

    ######################################################################
    # create pty for xterm
    ######################################################################
    set pid [spawn -noecho -pty]
    verbose "spawn -pty: pid = $pid, spawn_id = $spawn_id"
    set sidXterm $spawn_id
    stty raw -echo < $spawn_out(slave,name)

    regexp ".*(.)(.)" $spawn_out(slave,name) dummy c1 c2
    if {[string compare $c1 "/"] == 0} {
 set c1 0
    }

    ######################################################################
    # prepare to start xterm by making sure xterm name is unique
    # X doesn't care but active menu won't make sense unless names are unique
    ######################################################################
    set unique 1
    foreach oldName [array names ::names] {
 if {[string match "$name" $::names($oldName)]} {
     set unique 0
 }
    }
    verbose "uniqueness of $name: $unique"

    set safe [safe $name]

    # if not unique, look at the numerical suffixes of all matching
    # names, find the biggest and increment it
    if {!$unique} {
 set suffix 2
 foreach oldName [array names ::names] {
     verbose "regexp ^[set safe](\[0-9]+)$ $::names($oldName) X num"
     if {[regexp "^[set safe](\[0-9]+)$" $::names($oldName) X num]} {
  verbose "matched, checking suffix"
  if {$num >= $suffix} {
      set suffix [expr $num+1]
      verbose "new suffix: $suffix"
  }
     }
 }
 append name $suffix
 verbose "new name: $name"
    }

    ######################################################################
    # start new xterm
    ######################################################################
    set xtermpid [eval exec xterm -name [list $name] -S$c1$c2$spawn_out(slave,fd) $::xtermArgs &]
    verbose "xterm: pid = $xtermpid"
    close -slave

    # xterm first sends back window id, save in environment so it can be
    # passed on to the new process
    log_user 0
    expect {
 eof {wait;return}
 -re (.*)\n {
     # convert hex to decimal
     # note quotes must be used here to avoid diagnostic from expr
     set ::env(WINDOWID) [expr "0x$expect_out(1,string)"]
 }
    }

    ######################################################################
    # start new process
    ######################################################################
    set pid [eval spawn -noecho $cmd]
    verbose "$cmd: pid = $pid, spawn_id = $spawn_id"
    set sidCmd $spawn_id
    lappend ::activeList $sidCmd
    set ::activeArray($sidCmd) 1

    ######################################################################
    # link everything back to spawn id of new process
    ######################################################################
    set ::xtermSid($sidCmd) $sidXterm
    set ::names($sidCmd)    $name
    set ::xtermPid($sidCmd) $xtermpid

    ######################################################################
    # connect proc output to xterm output
    # connect xterm input to proc input
    ######################################################################
    expect_background {
 -i $sidCmd
 -re ".+" [list sendTo $sidXterm]
 eof [list xtermKill $sidCmd]
 -i $sidXterm
 -re ".+" [list sendTo $sidCmd]
 eof [list xtermKill $sidCmd]
    }

    .m.e entryconfig Active -state normal
    .m.e.active add checkbutton -label $name -variable activeArray($sidCmd) \
 -command [list xtermActiveUpdate $sidCmd]
    set ::activeArray($sidCmd) 1
}

proc xtermActiveUpdate {sid} {
    if {$::activeArray($sid)} {
 verbose "activating $sid"
    } else {
 verbose "deactivating $sid"
    }
    activeListUpdate
}

proc activeListUpdate {} {
    set ::activeList {}
    foreach n [array names ::activeArray] {
 if {$::activeArray($n)} {
     lappend ::activeList $n
 }
    }
}

# make a string safe to go through regexp
proc safe {s} {
    string map {{[} {\[} {*} {\*} {+} {\+} {^} {\^} {$} {\\$}} $s
}

# utility to map xterm name to spawn id
# multixterm doesn't use this but a user might want to
proc xtermGet {name} {
    foreach sid [array names ::names] {
 if {[string equal $name $::names($sid)]} {
     return $sid
 }
    }
    error "no such term with name: $name"
}

# utility to activate an xterm
# multixterm doesn't use this but a user might want to
proc xtermActivate {sid} {
    set ::activeArray($sid) 1
    xtermActiveUpdate $sid
}

# utility to deactivate an xterm
# multixterm doesn't use this but a user might want to
proc xtermDeactivate {sid} {
    set ::activeArray($sid) 0
    xtermActiveUpdate $sid
}

# utility to do an explicit Expect
# multixterm doesn't use this but a user might want to
proc xtermExpect {args} {
    # check if explicit spawn_id in args
    for {set i 0} {$i < [llength $args]} {incr i} {
 switch -- [lindex $args $i] "-i" {
     set sidCmd [lindex $args [incr i]]
     break
 }
    }

    if {![info exists sidCmd]} {
 # nothing explicit, so get it from the environment

 upvar spawn_id spawn_id

 # mimic expect's normal behavior in obtaining spawn_id
 if {[info exists spawn_id]} {
     set sidCmd $spawn_id
 } else {
     set sidCmd $::spawn_id
 }
    }

    # turn off bg expect, do fg expect, then re-enable bg expect

    expect_background -i $sidCmd ;# disable bg expect
    eval expect $args   ;# fg expect
     ;# reenable bg expect
    expect_background {
 -i $sidCmd
 -re ".+" [list sendTo $::xtermSid($sidCmd)]
 eof [list xtermKill $sidCmd]
    }
}

######################################################################
# connect main window keystrokes to all xterms
######################################################################
proc xtermSend {A} {
    if {[info exists ::afterId]} {
 after cancel $::afterId
    }
    .input config -bg $::colorTyping
    set ::afterId [after 1000 {.input config -bg $colorCurrent}]

    exp_send -raw -i $::activeList -- $A
}

proc sendTo {to} {
    exp_send -raw -i $to -- $::expect_out(buffer)
}

# catch the case where there's no selection
proc xtermPaste {} {catch {xtermSend [selection get]}}

######################################################################
# clean up an individual process death or xterm death
######################################################################
proc xtermKill {s} {
    verbose "killing xterm $s"

    if {![info exists ::xtermPid($s)]} {
 verbose "too late, already dead"
 return
    }

    catch {exec /bin/kill -9 $::xtermPid($s)}
    unset ::xtermPid($s)

    # remove sid from activeList
    verbose "removing $s from active array"
    catch {unset ::activeArray($s)}
    activeListUpdate

    verbose "removing from background handler $s"
    catch {expect_background -i $s}
    verbose "removing from background handler $::xtermSid($s)"
    catch {expect_background -i $::xtermSid($s)}
    verbose "closing proc"
    catch {close -i $s}
    verbose "closing xterm"
    catch {close -i $::xtermSid($s)}
    verbose "waiting on proc"
    wait -i $s
    wait -i $::xtermSid($s)
    verbose "done waiting"
    unset ::xtermSid($s)

    # remove from active menu
    verbose "deleting active menu entry $::names($s)"

    # figure out which it is
    # avoid using name as an index since we haven't gone to any pains to
    # make it safely interpreted by index-pattern code.  instead step
    # through, doing the comparison ourselves
    set last [.m.e.active index last]
    # skip over tearoff
    for {set i 1} {$i <= $last} {incr i} {
 if {![catch {.m.e.active entrycget $i -label} label]} {
     if {[string equal $label $::names($s)]} break
 }
    }
    .m.e.active delete $i
    unset ::names($s)

    # if none left, disable menu
    # this leaves tearoff clone but that seems reasonable
    if {0 == [llength [array names ::xtermSid]]} {
 .m.e entryconfig Active -state disable
    }
}

######################################################################
# create windows
######################################################################
tk_setPalette $palette

menu .m -tearoff 0
.m add cascade -menu .m.f    -label "File" -underline 0
.m add cascade -menu .m.e    -label "Edit" -underline 0
.m add cascade -menu .m.help -label "Help" -underline 0
set files [glob -nocomplain $cmdDir/*]
set filesLength [llength $files]
if {$filesLength >= $tearoffMenuMin} {
    set filesTearoff 1
} else {
    set filesTearoff 0
}
menu .m.f    -tearoff $filesTearoff -title "multixterm files"
menu .m.e    -tearoff 0
menu .m.help -tearoff 0
.m.f    add command -label Open -command openFile -underline 0

if {$filesLength} {
    .m.f add separator
    set files [lsort $files]
    set files [lrange $files 0 $fileMenuMax]
    foreach f $files {
 .m.f add command -label $f -command [list openFile $f]
    }
    .m.f add separator
}

.m.f    add command -label "Exit"     -command exit       -underline 0
.m.e    add command -label "Paste"    -command xtermPaste -underline 0
.m.e add cascade -label "Active"   -menu .m.e.active   -underline 0
.m.help add command -label "About"    -command about      -underline 0
.m.help add command -label "Man Page" -command help       -underline 0
. config -m .m

menu .m.e.active -tearoff 1 -title "multixterm active"
.m.e entryconfig Active -state disabled
# disable the Active menu simply because it looks goofy seeing an empty menu
# for consistency, though, it should be enabled

entry  .input -textvar inputLabel -justify center -state disabled
entry  .cmd   -textvar xtermCmd
button .exec  -text "new xterm" -command {xtermStart $xtermCmd $xtermCmd}

grid .input -sticky ewns
grid .cmd   -sticky ew
grid .exec  -sticky ew -ipadx 3 -ipady 3

grid columnconfigure . 0 -weight 1
grid    rowconfigure . 0 -weight 1  ;# let input window only expand

bind .cmd   <Return>        {xtermStart $xtermCmd $xtermCmd}

# send all keypresses to xterm 
bind .input <KeyPress>         {xtermSend %A ; break}
bind .input <Alt-KeyPress>     {xtermSend \033%A; break}
bind .input <Meta-KeyPress>    {xtermSend \033%A; break}
bind .input <<Paste>>          {xtermPaste ; break}
bind .input <<PasteSelection>> {xtermPaste ; break}

# arrow keys - note that if they've been rebound through .Xdefaults
# you'll have to change these definitions.
bind .input <Up>    {xtermSend \033OA; break}
bind .input <Down>  {xtermSend \033OB; break}
bind .input <Right> {xtermSend \033OC; break}
bind .input <Left>  {xtermSend \033OD; break}
# Strange: od -c reports these as \033[A et al but when keypad mode
# is initialized, they send \033OA et al.  Presuming most people
# want keypad mode, I'll go with the O versions.  Perhaps the other
# version is just a Sun-ism anyway.

set colorCurrent [.input cget -bg]
set colorFocusOut $colorCurrent

# change color to show focus
bind .input <FocusOut> colorFocusOut
bind .input <FocusIn>  colorFocusIn
proc colorFocusIn  {} {.input config -bg [set ::colorCurrent $::colorFocusIn]}
proc colorFocusOut {} {.input config -bg [set ::colorCurrent $::colorFocusOut]}

# convert normal mouse events to focusIn
bind .input <1>       {focus .input; break}
bind .input <Shift-1> {focus .input; break}

# ignore all other mouse events that might make selection visible
bind .input <Double-1>  break
bind .input <Triple-1>  break
bind .input <B1-Motion> break
bind .input <B2-Motion> break

set scriptName [info script] ;# must get while it's active

proc about {} {
    set w .about
    if {[winfo exists $w]} {
 wm deiconify $w
 raise $w
 return
    }
    toplevel     $w
    wm title     $w "about multixterm"
    wm iconname  $w "about multixterm"
    wm resizable $w 0 0

    button $w.b -text Dismiss -command [list wm withdraw $w]

    label $w.title -text "multixterm" -font "Times 16" -borderwidth 10 -fg red
    label $w.version -text "Version $::versionString, Released $::versionDate"
    label $w.author -text "Written by Don Libes <don@libes.com>"
    label $w.using -text "Using Expect [exp_version],\
                                Tcl $::tcl_patchLevel,\
                                Tk $::tk_patchLevel"
    grid $w.title
    grid $w.version
    grid $w.author
    grid $w.using
    grid $w.b -sticky ew
}

proc help {} {
    if {[winfo exists .help]} {
 wm deiconify .help
 raise .help
 return
    }
    toplevel    .help
    wm title    .help "multixterm help"
    wm iconname .help "multixterm help"

    scrollbar .help.sb -command {.help.text yview}
    text .help.text -width 74 -height 30 -yscroll {.help.sb set} -wrap word

    button .help.ok -text Dismiss -command {destroy .help} -relief raised
    bind .help <Return> {destroy .help;break}
    grid .help.sb   -row 0 -column 0     -sticky ns
    grid .help.text -row 0 -column 1     -sticky nsew
    grid .help.ok   -row 1 -columnspan 2 -sticky ew -ipadx 3 -ipady 3

    # let text box only expand
    grid rowconfigure    .help 0 -weight 1
    grid columnconfigure .help 1 -weight 1

    set script [auto_execok $::scriptName]
    if {[llength $script] == 0} {
 set script /depot/tcl/bin/multixterm     ;# fallback
    }
    if {[catch {open $script} fid]} {
 .help.text insert end "Could not open help file: $script"
    } else {
 # skip to the beginning of the actual help (starts with "NAME")
 while {-1 != [gets $fid buf]} {
     if {1 == [regexp "NAME" $buf]} {
  .help.text insert end "\n NAME\n"
  break
     }
 }
 
 while {-1 != [gets $fid buf]} {
     if {0 == [regexp "^#(.?)(.*)" $buf X key buf]} break
     if {$key == "!"} {
  set buf [subst -nocommands $buf]
  set key " "
     }
     .help.text insert end $key$buf\n
 }
    }

    # support scrolling beyond Tk's built-in Next/Previous
    foreach w {"" .sb .text .ok} {
 set W .help$w
 bind $W <space>  {scrollPage  1}  ;#more
 bind $W <Delete>  {scrollPage -1}  ;#more
 bind $W <BackSpace>  {scrollPage -1}  ;#more
 bind $W <Control-v> {scrollPage  1}  ;#emacs
 bind $W <Meta-v> {scrollPage -1}  ;#emacs
 bind $W <Control-f> {scrollPage  1}  ;#vi
 bind $W <Control-b> {scrollPage -1}  ;#vi
 bind $W <F35>  {scrollPage  1}  ;#sun
 bind $W <F29>  {scrollPage -1}  ;#sun
 bind $W <Down>         {scrollLine  1}
 bind $W <Up>  {scrollLine -1}
    }
}

proc scrollPage {dir} {
    tkScrollByPages .help.sb v $dir
    return -code break
}

proc scrollLine {dir} {
    tkScrollByUnits .help.sb v $dir
    return -code break
}

######################################################################
# exit handling
######################################################################

# xtermKillAll is not intended to be user-callable.  It just kills
# the processes and that's it. A user-callable version would update
# the data structures, close the channels, etc.

proc xtermKillAll {} {
    foreach sid [array names ::xtermPid] {
 exec /bin/kill -9 $::xtermPid($sid)
    }
}

rename exit _exit
proc exit {{x 0}} {xtermKillAll;_exit $x}

wm protocol . WM_DELETE_WINDOW exit
trap exit SIGINT

######################################################################
# start any xterms requested
######################################################################
proc xtermStartAll {} {
    verbose "xtermNames = \"$::xtermNames\""
    foreach n $::xtermNames {
 regsub -all "%n" $::xtermCmd $n cmdOut
 xtermStart $cmdOut $n
    }
    set ::xtermNames {}
}

initLate

# now that xtermStartAll and its accompanying support has been set up
# run it to start anything defined by rc file or command-line args.

xtermStartAll     ;# If nothing has been requested, this is a no-op.

# finally do any explicit command file
if {[info exists cmdFile]} {
    openFile $cmdFile
}

puts hello

Tcl and UNIX scripting saved the day again.

Labels:

Wednesday, April 02, 2008

Web Service

Bash Curses Cancer recently posted an article, Exposing Command Line Programs As Web Services, which I found it both interesting and useful. The web service web server script, to_web.py, is written in Python. It contains only 53 lines of code excluding blank lines. The example in the web site showed how you can expose "sort" and "gzip" to make wonder. Since "sort" and "gzip" are mostly available in most of the UNIXes, I think the concept can be adopted so that we can expose some "heavy weight" command line application as web service.

Octave immediately came to my mind. Quoted from the Octave web site: "GNU Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language.".

Let's start up the web service server on my Cygwin

$ python -V
Python 2.5.1

$ uname -a
CYGWIN_NT-5.1 CHCHAN 1.5.25(0.156/4/2) 2007-12-14 19:21 i686 Cygwin

$ ./to_web.py -p8000 octave
Tue Apr  1 23:07:14 2008 octave server started - 8000
localhost - - [01/Apr/2008 23:07:37] "POST /octave+-q HTTP/1.1" 200 -
localhost - - [01/Apr/2008 23:08:12] "POST /octave+-q HTTP/1.1" 200 -
localhost - - [01/Apr/2008 23:08:50] "POST /octave+-q HTTP/1.1" 200 -
localhost - - [01/Apr/2008 23:08:57] "POST /octave+-q HTTP/1.1" 200 -

Below showed a couple of examples.

  1. Inverse of a 3x3 matrix
  2. Solving a set of equations: 2x+3y+4z=1 ; 3x+2y+z=2 ; 4x+y+3z = 3
  3. Compute product of a lower and upper triangular matrix based on LU Decomposition
  4. Find a 5x5 Magic Square
$ echo "a=[2,3,4;3,2,1;4,1,3];inv(a)" | curl --data-binary @- http://localhost:8000/octave+-q
ans =

-0.20000   0.20000   0.20000
0.20000   0.40000  -0.40000
0.20000  -0.40000   0.20000


$ echo "a=[2,3,4;3,2,1;4,1,3];b=[1;2;3];inv(a)*b" | curl --data-binary @- http://localhost:8000/octave+-q
ans =

 8.0000e-01
-2.0000e-01
-5.5511e-17


$ echo "[l,u]=lu(magic(5))" | curl --data-binary @- http://localhost:8000/octave+-q
l =

0.73913  1.00000  0.00000  0.00000  0.00000
1.00000  0.00000  0.00000  0.00000  0.00000
0.17391  0.25268  0.51637  1.00000  0.00000
0.43478  0.48394  0.72308  0.92308  1.00000
0.47826  0.76874  1.00000  0.00000  0.00000

u =

23.00000    5.00000    7.00000   14.00000   16.00000
 0.00000   20.30435   -4.17391   -2.34783    3.17391
 0.00000    0.00000   24.86081   -2.89079   -1.09208
 0.00000    0.00000    0.00000   19.65116   18.97933
 0.00000    0.00000    0.00000    0.00000  -22.22222


$ echo "magic(5)" | curl --data-binary @- http://localhost:8000/octave+-q
ans =

17  24   1   8  15
23   5   7  14  16
4   6  13  20  22
10  12  19  21   3
11  18  25   2   9

I think the possibility is endless. BTW, I started writing my own version based on Tcl and gotten the HTTP GET working. Have to find time to work on the HTTP POST. Python comes with BaseHTTPServer module that is pretty handy to build simple web server type of applications. Although Tcl has it own Tclhttpd web server, it is almost a full-blown standalone web server. Anyway, my version will be based on Tcl socket and I will try to achieve the same functionalities under 100 lines of code. So stay tune.

Labels: , ,