SGE qstat XML Stylesheet

You can query the Sun Grid Engine status with XML output using qstat -xml

<?xml version="1.0"?>
<job_info xmlns:xsd="http://www.w3.org/2001/XMLScenehema">
  <queue_info>
    <Queue-List>
      <name>all.q@l1</name>
      <qtype>BIP</qtype>
      <slots_used>1</slots_used>
      <slots_total>2</slots_total>
      <arch>lx24-amd64</arch>
      <job_list state="running">
        <JB_job_number>652</JB_job_number>
        <JAT_prio>0.55500</JAT_prio>
        <JB_name>Scene149_V01</JB_name>
        <JB_owner>renderer</JB_owner>
        <state>r</state>
        <JAT_start_time>2008-04-29T22:21:41</JAT_start_time>
        <slots>1</slots>
        <tasks>38</tasks>
      </job_list>
    </Queue-List>
    <Queue-List>
      <name>all.q@l10</name>
      <qtype>BIP</qtype>
      <slots_used>1</slots_used>
      <slots_total>2</slots_total>
      <arch>lx24-amd64</arch>
      <job_list state="running">
        <JB_job_number>652</JB_job_number>
        <JAT_prio>0.55500</JAT_prio>
        <JB_name>Scene149_V01</JB_name>
        <JB_owner>renderer</JB_owner>
        <state>r</state>
        <JAT_start_time>2008-04-29T22:21:56</JAT_start_time>
        <slots>1</slots>
        <tasks>45</tasks>
      </job_list>
    </Queue-List>
    ...
  <job_info>
    <job_list state="pending">
      <JB_job_number>652</JB_job_number>
      <JAT_prio>0.55500</JAT_prio>
      <JB_name>Scene149_V01</JB_name>
      <JB_owner>renderer</JB_owner>
      <state>qw</state>
      <JB_submission_time>2008-04-29T09:13:09</JB_submission_time>
      <slots>1</slots>
      <tasks>65-84:1</tasks>
    </job_list>
    <job_list state="pending">
      <JB_job_number>653</JB_job_number>
      <JAT_prio>0.55500</JAT_prio>
      <JB_name>Scene150_V01</JB_name>
      <JB_owner>renderer</JB_owner>
      <state>qw</state>
      <JB_submission_time>2008-04-29T09:13:11</JB_submission_time>
      <slots>1</slots>
    </job_list>
    ...
  </job_info>
</job_info>

However, there is no corresponding XML stylesheet. It is trivial to wrap the qstat -xml with a CGI program to include a customised stylesheet so that the output can be visulaised in a browser. A shell script with AWK will do the job.

#! /bin/sh

echo "Content-type: text/xml"
echo
SGE_ROOT=/gridware/sge /gridware/sge/bin/lx24-amd64/qstat -f -xml | \
awk '
{
        print $0
        if ( NR==1 ) {
                printf("<?xml-stylesheet type="text/xsl" href="qstat.xsl"?>\n")
        }
}'

Below is my XML stylesheet (qstat.xsl) and it's corresponding CSS (qstat.css)

<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/job_info">
<html>
<head>
<title>SGE qstat</title>
<link rel="stylesheet" href="qstat.css" type="text/css"/>
</head>


<body background="#000">

<table cellspacing="10" border="0">
<tr><td align="left" valign="top">

<!-- Running -->
<h1>Running</h1>
<table border="1" cellpadding="4" cellspacing="0">
<tr>
<th>Queue Name</th>
<th>Job ID</th>
<th>Job Name</th>
<th>Task ID</th>
<th>State</th>
<th>Start Time</th>
</tr>
<xsl:for-each select="queue_info/Queue-List">
<tr>
<td class="value"><xsl:value-of select="name/text()" /></td>
<td class="value"><xsl:value-of select="job_list/JB_job_number/text()" /></td>
<td class="value"><xsl:value-of select="job_list/JB_name/text()" /></td>
<td class="value"><xsl:value-of select="job_list/tasks/text()" /></td>
<td class="value"><span class="running"><xsl:value-of select="job_list/@state" /></span></td>
<td class="value"><xsl:value-of select="job_list/JAT_start_time/text()" /></td>
</tr>
</xsl:for-each>
</table>


</td>
<td align="left" valign="top">


<!-- Pending -->
<h1>Pending</h1>
<table border="1" cellpadding="4" cellspacing="0">
<tr>
<th>Job ID</th>
<th>Job Name</th>
<th>Task ID</th>
<th>State</th>
<th>Submission Time</th>
</tr>
<xsl:for-each select="job_info/job_list">
<tr>
<td class="value"><xsl:value-of select="JB_job_number/text()" /></td>
<td class="value"><xsl:value-of select="JB_name/text()" /></td>
<td class="value"><xsl:value-of select="tasks/text()" /></td>
<td class="value"><span class="pending"><xsl:value-of select="@state" /></span></td>
<td class="value"><xsl:value-of select="JB_submission_time/text()" /></td>
</tr>
</xsl:for-each>
</table>

</td>
</tr>
</table>


</body>
</html>
</xsl:template>
</xsl:stylesheet>

body { 
 background: #000;
 color: #fff;
 font-family: Arial;
 font-size: 12px;
}
h1 { 
 color: #ff7800;
 font-size: 32px;
 font-family: Times;
 font-weight: bold;
}
.value { 
 color: #aaa; 
 font-size: 12px;
 font-family: Arial;
}
.running { 
 color: #0b0; 
 font-size: 12px;
 font-family: Arial;
 font-weight: bold;
}
.pending { 
 color: #b00; 
 font-size: 12px;
 font-family: Arial;
 font-weight: bold;
}

The output on the browser will look like this:

SGE Accounting

Sun Grid Engine accounting (5) provides a very nice utility to summarise accounting information of all your grid jobs. In my recent blog regarding SGE for rendering, I recommeneded to include "-A" flag (for accounting string) and "-P" flag (for project name) in qsub. This will help you to extract the accounting information for a particular job. Now it is time to reap the benefits. Below shows how you can extract the total accounting information per project as well as per accounting string. In my case, the accounging string and job name are the same as the scene file name without the file extension.

$ qacct -P myproject
PROJECT     WALLCLOCK         UTIME         STIME           CPU             MEMORY                 IO                IOW
========================================================================================================================
myproject    33970465      47273362       1178602      48638827       56454795.153              0.000              0.000

$ qacct -A Scene1
Total System Usage
    WALLCLOCK         UTIME         STIME           CPU             MEMORY                 IO                IOW
================================================================================================================
       101114        169445          4951        174490         216975.065              0.000              0.000

In my recent rendering project, I have to deal with 1000+ scene files. To be extact, 1224 unique scene files. We should be able to find out the run time information of every single job by looping through them and run qacct on that. However, it is going to be very inefficient 'cos we have to read the accounting file 1224 times. Also the output format cannot be imported to any spreadsheet program for further analysis.

$ awk -F: '$32~/^myproject$/{print $7}' accounting | sort | uniq | wc -l
1224

$ for i in `awk -F: '$32~/^myproject$/{print $7}' accounting | sort | uniq`
do
 echo $i
 qacct -A $i
done

accounting (5) has documented clearly every field in the accounting file. We can write an awk program to loop through the file once and print out all these information as CVS (Comma Separated Variable).

#! /bin/sh


awk '
BEGIN {
        FS=":"
        OFS=","
}
$32 ~ /^myproject$/ {
        scene=$7
        wallclock[scene]+=$14
        utime[scene]+=$15
        stime[scene]+=$16
        cpu[scene]+=$37
        mem[scene]+=$38
        io[scene]+=$39
        iow[scene]+=$41

        # no of jobs per scene
        ++job[scene]
}
END {
        print "SCENE_NAME","NO_OF_JOBS","WALLCLOCK","UTIME","STIME","CPU","MEMORY","IO","IOW"
        for ( i in wallclock ) {
                print i,job[i],wallclock[i],utime[i],stime[i],cpu[i],mem[i],io[i],iow[i]
        }
}' accounting

However, we are still dealing with lot of data. How about visualising the jobs using Gnuplot. With the raw accounting data, you can find out the start and end time of each job. Tcl has excellent utility to convert epoch time to other date/time format. Although gnuplot can handle epoch time plotting, I realised that they always based on GMT+0 and that will mess up the x-axis label. Anyway, here is the Tcl program to extract the start time and summarise it based on per day per hour.

set fp [open accounting r]
while { [gets $fp line] >= 0 } {
        set lline [split $line :]

        set project [lindex $lline 31]

        if { $project != "myproject" } {
                continue
        }

        set jobname [lindex $lline 4]
        set starttime [lindex $lline 9]

        set ymdh [clock format $starttime -format {%Y-%m-%d %H:00:00}]

        if { [info exists stats($ymdh)] == 0 } {
                set stats($ymdh) 1
        } else { 
                incr stats($ymdh)
        }
}
close $fp


foreach i [lsort [array names stats]] {
        puts "$i $stats($i)"
}

And the corresponding output (stats.txt) is like this:

...
2008-04-25 22:00:00 137
2008-04-25 23:00:00 83
2008-04-26 00:00:00 83
2008-04-26 01:00:00 86
2008-04-26 02:00:00 84
...

Below gnuplot file will visualise the number of jobs started hourly for your entire project.

set terminal png
set output 'stats.png'
set xdata time
set size 1,0.5
set timefmt '%Y-%m-%d %H:%M:%S'
set xrange ['2008-04-23 00:00:00':'2008-05-14 23:59:00']
set yrange [0:]
set title 'Rendering - myproject'
set ylabel 'Jobs per hour'
set xtics 86400 offset 2
set format x "%d\n%b\n%a"
set grid
plot 'stats.txt' using 1:3 with impulses title '#Jobs Started'

Sun Grid Engine for Rendering

In the past, I used to submit animation scene files to SGE (Sun Grid Engine) to render as individual job per frame. It is very hard to manage because we have to deal with lots of SGE job IDs for a scene.

Do you know that in SGE you can treat that as an array job and let SGE to handle the scheduling of individual sub-task. Eg. if you have a scene file to render with frame 1 to 256, you can submit as qsub -t 1-256:1 render.sh scenefile.mb. SGE will only assign one SGE job ID. This give you the flexibility to modify the entire job with a single SGE job ID, eg. alter the job dependency, change the priority, hold the job in queue, remove from queue. With this, you do not have to keep track in terms of individual frame. In SGE man page, -t option is described as:

Submits a so called Array Job, i.e. an array of identical tasks
being differentiated only by an index number and being treated
by Sun Grid Engine almost like a series of jobs. The option
argument to -t specifies the number of array job tasks and the
index number which will be associated with the tasks. The index
numbers will be exported to the job tasks via the environment
variable SGE_TASK_ID.

In essence,

• Scene file == SGE Job ID / Name (set in JOB_ID / JOB_NAME environment variable)
• Frame in scene file == SGE Task ID (set in SGE_TASK_ID environment variable)

When a job is scheduled to run on a execution node, SGE_TASK_ID environment variable will be set based on the task id in the array job. In this case, the SGE_TASK_ID will be your frame number.

You may also want to couple the submission command (qsub) with "-P" (for project), "-A" (for accounting) and "-N" (for job name) to help you streamline other activities such as accounting, monitoring, ...

The images generated from the rendering can easily consists of hundred of files with a total file size of MB or GB. It would be too troublesome to download individual images and 'brandwidth unfriendly' if they are not compressed. I explored a number of ways to effectively accomplish this:

1. Write a "epilog" script in the SGE queue configuration (qconf -mq all.q) so that it will run at the end of each job. However, once the array job is run in the queue, we have no control which task will complete first. We cannot assume the last frame will be rendered last. The script has to be smart enough to know that your current task is indeed the last task in the queue. In my newly revamped cluster, I ensure the SGE job name is unique and therefore I can easily identify the scene file from qstat -xml

   njob=`SGE_ROOT=/gridware/sge /gridware/sge/bin/lx24-amd64/qstat -xml | grep -c "<JB_name>$SGE_JOB_NAME</JB_name>"`
   if [ $njob == 1 ]; then
    cd /san/renderImages
    tar cf $SGE_JOB_NAME.tar ./$SGE_JOB_NAME
    gzip $SGE_JOB_NAME.tar
   fi
   

2. Another method is to take advantage of SGE job dependency. First job to do the rendering and the second job to do the compression. The dependency is that second job will only start if first job finishes.
3. Run a cron job in the server where the storage is directly attached. The script need to ensure it will not compress if the job is still running. Since the script will list out all the directories in reverse chronological order, we can terminate the loop once the directory has been processed before.

   #! /bin/sh
   
   
   dir="/san/renderImages"
   
   cd $dir
   for i in `ls -1td scenefile*`
   do
           # still running
           SGE_ROOT=/gridware/sge /gridware/sge/bin/lx24-amd64/qstat -xml | grep "<JB_name>$i</JB_name>" > /dev/null
           if [ $? -eq 0 ]; then
                   continue
           fi
   
   
           # previously compressed; we can break from here
           tarfile="$i.tar"
           if [ -f "$tarfile.gz" ]; then
                   break
           fi
   
           # compress now
           tar cf $tarfile ./$i
           gzip $tarfile
   
   done
   

I opted for the last method because I can do the 'tar' and 'gzip' in the server with direct attached storage, no network traffic incurred. Also, it has no additional SGE job unless the second method. The first and second methods will have to run the compression in the execution node and it has to read and write over the network.

BTW, the output of qstat -xml looks like this:

<?xml version='1.0'?>
<job_info  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <queue_info>
    <job_list state="running">
      <JB_job_number>572</JB_job_number>
      <JAT_prio>0.55500</JAT_prio>
      <JB_name>scenefile_001</JB_name>
      <JB_owner>renderer</JB_owner>
      <state>r</state>
      <JAT_start_time>2008-04-29T10:48:11</JAT_start_time>
      <queue_name>all.q@l1</queue_name>
      <slots>1</slots>
      <tasks>9</tasks>
    </job_list>
    <job_list state="running">
      <JB_job_number>570</JB_job_number>
      <JAT_prio>0.55500</JAT_prio>
      <JB_name>scenefile_002</JB_name>
      <JB_owner>renderer</JB_owner>
      <state>r</state>
      <JAT_start_time>2008-04-29T10:47:16</JAT_start_time>
      <queue_name>all.q@l10</queue_name>
      <slots>1</slots>
      <tasks>68</tasks>
    </job_list>
 ...

Sun Grid Engine Accounting Users Summary

In Sun Grid Engine (SGE), all job accounting information is written to $SGE_ROOT/$SGE_CELL/common/accounting file. Each record of the accounting information consists of 43 fields separated by colon (':') signs.

For details of all the fields, read up the man page of accounting(5).

If you want to find out distribution of jobs and their corresponding elpased time per user, you will be interested in 4th and 14th field in the accounting record.

$ man accounting


N1 Grid Engine File Formats                         ACCOUNTING(5)

NAME
     accounting - N1 Grid Engine accounting file format

DESCRIPTION
     An accounting record  is  written  to  the  N1  Grid  Engine
     accounting file for each job having finished. The accounting
     file is processed by qacct(1) to derive  accounting  statis-
     tics.

FORMAT
     Each job is represented by a line in  the  accounting  file.
     Empty  lines  and  lines which contain one character or less
     are ignored.  Accounting record  entries  are  separated  by
     colon  (':')  signs.  The  entries  denote in their order of
     appearance:

     qname
          Name of the cluster queue in which the job has run.

     hostname
          Name of the execution host.

     group
          The effective group id of the job owner when  executing
          the job.

     owner
          Owner of the N1 Grid Engine job.

     job_name
          Job name.

     job_number
          Job identifier - job number.

     account
          An account  string  as  specified  by  the  qsub(1)  or
          qalter(1) -A option.

     priority
          Priority value assigned to the job corresponding to the
          priority  parameter  in  the  queue  configuration (see
          queue_conf(5)).

     submission_time
          Submission time in seconds (since epoch format).

     start_time
          Start time in seconds (since epoch format).

     end_time
          End time in seconds (since epoch format).

N1GE 6          Last change: 2004/04/19 10:52:07                1

N1 Grid Engine File Formats                         ACCOUNTING(5)

     failed
          Indicates the problem which  occurred  in  case  a  job
          could  not  be  started  on  the  execution  host (e.g.
          because the owner of the  job  did  not  have  a  valid
          account  on  that  machine). If N1 Grid Engine tries to
          start a job multiple times, this may lead  to  multiple
          entries  in  the  accounting  file corresponding to the
          same job ID.

     exit_status
          Exit status of  the  job  script  (or  N1  Grid  Engine
          specific status in case of certain error conditions).

     ru_wallclock
          Difference between end_time and start_time (see above).

Below is an AWK script to summarise the accounting information and it's corresponding output. FYI, the usernames are fictitious.

$ cat sge-summary.sh
#! /bin/sh

awk -F":" '
NF==43 {
        # $4 - owner
        # $14 - wallclock
        ++jsum[$4]
        ++jcnt
        tsum[$4]+=$14
        tcnt+=$14
}
END {
        printf("User\tJob\tRun Time\n")
        for(i in jsum) {
                printf("%-10s\t%.2f%\t%.2f%\n", i, jsum[i]*100/jcnt, tsum[i]*100/tcnt)
        }
}' $SGE_ROOT/$SGE_CELL/common/accounting


$ ./sge-summary.sh
User            Job     Run Time
alan            0.02%    0.00%
bob             2.43%    0.00%
carl            0.02%    0.00%
daryl           0.84%    0.00%
edwin           0.20%    0.00%
francis         0.01%    0.00%
george          0.06%    0.00%
harry           0.02%    0.00%
irene           0.02%    0.00%
jeffrey         0.71%    99.36%
karen           0.05%    0.00%
leo             0.04%    0.00%
mark            0.04%    0.00%
nelson          95.32%   0.64%
oliver          0.22%    0.00%

Sun Grid Engine (SGE) Solving Recursion

During one of the demonstrations or benchmarks, whatever you want to call it, we were asked to use Sun Grid Engine (SGE) to show how it can be used to solve a recursive calculation.

The formula is like this: I_n = I_n-1 + I_n-2 , if n=0, I₀ = 1 , if n=1, I₁ = 1

If n=4, anwser is 8; if n=5, answer is 13

Easy, right? Yes, if you are going to program using your favourite programming language. But not using a grid engine, truss me.

Programming in a shell script way, it will be like this, easy peezy!

#! /bin/sh

calculate()
{
      echo `expr $1 + $2`
}


if [ $# -ne 1 ]; then
      echo "Usage: $0 <no>"
      exit 1
fi
num=$1


if [ $num -eq 0 ]; then
      result=1
elif [ $num -eq 1 ]; then
      result=2
else
      d1=`expr $num - 1`
      d2=`expr $num - 2`
      result1=`$0 $d1`
      result2=`$0 $d2`
      result=`calculate $result1 $result2`
fi
echo $result

Let's visualise the tree in order to get a feel of what we are going to deal with. Let's start with n=4

        4
       / \
      3    2
     / \  / \
    2  1  1  0
   / \
  1   0

How can we use SGE to simulate this recursion ? How can the 2 children know who their parent is after the jobs are submitted to execution nodes ? How to keep track of their results ? How can these results get pass back to the parent ? How to eventually stop and wind back up to the root node ?

Yes, these are the questions in my mind when I started off with this exercise. Becasue SGE is not a sub-second scheduler and therefore I can "qsub" 2 jobs and immediate establish the parent-child relationship using the "-hold_jid" (hold job dependency). In order for the children to know what their parent is, I parsed the "-v PARENT=value" in the "qsub", so that they can get the PARENT environment variable.

How to keep track of result at every stage ? I made use of some temporary files, however I need to ensure uniqueness of these files. We can accomplish this by taking a MD5 of the first 32 bytes (can be any size of bytes) in /dev/urandom device in Solaris.

At first, my SGE environment is configured with 12 slots and I can only run n=3 and n=4. When n=5, it simply ran out of slots. Here, I realised that slots/queue is very similar to memory/stack in a computer. In a recursive program, it has to push things onto the stack and let the pop/push of stack to sort itself out. If your stack size is too small, your recursive program will break which is similar to our slots in SGE, except that all those jobs will be on hold. They are going to hold forever until you "gdel" them.

As for n > 5, I have to increase the number of slots to make it work. Below shows the two scripts, namely the one doing all the recursion (rec.sh) and the one doing the calculation (cal.sh). It also shows "qstat" in action every 5 seconds interval. Job name "-N" in "qsub" is needed in order for the final value to be stored in the file "five".

$ cat rec.sh
#! /bin/sh
#$ -cwd
#$ -S /bin/sh
#$ -o /dev/null
#$ -e /dev/null


jndir=tmp
[ ! -d $jndir ] && mkdir $jndir


# loop around qacct to wait for result
getResult()
{
      while :
      do
              qacct -j $1 > /dev/null 2>&1
              if [ $? -eq 0 ]; then
                      cat $jndir/$1
                      return
              fi
              sleep 1
      done

      # it shouldn't reach here
      exit 1
}



if [ $# -ne 1 ]; then
      echo "Usage: $0 "
      exit 1
fi

. /opt/n1ge/default/common/settings.sh


number=$1
jobname=$JOB_NAME


if [ $number -eq 0 ]; then
      result=1
elif [ $number -eq 1 ]; then
      result=2
else
      d1=`expr $number - 1`
      d2=`expr $number - 2`
      jn1="jn-`dd if=/dev/urandom bs=32 count=1 2>/dev/null | digest -a md5`"
      jn2="jn-`dd if=/dev/urandom bs=32 count=1 2>/dev/null | digest -a md5`"

      qsub -N $jn1 -v PARENT=$jobname $0 $d1 > /dev/null 2>&1
      qsub -N $jn2 -v PARENT=$jobname $0 $d2 > /dev/null 2>&1

      # if there is parent-child relationship, establish it
      if [ "X$PARENT" != "X" ]; then
              qalter -hold_jid $jn1,$jn2 $jobname > /dev/null 2>&1
      fi

      result1=`getResult $jn1`
      result2=`getResult $jn2`

      result=`./cal.sh $result1 $result2`
fi


# write result to file
echo $result > $jndir/$jobname

$ cat cal.sh
#! /bin/sh

echo `expr $1 + $2`

$ qsub -N five rec.sh 5
Your job 463 ("five") has been submitted.

$ while :
do
qstat
sleep 5
done
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  464 0.00000 jn-1c84996 chihung      qw    07/27/2007 23:15:31                                    1
  465 0.00000 jn-0ed40af chihung      qw    07/27/2007 23:15:31                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  464 0.00000 jn-1c84996 chihung      qw    07/27/2007 23:15:31                                    1
  465 0.00000 jn-0ed40af chihung      qw    07/27/2007 23:15:31                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  464 0.00000 jn-1c84996 chihung      qw    07/27/2007 23:15:31                                    1
  465 0.00000 jn-0ed40af chihung      qw    07/27/2007 23:15:31                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      hr    07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  466 0.00000 jn-66ad4b8 chihung      qw    07/27/2007 23:15:46                                    1
  467 0.00000 jn-1ddba59 chihung      qw    07/27/2007 23:15:46                                    1
  468 0.00000 jn-156ac6d chihung      qw    07/27/2007 23:15:46                                    1
  469 0.00000 jn-260acb5 chihung      qw    07/27/2007 23:15:46                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      hr    07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  466 0.00000 jn-66ad4b8 chihung      qw    07/27/2007 23:15:46                                    1
  467 0.00000 jn-1ddba59 chihung      qw    07/27/2007 23:15:46                                    1
  468 0.00000 jn-156ac6d chihung      qw    07/27/2007 23:15:46                                    1
  469 0.00000 jn-260acb5 chihung      qw    07/27/2007 23:15:46                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      hr    07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  466 0.00000 jn-66ad4b8 chihung      qw    07/27/2007 23:15:46                                    1
  467 0.00000 jn-1ddba59 chihung      qw    07/27/2007 23:15:46                                    1
  468 0.00000 jn-156ac6d chihung      qw    07/27/2007 23:15:46                                    1
  469 0.00000 jn-260acb5 chihung      qw    07/27/2007 23:15:46                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      hr    07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  466 0.55500 jn-66ad4b8 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec2                     1
  469 0.55500 jn-260acb5 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec2                     1
  470 0.00000 jn-85a72cc chihung      qw    07/27/2007 23:16:01                                    1
  471 0.00000 jn-0ecd1c5 chihung      qw    07/27/2007 23:16:01                                    1
  472 0.00000 jn-f7b0c72 chihung      qw    07/27/2007 23:16:01                                    1
  473 0.00000 jn-c69525b chihung      qw    07/27/2007 23:16:01                                    1
  474 0.00000 jn-f7175e4 chihung      qw    07/27/2007 23:16:02                                    1
  475 0.00000 jn-f31a24c chihung      qw    07/27/2007 23:16:02                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      hr    07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  466 0.55500 jn-66ad4b8 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec2                     1
  469 0.55500 jn-260acb5 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec2                     1
  470 0.00000 jn-85a72cc chihung      qw    07/27/2007 23:16:01                                    1
  471 0.00000 jn-0ecd1c5 chihung      qw    07/27/2007 23:16:01                                    1
  472 0.00000 jn-f7b0c72 chihung      qw    07/27/2007 23:16:01                                    1
  473 0.00000 jn-c69525b chihung      qw    07/27/2007 23:16:01                                    1
  474 0.00000 jn-f7175e4 chihung      qw    07/27/2007 23:16:02                                    1
  475 0.00000 jn-f31a24c chihung      qw    07/27/2007 23:16:02                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      hr    07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  466 0.55500 jn-66ad4b8 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec2                     1
  469 0.55500 jn-260acb5 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec2                     1
  470 0.00000 jn-85a72cc chihung      qw    07/27/2007 23:16:01                                    1
  471 0.00000 jn-0ecd1c5 chihung      qw    07/27/2007 23:16:01                                    1
  472 0.00000 jn-f7b0c72 chihung      qw    07/27/2007 23:16:01                                    1
  473 0.00000 jn-c69525b chihung      qw    07/27/2007 23:16:01                                    1
  474 0.00000 jn-f7175e4 chihung      qw    07/27/2007 23:16:02                                    1
  475 0.00000 jn-f31a24c chihung      qw    07/27/2007 23:16:02                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      hr    07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec1                     1
  471 0.55500 jn-0ecd1c5 chihung      hr    07/27/2007 23:16:16 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  466 0.55500 jn-66ad4b8 chihung      r     07/27/2007 23:16:01 all.q@sgeexec2                     1
  476 0.00000 jn-9ae8029 chihung      qw    07/27/2007 23:16:16                                    1
  477 0.00000 jn-d126937 chihung      qw    07/27/2007 23:16:16                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      r     07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec1                     1
  471 0.55500 jn-0ecd1c5 chihung      hr    07/27/2007 23:16:16 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  476 0.00000 jn-9ae8029 chihung      qw    07/27/2007 23:16:16                                    1
  477 0.00000 jn-d126937 chihung      qw    07/27/2007 23:16:16                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  465 0.55500 jn-0ed40af chihung      r     07/27/2007 23:15:46 all.q@sgeexec0                     1
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      hr    07/27/2007 23:16:01 all.q@sgeexec1                     1
  471 0.55500 jn-0ecd1c5 chihung      hr    07/27/2007 23:16:16 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
  476 0.00000 jn-9ae8029 chihung      qw    07/27/2007 23:16:16                                    1
  477 0.00000 jn-d126937 chihung      qw    07/27/2007 23:16:16                                    1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      r     07/27/2007 23:16:01 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      r     07/27/2007 23:16:01 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  464 0.55500 jn-1c84996 chihung      hr    07/27/2007 23:15:46 all.q@sgeexec1                     1
  467 0.55500 jn-1ddba59 chihung      r     07/27/2007 23:16:01 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  464 0.55500 jn-1c84996 chihung      r     07/27/2007 23:15:46 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  464 0.55500 jn-1c84996 chihung      r     07/27/2007 23:15:46 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  464 0.55500 jn-1c84996 chihung      r     07/27/2007 23:15:46 all.q@sgeexec1                     1
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
  463 0.55500 five       chihung      r     07/27/2007 23:15:31 all.q@sgeexec2                     1
^C

$ ls tmp
five                                 jn-85a72cc20898241095d1489ebbb02ca7
jn-0ecd1c5c6721976cb9ca96c902fb1044  jn-9ae80293acaf8fc426bf7972ad8f6c38
jn-0ed40afa95dde7d92d3f5784fc397a2c  jn-c69525b4ba477a7ce110aab8a2438c45
jn-156ac6d58000f8c8c18b973959622464  jn-d126937687b91b5f6b14e1b642aedc60
jn-1c849969b8c86e907bd32255058313e6  jn-f31a24c2219734f3bef13f6ce5addf78
jn-1ddba5925ff899f36812527220fe19a9  jn-f7175e4d15c7e08a98a7c13915d0121e
jn-260acb532729915691a6b096eeb69e9b  jn-f7b0c72538b479c4615ad5abd0214660
jn-66ad4b80af12a4b4c7469e76d50406a7

$ cat tmp/five
13

Interesting ? For me, definitely.

SGE Grid Job Dependency

It is possible to describe SGE (Sun Grid Engine) job (or any other grid engine) dependency in a DAG (Directed Acyclic Graph) format. By taking advantage of the opensource Graphviz, it is very easy to document this dependency in DOT language format. Below shows you a sample DOT file:

$ cat job-dep.dot
digraph jobs101 {
        job_1 -> job_11;
        job_1 -> job_12;
        job_1 -> job_13;
        job_11 -> job_111;
        job_12 -> job_111;
        job_2 -> job_13;
        job_2 -> job_21;
        job_3 -> job_21;
        job_3 -> job_31;
}

With this DOT file, one can generate the graphical representation:

$ dot -Tpng -o job-dep.png job-dep.dot

It is also possible to derive the corresponding SGE commands by the following Tcl script.

$ cat ./dot2sge.tcl
#! /usr/local/bin/tclsh


if { $argc != 1 } {
        puts stderr "Usage: $argv0 "
        exit 1
}
set dotfile [lindex $argv 0]
if { [file exists $dotfile] == 0 } {
        puts stderr "Error. $dotfile does not exist"
        exit 2
}


# assume simple directed graph a -> b

set fp [open $dotfile r]
set data [read $fp]
close $fp


set sge_jobs {}
foreach i [split [lindex $data 2] {;}] {
        if { [regexp {(\w+)\s*->\s*(\w+)} $i x parent child] != 0 } {
                lappend sge_jobs $parent
                lappend sge_jobs $child

                lappend sge_job_rel($parent) $child
        }
}


# submit unique jobs, and hold
set queue all.q
set sge_unique_jobs [lsort -unique $sge_jobs]
foreach i $sge_unique_jobs {
        puts "qsub -h -q $queue -N $i job-submit.sh"
}


# alter the job dependency, but unhold after all the hold relationships are
# established
foreach i $sge_unique_jobs {
        if { [info exists sge_job_rel($i)] } {
                # with dependency
                puts "qalter -hold_jid [join $sge_job_rel($i) {,}] $i"
        }
}
foreach i $sge_unique_jobs {
        puts "qalter -h U $i"
}

Run this Tcl script to generate the SGE submission commands and alternation commands to register the job dependency

$ ./dot2sge.tcl job-dep.dot
qsub -h -q all.q -N job_1 job-submit.sh
qsub -h -q all.q -N job_11 job-submit.sh
qsub -h -q all.q -N job_111 job-submit.sh
qsub -h -q all.q -N job_12 job-submit.sh
qsub -h -q all.q -N job_13 job-submit.sh
qsub -h -q all.q -N job_2 job-submit.sh
qsub -h -q all.q -N job_21 job-submit.sh
qsub -h -q all.q -N job_3 job-submit.sh
qsub -h -q all.q -N job_31 job-submit.sh
qalter -hold_jid job_11,job_12,job_13 job_1
qalter -hold_jid job_111 job_11
qalter -hold_jid job_111 job_12
qalter -hold_jid job_13,job_21 job_2
qalter -hold_jid job_21,job_31 job_3
qalter -h U job_1
qalter -h U job_11
qalter -h U job_111
qalter -h U job_12
qalter -h U job_13
qalter -h U job_2
qalter -h U job_21
qalter -h U job_3
qalter -h U job_31

Below show the above proof-of-concept in action. So sit back....

#
# ----------below is a very simple script
#
$ cat job-submit.sh
#! /bin/sh
#$ -S /bin/sh

date
sleep 10


#
# ----------run all the qsub to submit jobs, but put them on hold
#
$ qsub -h -q all.q -N job_1 job-submit.sh
Your job 333 ("job_1") has been submitted.
$ qsub -h -q all.q -N job_11 job-submit.sh
Your job 334 ("job_11") has been submitted.
$ qsub -h -q all.q -N job_111 job-submit.sh
Your job 335 ("job_111") has been submitted.
$ qsub -h -q all.q -N job_12 job-submit.sh
Your job 336 ("job_12") has been submitted.
$ qsub -h -q all.q -N job_13 job-submit.sh
Your job 337 ("job_13") has been submitted.
$ qsub -h -q all.q -N job_2 job-submit.sh
Your job 338 ("job_2") has been submitted.
$ qsub -h -q all.q -N job_21 job-submit.sh
Your job 339 ("job_21") has been submitted.
$ qsub -h -q all.q -N job_3 job-submit.sh
Your job 340 ("job_3") has been submitted.
$ qsub -h -q all.q -N job_31 job-submit.sh
Your job 341 ("job_31") has been submitted.


#
# ----------show the status, all jobs are in hold position (hqw)
#
$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   0/4       0.01     sol-amd64

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
    333 0.00000 job_1      chihung      hqw   07/19/2007 21:04:34     1
    334 0.00000 job_11     chihung      hqw   07/19/2007 21:04:34     1
    335 0.00000 job_111    chihung      hqw   07/19/2007 21:04:34     1
    336 0.00000 job_12     chihung      hqw   07/19/2007 21:04:34     1
    337 0.00000 job_13     chihung      hqw   07/19/2007 21:04:34     1
    338 0.00000 job_2      chihung      hqw   07/19/2007 21:04:34     1
    339 0.00000 job_21     chihung      hqw   07/19/2007 21:04:34     1
    340 0.00000 job_3      chihung      hqw   07/19/2007 21:04:34     1
    341 0.00000 job_31     chihung      hqw   07/19/2007 21:04:34     1


#
# ----------register the job dependency
#
$ qalter -hold_jid job_11,job_12,job_13 job_1
modified job id hold list of job 333
   blocking jobs: 334,336,337
   exited jobs:   NONE
$ qalter -hold_jid job_111 job_11
modified job id hold list of job 334
   blocking jobs: 335
   exited jobs:   NONE
$ qalter -hold_jid job_111 job_12
modified job id hold list of job 336
   blocking jobs: 335
   exited jobs:   NONE
$ qalter -hold_jid job_13,job_21 job_2
modified job id hold list of job 338
   blocking jobs: 337,339
   exited jobs:   NONE
$ qalter -hold_jid job_21,job_31 job_3
modified job id hold list of job 340
   blocking jobs: 339,341
   exited jobs:   NONE


#
# ----------release all the holds and let SGE to sort itself out
#
$ qalter -h U job_1
modified hold of job 333
$ qalter -h U job_11
modified hold of job 334
$ qalter -h U job_111
modified hold of job 335
$ qalter -h U job_12
modified hold of job 336
$ qalter -h U job_13
modified hold of job 337
$ qalter -h U job_2
modified hold of job 338
$ qalter -h U job_21
modified hold of job 339
$ qalter -h U job_3
modified hold of job 340
$ qalter -h U job_31
modified hold of job 341


#
# ----------query SGE stats
#
$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   0/4       0.01     sol-amd64

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
    333 0.00000 job_1      chihung      hqw   07/19/2007 21:04:34     1
    334 0.00000 job_11     chihung      hqw   07/19/2007 21:04:34     1
    335 0.00000 job_111    chihung      qw    07/19/2007 21:04:34     1
    336 0.00000 job_12     chihung      hqw   07/19/2007 21:04:34     1
    337 0.00000 job_13     chihung      qw    07/19/2007 21:04:34     1
    338 0.00000 job_2      chihung      hqw   07/19/2007 21:04:34     1
    339 0.00000 job_21     chihung      qw    07/19/2007 21:04:34     1
    340 0.00000 job_3      chihung      hqw   07/19/2007 21:04:34     1
    341 0.00000 job_31     chihung      qw    07/19/2007 21:04:34     1


#
# ----------some jobs started to run
#
$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   2/4       0.01     sol-amd64
    339 0.55500 job_21     chihung      r     07/19/2007 21:05:36     1
    341 0.55500 job_31     chihung      r     07/19/2007 21:05:36     1
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   1/4       0.01     sol-amd64
    335 0.55500 job_111    chihung      r     07/19/2007 21:05:36     1
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   1/4       0.01     sol-amd64
    337 0.55500 job_13     chihung      r     07/19/2007 21:05:36     1

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
    333 0.00000 job_1      chihung      hqw   07/19/2007 21:04:34     1
    334 0.00000 job_11     chihung      hqw   07/19/2007 21:04:34     1
    336 0.00000 job_12     chihung      hqw   07/19/2007 21:04:34     1
    338 0.00000 job_2      chihung      hqw   07/19/2007 21:04:34     1
    340 0.00000 job_3      chihung      hqw   07/19/2007 21:04:34     1


$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   2/4       0.01     sol-amd64
    339 0.55500 job_21     chihung      r     07/19/2007 21:05:36     1
    341 0.55500 job_31     chihung      r     07/19/2007 21:05:36     1
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   1/4       0.01     sol-amd64
    335 0.55500 job_111    chihung      r     07/19/2007 21:05:36     1
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   1/4       0.01     sol-amd64
    337 0.55500 job_13     chihung      r     07/19/2007 21:05:36     1

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
    333 0.00000 job_1      chihung      hqw   07/19/2007 21:04:34     1
    334 0.00000 job_11     chihung      hqw   07/19/2007 21:04:34     1
    336 0.00000 job_12     chihung      hqw   07/19/2007 21:04:34     1
    338 0.00000 job_2      chihung      hqw   07/19/2007 21:04:34     1
    340 0.00000 job_3      chihung      hqw   07/19/2007 21:04:34     1


$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   0/4       0.01     sol-amd64

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
    333 0.00000 job_1      chihung      hqw   07/19/2007 21:04:34     1
    334 0.00000 job_11     chihung      qw    07/19/2007 21:04:34     1
    336 0.00000 job_12     chihung      qw    07/19/2007 21:04:34     1
    338 0.00000 job_2      chihung      qw    07/19/2007 21:04:34     1
    340 0.00000 job_3      chihung      qw    07/19/2007 21:04:34     1


$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   2/4       0.01     sol-amd64
    338 0.55500 job_2      chihung      r     07/19/2007 21:05:51     1
    340 0.55500 job_3      chihung      r     07/19/2007 21:05:51     1
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   1/4       0.01     sol-amd64
    334 0.55500 job_11     chihung      r     07/19/2007 21:05:51     1
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   1/4       0.01     sol-amd64
    336 0.55500 job_12     chihung      r     07/19/2007 21:05:51     1

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
    333 0.00000 job_1      chihung      hqw   07/19/2007 21:04:34     1


$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   2/4       0.01     sol-amd64
    338 0.55500 job_2      chihung      r     07/19/2007 21:05:51     1
    340 0.55500 job_3      chihung      r     07/19/2007 21:05:51     1
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   1/4       0.01     sol-amd64
    334 0.55500 job_11     chihung      r     07/19/2007 21:05:51     1
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   1/4       0.01     sol-amd64
    336 0.55500 job_12     chihung      r     07/19/2007 21:05:51     1

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
    333 0.00000 job_1      chihung      hqw   07/19/2007 21:04:34     1


$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   0/4       0.01     sol-amd64

############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
    333 0.00000 job_1      chihung      qw    07/19/2007 21:04:34     1


$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   1/4       0.01     sol-amd64
    333 0.55500 job_1      chihung      r     07/19/2007 21:06:06     1


$ qstat -f
queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@sgeexec0                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec1                 BIP   0/4       0.01     sol-amd64
----------------------------------------------------------------------------
all.q@sgeexec2                 BIP   1/4       0.01     sol-amd64
    333 0.55500 job_1      chihung      r     07/19/2007 21:06:06     1


#
# ----------output of all jobs, you can see job job_1/2/3 finished last
#
$ grep 2007 job_*.o*
job_111.o335:Thu Jul 19 21:05:36 SGT 2007
job_11.o334:Thu Jul 19 21:05:51 SGT 2007
job_12.o336:Thu Jul 19 21:05:51 SGT 2007
job_13.o337:Thu Jul 19 21:05:36 SGT 2007
job_1.o333:Thu Jul 19 21:06:06 SGT 2007
job_21.o339:Thu Jul 19 21:05:36 SGT 2007
job_2.o338:Thu Jul 19 21:05:51 SGT 2007
job_31.o341:Thu Jul 19 21:05:37 SGT 2007
job_3.o340:Thu Jul 19 21:05:52 SGT 2007

Another successful proof-of-concept. :-)