Avoid Using Temporary Files

If your style of writing shell script is usually based on outputting stuff to temporary files for future processing, I can tell you that in most cases it is possible to do the same job without writing anything to the OS simply by using some UNIX shell tricks. By doing this, your script will be more efficient and more portable.

Suppose you have two functions (or commands) that produce SAME number of lines of output and you want to 'paste' the two output together. The easiest way out is to store the output in separate files. In this blog, I will introduce two functions, namely calc1 (calculate n*n) and calc2 (calculate n*n*n).

$ cat t0.sh
#! /bin/sh

PATH=/usr/bin:/bin

seq()
{
        nawk -v start=$1 -v end=$2 '
                END {for(i=start;i<=end;++i){print i}}' /dev/null
}
calc1()
{
        for i in `seq $1 $2`
        do
                echo `expr $i \* $i`
        done
}
calc2()
{
        for i in `seq $1 $2`
        do
                echo `expr $i \* $i \* $i`
        done
}

calc1 1 10 > sometempfile1
calc2 1 10 > sometempfile2
paste sometempfile1 sometempfile2
rm -f sometempfile1 sometempfile2

$ ./t0.sh
1       1
4       8
9       27
16      64
25      125
36      216
49      343
64      512
81      729
100     1000

In this scenario, the output from calc1 and calc2 are having the same number of records. We can simply take advantage of this by combining the output using UNIX sub-shell and have the output to be handled by AWK. In the AWK, I will store the output in an associative array (line) based on the record number (NR) and the array will be processed at the END block.

$ cat t1.sh
#! /bin/sh

PATH=/usr/bin:/bin

seq()
{
        nawk -v start=$1 -v end=$2 '
                END {for(i=start;i<=end;++i){print i}}' /dev/null
}
calc1()
{
        for i in `seq $1 $2`
        do
                echo `expr $i \* $i`
        done
}
calc2()
{
        for i in `seq $1 $2`
        do
                echo `expr $i \* $i \* $i`
        done
}

# using sub shell to group the output
( calc1 1 10 ; calc2 1 10) | \
nawk '
{ line[NR]=$0 }
END {
        for(i=1;i<=NR/2;++i) {
                print line[i] "\t" line[NR/2+i]
        }
}'

$ ./t1.sh
1       1
4       8
9       27
16      64
25      125
36      216
49      343
64      512
81      729
100     1000

As I mentioned earlier on, one can accomplish the same task without temporary files.

Wait, the task has not finished yet. What if the output records are not the same ?

( calc1 1 10 ; calc2 1 13 ) | ...

Obviously the second script will break. Can you fix it for me ? Do give it a try and I will provide my solution in a couple of days time.

Many Similar Scripts to One

Last month, I blogged about consolidating shell scripts by generalising scripts into a single script. Now I am proposing an alternative way in combining many scripts to one.

Suppose you have a dozen scripts that are scheduled to generate system information and redirect output to specific files across a number of servers. You may want to explore the use of function in shell. So each function will encapsulate individual script. If your scripts contain some common environment variables to define the PATH, output directory, and other stuffs, these common variables can now be centralised in this single script. Therefore you only have to edit this script to reflect the changes. Say you have three scripts, namely f1.sh, f2.sh and f3.sh, and they are doing similar things. You can put them like this:

$ cat all-func.sh
#! /bin/sh
#
# Consolidating all the similar functions into a single script
#


usage()
{
        echo "Usage: $0 [function|-a]"
        echo "       -a Run all the functions"
}
listfunc()
{
        for i in `egrep '^_' $0 | sed -e 's/..$//;s/^_//'`
        do
                echo $i
        done
}


globaldir="/some/global/dir"
PATH=/usr/bin:/bin:$globaldir/bin


#
# all user functions are prepend with "_" (underscore)
#
_f1()
{
        echo Running function f1 from $globaldir directory
}
_f2()
{
        echo Running function f2 from $globaldir directory
}
_f3()
{
        echo Running function f3 from $globaldir directory
}


allfunc=`listfunc`
if [ $# -eq 0 ]; then
        usage
        echo "List of functions available in this script:"
        for f in $allfunc
        do
                echo "    $f"
        done
elif [ $# -eq 1 ]; then
        case "$1" in
        -a)
                # run all the functions in this script
                for f in $allfunc
                do
                        _$f
                done
                ;;
        *)
                for f in $allfunc
                do
                        if [ "$1" = "$f" ]; then
                                _$f
                                exit 0
                        fi
                done
                echo "Error. Function \"$1\" does not exist"
                exit 1
                ;;
        esac
fi

By having all my user-defined functions prepended with "_" underscore, I can tell what are the user-defined functions wrapped in this script. Below shows the above script in action:

$ ./all-func.sh
Usage: ./all-func.sh [function|-a]
       -a Run all the functions
List of functions available in this script:
    f1
    f2
    f3

$ ./all-func.sh -a
Running function f1 from /some/global/dir directory
Running function f2 from /some/global/dir directory
Running function f3 from /some/global/dir directory

$ ./all-func.sh f2
Running function f2 from /some/global/dir directory

$ ./all-func.sh f4
Error. Function "f4" does not exist

Caller and Callee

Suppose you have developed a central alerting script (callee) that all your other scripts (caller) will execute if certain threshold is reached. In your alert script, you definitely want to inform the user where this alert is initiated. Now the question is how can I find out who called me.

In Bash shell, you can rely on the PPID environment variable to determine the parent process ID (PPID) in order to find out who actually called you. However, this PPID variable will not be available in Bourne shell. In order to write cross-platform script, it is better to rely on Bourne shell because it is definitely available in almost all UNIXes. FYI, the default shell for root account in Solaris is still Bourne shell (/bin/sh) and the shell for single-user mode is the statically-link Bourne shell (/sbin/sh). Over the years, I prefer to write in Bourne shell ("the lowest common denominator for shell") unless there is a performance issue.

In Bash, you can simply use the PPID

$ cat caller.bash
#! /bin/sh

./callee.sh

$ cat callee.bash
#! /bin/sh

pgrep $PPID

$ ./caller.bash
./caller.bash

In Bourne shell, I came up with this function that uses AWK to store the 'ppid' and 'command' in an associative array using the pid as the index. In Solaris or even other UNIX, you can ask 'ps' to output (-o) only those columns that you are interested in and in my case they are pid (process id), ppid (parent process id) and args (command with arguments). You must be wondering why we output only those information that is needed for the downstream commands ? It is for performance and easy parsing reason. Here is a typical scenario.

$ cat caller.sh
#! /bin/sh

./callee.sh

$ cat callee.sh
#! /bin/sh

caller()
{
        ps -ef -o 'pid,ppid,args' | \
        awk '
        NR>1 {
                ppid[$1]=$2

                # if command is /bin/*sh, use the name of the script
                if ( $3 ~ /^\/bin\/.*sh$/ ) {
                        command[$1]=$4
                } else {
                        command[$1]=$3
                }
        }
        END {
                print command[ppid['$$']]
        }'
}

caller

$ ./caller.sh
./caller.sh

String and Number Comparison In Shell Scripting

In shell, you will need to compare either string or number. There is a subtle different between the two and here I will try to explain that in details.

For string comparison, you will use these operators

• = (string equal)
• != (sting not equal)

For number comparison, you will use these operators

• -eq (number eqaul)
• -ne (number not equal)
• -gt (number greater than)
• -ge (number greater than or equal)
• -lt (number less than)
• -le (number less than or equal)

So what's the different between the two equals. Let's find out from the these two cases

1. 6 = 6 vs 6 -eq 6
2. 06 = 6 vs 06 -eq 6

Case 1: they are both equated to true because string '6' and string '6' are the same, and number 6 and number 6 are equal
Case 2: string '06' and '6' are different string, so it will equate to false. However, for number comparison 06 is 6 and therefore number 6 equals to 6

$ if [ 6 = 6 ]; then echo equal; else echo not equal; fi
equal

$ if [ 6 -eq 6 ]; then echo equal; else echo not equal; fi
equal

$ if [ 06 = 6 ]; then echo equal; else echo not equal; fi
not equal

$ if [ 06 -eq 6 ]; then echo equal; else echo not equal; fi
equal

In other scripting languages, like Tcl, anything number with a '0' in front is considered octal (base 8). So '09' is invalid in Tcl. Whereas in shell, they simply treat every number as decimal.

$ if [ 09 -eq 9 ]; then
      echo equal
  fi
equal

$ tclsh
% if { 06 == 6 } { puts equal }
equal
% if { 09 == 9 } { puts equal }
expected integer but got "09" (looks like invalid octal number)
%

Defensive Scripting, The Double Quote Way

If you want to ensure your shell script is fool-proof and that it will not break in all kind of situation, you should script it in a 'defensive' way. What I mean defensive is do not assume things. Often time people assume that their variable is set with non-empty value and they will try to use the variable value to compare with some pre-defined value. If the variable value is an empty string, your test condition will fall apart if it is not double quoted. Here I am going to simulate this type of situation where the 'ostype' variable will be empty because the command to run does not exist.

$ cat c.sh
#! /bin/sh


ostype=`unknown_cmd_get_os`

if [ $ostype = unix ]; then
        echo unix
else
        echo windows
fi

echo here

$ ./c.sh
./c.sh: line 4: unknown_cmd_get_os: command not found
./c.sh: line 6: [: =: unary operator expected
windows
here

In the above execution, the 'if' command thrown exception because shell carried out command and variable substitutions before the shell parsed the command line. Here are the steps the shell will go through

1. if [ $ostype = unix ]; then ... is typed in before you hit the return button
2. Upon hitting the return button, the shell will do substitution and in our case it will do variable substitution by replacing the $ostype with it's value. So the command going to the shell will be
   if [ = unix ]; then ...
3. When shell read from the standard input, it knows that you are trying to run the "if" command
4. The if" command is a built-in shell syntax and the correct syntax should be 'if' 'condition' 'then' .... 'fi'
5. Syntax for 'condition' should be '[' 'expr' ']' , and 'expr' is a conditional expression
6. What you want is to compare the value of variable "ostype" with the string "unix". However, after the variable substitution there isn't anything there and your 'expr' becomes '= unix' which is an incorrect conditional expression.
7. Exception occurred.

To be 'defensive', you can do either the following:

• Double quote everything. As you can see, it does not break the if statement because you are comparing an empty string with the string "unix"

  $ cat c.sh
  #! /bin/sh
  
  
  ostype=`unknown_cmd_get_os`
  
  if [ "$ostype" = "unix" ]; then
          echo unix
  else
          echo windows
  fi
  
  echo here
  
  $ ./c.sh
  ./c.sh: line 4: unknown_cmd_get_os: command not found
  windows
  here
  

• Prepend some arbitrary character(s), in our case a character 'x', to ensure the string is non-empty. With this technique you do not necessarily need to double quote the string

  $ cat c.sh
  #! /bin/sh
  
  
  ostype=`unknown_cmd_get_os`
  unix="unix"
  
  if [ x$ostype = x$unix ]; then
          echo unix
  else
          echo windows
  fi
  
  echo here
  
  $ ./c.sh
  ./c.sh: line 4: unknown_cmd_get_os: command not found
  windows
  here
  

IMHO, the best practice will be to double quote all your string. So next time when you script your string comparison, remember to double quote them.

Overloading My Age

Today I came across this blog with this quote:

"There are 10 types of people in the world — those who understand binary, and those who don’t."

To "overloading" my age, I will start to tell my friend that I will be 30 (in Hex) in two years time. :-)

Manipulate lots of .tar.gz files

If you need to manipulate a lot of .tar.gz files and you do not want to uncompress & untar them out, you may want to use Python to script it. With Python 'battery included' modules and libraries, you almost do not have to re-invent the wheel.

The tarfile module allows you to work with 'tar' file with ease. It even let you work with 'tar' file in either gzip or bzip2 compression mode.

Recently I need to extract information from a lot of Sun's SUNWexplo explorer files in .tar.gz format. Suppose I have a few explorer files and I need to print out the disk error line (every 5th line in "iostat -E" output from the file "explorer.hostid.machine-yyyy.mm.dd.hh.min/disks/iostat_-E.out"

$ ls *.tar.gz
explorer.12345678.machine1-2008.10.11.13.03.tar.gz
explorer.90abcdef.machine2-2008.10.04.19.04.tar.gz
explorer.98765432.machine3-2008.11.15.09.05.tar.gz

$ for i in *.tar.gz; do gunzip < $i | tar tvf - | grep iostat_-E.out; done<
-rw-r--r-- root/root      6428 2008-10-11 13:03 explorer.12345678.machine1-2008.10.11.13.03.tar.gz/disks/iostat_-E.out
-rw-r--r-- root/root      8334 2008-10-14 19:04 explorer.90abcdef.machine2-2008.10.04.19.04.tar.gz/disks/iostat_-E.out
-rw-r--r-- root/root     12864 2008-11-15 09:05 explorer.98765432.machine3-2008.11.15.09.05.tar.gz/disks/iostat_-E.out

Here is the code snippet that you may want to try out yourself.

import tarfile
import glob
import os

for explo in glob.glob("*.tar.gz"):
    targz=tarfile.open(explo, 'r:gz')
    iostat=targz.extractfile('%s/disks/iostat_-E.out' % os.path.basename(explo)[0:-7])
    count=0
    for line in iostat:
        if count%5 == 0:
            line=line.rstrip()
            print line
        count+=1
    
    iostat.close()
    targz.close()

You may want expand the above code so that you can alert the sys admin folk if the number of soft/hard/transport errors exceed certain threshold.

Netmask in decimal form

In Solaris, the ifconfig <interface> gives you hex format for netmask. What if you want to conver that "ffff0000" to "255.255.0.0", you will need some utility to do this. Below shell function is able to solve your problem:

$ netmask_h2d()
{
set -- `echo $1 | sed -e 's/\([0-9a-fA-F][0-9a-fA-F]\)/\1\ /g'`
perl -e '$,=".";print 0x'$1',0x'$2',0x'$3',0x'$4
}

$ netmask_h2d ffff0000
255.255.0.0

Although Perl has been in Solaris since version 8, we may want to use more primitive unix utility to do the job. "bc" (arbitrary precision arithmetic language) can work with any input base and output base. The default is base 10. By setting ibase=16, any input will be treated as base 16. Note that "bc" only take upper case [A-F]. Here is a function to convert Hex to Dec, also it auto convert to uppercase.

$ h2d()
{
(echo ibase=16; echo $1 | tr '[a-z]' '[A-Z]') | bc
}

So we can have a more "portable" version of netmask_h2d without having to reply on Perl.

$ netmask_h2d()
{
set -- `echo $1 | sed -e 's/\([0-9a-fA-F][0-9a-fA-F]\)/\1\ /g'`
echo "`h2d $1`.`h2d $2`.`h2d $3`.`h2d $4`"
}

$ netmask_h2d ffff0000
255.255.0.0

Shell Script Documentation

In Python, you can have built-in documentation within your Python script using three double-quotes to encapsulate the help doc.

$ python
Python 2.5.1 (r251:54863, May 18 2007, 16:56:43)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def mysquare(n):
...     """
...     This function returns the square value of
...     input n
...     """
...     return n*n
...
>>> mysquare(4)
16
>>> help(mysquare)
Help on function mysquare in module __main__:

mysquare(n)
    This function returns the square value of
    input n

In Perl, you can include the Perl module documentation after __END__ keyword. The documentation syntax is in Perl pod format. You may want to see how the documentation looks like by viewing some of the modules in the perl library. Example:

$ cd /usr/lib/perl5/5.10/

$ grep -B 2 -A 10 __END__ CGI.pm
1;

__END__

=head1 NAME

CGI - Simple Common Gateway Interface Class

=head1 SYNOPSIS

  # CGI script that creates a fill-out form
  # and echoes back its values.

$ perldoc CGI.pm
CGI(3)                User Contributed Perl Documentation               CGI(3)

NAME
       CGI - Simple Common Gateway Interface Class

SYNOPSIS
         # CGI script that creates a fill-out form
         # and echoes back its values.

         use CGI qw/:standard/;
         print header,
               start_html('A Simple Example'),
               h1('A Simple Example'),
               start_form,
               "What's your name? ",textfield('name'),p,
               "What's the combination?", p,
               checkbox_group(-name=>'words',
                              -values=>['eenie','meenie','minie','moe'],
                              -defaults=>['eenie','minie']), p,
               "What's your favorite color? ",
               popup_menu(-name=>'color',
                          -values=>['red','green','blue','chartreuse']),p,
               submit,
               end_form,

However, it seems that there isn't anything equivalent for shell script. Most people will include comments at the beginning of the file just after the Shebang. Suppose the shell script documentation is located at the beginning of the file and a blank line terminates the documentation. Base on this principle, we can simply write a wrapper function (shdoc) to extract the shell script documentation.

$ shdoc()
{ 
        echo "Documentation - \"$1\""
        echo ""
        awk '
                NR>1 && /^#/ {
                        print substr($0,2,length($0))
                }
                /^$/ {
                        exit
                }' $1
}


$ cat a.sh
#! /bin/sh
#
# This program output the current date
# from the system
#
# Exit status:
# 0 - Weekday
# 1 - Sunday
# 2 - Saturday


echo "Today date is `date '+%Y-%m-%d'`"
d=`date '+%w'`
if [ $d -eq 0 ]; then
        exit 1
elif [ $d -eq 6 ]; then
        exit 2
else
        exit 0
fi

$ shdoc a.sh
Documentation - "a.sh"


 This program output the current date
 from the system

 Exit status:
 0 - Weekday
 1 - Sunday
 2 - Saturday

Now we can auto generate documentation for all my scripts without having to worry about whether the MS Words doc in-sync with the script.

IPMI on Sun x64 Server

If your system supports IPMI (Intelligent Platform Management Interface), you may want to explore to use it to monitor your hardware. Do you know that you can use it to monitor LED lights, CPU temperature, fan speed, event log, ...

For Sun hardware, IPMI is available to all their x64 types of servers. In my office, I use it monitor my Sun Fire V20z, V40z, X4500, X4600. As for SPARC servers, as far as I know it is available only to T5120 and T5220 UltraSPARC T2 type of servers or newer. Sun has a blueprint, Remote Monitoring of Sun x64 Systems using IPMITOOL and IPMIEVD, showing you how to use IPMI. BTW, ipmievd - IPMI event daemon for sending events to syslog

If you do not have IPMI, here I will show you what it can do. They output is from one of my Sun Fire X4600. BTW, some of the commands require root access.

Here shows you what are the sub-commands available in IPMI

# /usr/sfw/bin/ipmitool
No command provided!
Commands:
        raw           Send a RAW IPMI request and print response
        i2c           Send an I2C Master Write-Read command and print response
        lan           Configure LAN Channels
        chassis       Get chassis status and set power state
        event         Send pre-defined events to MC
        mc            Management Controller status and global enables
        sdr           Print Sensor Data Repository entries and readings
        sensor        Print detailed sensor information
        fru           Print built-in FRU and scan SDR for FRU locators
        sel           Print System Event Log (SEL)
        pef           Configure Platform Event Filtering (PEF)
        sol           Configure IPMIv2.0 Serial-over-LAN
        isol          Configure IPMIv1.5 Serial-over-LAN
        user          Configure Management Controller users
        channel       Configure Management Controller channels
        session       Print session information
        sunoem        OEM Commands for Sun servers
        exec          Run list of commands from file
        set           Set runtime variable for shell and exec

Here, you can check on the status of the chassis, LED lights, list of field replacement unit (FRU), event list, ...

# /usr/sfw/bin/ipmitool chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     :
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false


# /usr/sfw/bin/ipmitool sunoem led get all
sys.power.led    | ON
sys.locate.led   | OFF
sys.alert.led    | OFF
sys.psfail.led   | OFF
sys.tempfail.led | OFF
sys.fanfail.led  | OFF
ft0.fm0.led      | OFF
ft1.fm0.led      | OFF
ft2.fm0.led      | OFF
ft3.fm0.led      | OFF
io.hdd0.led      | OFF
io.hdd1.led      | OFF
io.hdd2.led      | OFF
io.hdd3.led      | OFF
p0.led           | OFF
p0.d0.led        | OFF
p0.d1.led        | OFF
p0.d2.led        | OFF
p0.d3.led        | OFF
p1.led           | OFF
p1.d0.led        | OFF
p1.d1.led        | OFF
p1.d2.led        | OFF
p1.d3.led        | OFF
p2.led           | OFF
p2.d0.led        | OFF
p2.d1.led        | OFF
p2.d2.led        | OFF
p2.d3.led        | OFF
p3.led           | OFF
p3.d0.led        | OFF
p3.d1.led        | OFF
p3.d2.led        | OFF
p3.d3.led        | OFF
p4.led           | OFF
p4.d0.led        | OFF
p4.d1.led        | OFF
p4.d2.led        | OFF
p4.d3.led        | OFF
p5.led           | OFF
p5.d0.led        | OFF
p5.d1.led        | OFF
p5.d2.led        | OFF
p5.d3.led        | OFF
p6.led           | OFF
p6.d0.led        | OFF
p6.d1.led        | OFF
p6.d2.led        | OFF
p7.led           | OFF
p7.d0.led        | OFF
p7.d1.led        | OFF
p7.d2.led        | OFF
p6.d3.led        | OFF
p7.d3.led        | OFF
p0.card.led      | OFF
p1.card.led      | OFF
p2.card.led      | OFF
p3.card.led      | OFF
p4.card.led      | OFF
p5.card.led      | OFF
p6.card.led      | OFF
p7.card.led      | OFF


# /usr/sfw/bin/ipmitool fru
FRU Device Description : Builtin FRU Device (ID 0)
 Board Product         : ASSY,SERV PROCESSOR,X4600
 Board Serial          : 1762TH1-0627002736
 Board Part Number     : 501-7382-01
 Board Extra           : 51
 Board Extra           : G4_SP
 Product Manufacturer  : SUN MICROSYSTEMS
 Product Name          : ILOM

FRU Device Description : sp.net0.fru (ID 1)
 Product Manufacturer  : MOTOROLA
 Product Name          : FAST ETHERNET CONTROLLER
 Product Part Number   : MPC8248 FCC
....


# /usr/sfw/bin/ipmitool sel list
 100 | 09/13/2006 | 21:15:27 | System Firmware Progress | Motherboard initialization
 200 | 09/13/2006 | 21:15:27 | System Firmware Progress | Video initialization
 300 | 09/13/2006 | 21:15:35 | System Firmware Progress | USB resource configuration
 400 | 09/13/2006 | 21:16:41 | System Firmware Progress | Option ROM initialization
 500 | 09/13/2006 | 21:17:31 | System Firmware Progress | System boot initiated
 600 | 09/13/2006 | 21:46:34 | System Firmware Progress | Motherboard initialization
 700 | 09/13/2006 | 21:46:34 | System Firmware Progress | Video initialization
 800 | 09/13/2006 | 21:46:42 | System Firmware Progress | USB resource configuration
 900 | 09/13/2006 | 21:46:58 | System Firmware Progress | Option ROM initialization
 a00 | 09/13/2006 | 21:47:45 | System Firmware Progress | System boot initiated
 b00 | 09/13/2006 | 22:29:10 | Power Supply #0x20 | State Deasserted
 
 
 # /usr/sfw/bin/ipmitool sdr
sys.power        | 0x02              | ok
sys.locate       | 0x01              | ok
sys.locate.btn   | 0x01              | ok
sys.alert        | 0x01              | ok
sys.psfail       | 0x01              | ok
sys.tempfail     | 0x01              | ok
sys.fanfail      | 0x01              | ok
sys.intsw        | 0x00              | ok
fp.prsnt         | 0x02              | ok
mb.t_amb0        | 31 degrees C      | ok
mb.t_amb1        | 32 degrees C      | ok
mb.t_amb2        | 37 degrees C      | ok
mb.v_bat         | 3.09 Volts        | ok
mb.v_+3v3aux_r   | 3.29 Volts        | ok
mb.v_+3v3        | 3.32 Volts        | ok
mb.v_+5v         | 5.10 Volts        | ok
mb.v_+12v        | 12.16 Volts       | ok
mb.v_-12v        | -12.20 Volts      | ok
mb.v_+2v5        | 2.56 Volts        | ok
mb.v_+1v5        | 1.52 Volts        | ok
mb.v_+1v2        | 1.26 Volts        | ok
ps0.prsnt        | 0x02              | ok
ps0.vinok        | 0x02              | ok
ps0.pwrok        | 0x02              | ok
ps1.prsnt        | 0x02              | ok
ps1.vinok        | 0x02              | ok
ps1.pwrok        | 0x02              | ok
ps2.prsnt        | 0x02              | ok
ps2.vinok        | 0x02              | ok
ps2.pwrok        | 0x02              | ok
ps3.prsnt        | 0x02              | ok
ps3.vinok        | 0x02              | ok
ps3.pwrok        | 0x02              | ok
io.prsnt         | 0x02              | ok
io.hdd0.fail     | 0x01              | ok
io.hdd1.fail     | 0x01              | ok
io.hdd2.fail     | 0x01              | ok
io.hdd3.fail     | 0x01              | ok
ft0.fm0.prsnt    | 0x02              | ok
ft0.fm0.f0.speed | 2500 RPM          | ok
ft0.fm0.fail     | 0x01              | ok
ft1.fm0.prsnt    | 0x02              | ok
ft1.fm0.f0.speed | 2500 RPM          | ok
ft1.fm0.fail     | 0x01              | ok
ft2.fm0.prsnt    | 0x02              | ok
ft2.fm0.f0.speed | 2500 RPM          | ok
ft2.fm0.fail     | 0x01              | ok
ft3.fm0.prsnt    | 0x02              | ok
ft3.fm0.f0.speed | 2600 RPM          | ok
ft3.fm0.fail     | 0x01              | ok
p0.cardok        | 0x01              | ok
p0.fail          | 0x01              | ok
p0.d0.fail       | 0x01              | ok
p0.d1.fail       | 0x01              | ok
p0.d2.fail       | 0x01              | ok
p0.d3.fail       | 0x01              | ok
p0.t_amb         | 17 degrees C      | ok
p0.v_+2v5        | 2.59 Volts        | ok
p0.v_core        | 1.34 Volts        | ok
p0.v_+3v3aux_r   | 3.22 Volts        | ok
p0.v_+12v        | 12.16 Volts       | ok
p0.v_+3v3led     | 3.08 Volts        | ok
p0.v_+1v2        | 1.19 Volts        | ok
p0.v_+1v25core   | 1.29 Volts        | ok
p1.cardok        | 0x01              | ok
p1.fail          | 0x01              | ok
p1.d0.fail       | 0x01              | ok
p1.d1.fail       | 0x01              | ok
p1.d2.fail       | 0x01              | ok
p1.d3.fail       | 0x01              | ok
p1.t_amb         | 17 degrees C      | ok
p1.v_+2v5        | 2.57 Volts        | ok
p1.v_core        | 1.34 Volts        | ok
p1.v_+3v3aux_r   | 3.22 Volts        | ok
p1.v_+12v        | 12.16 Volts       | ok
p1.v_+3v3led     | 3.08 Volts        | ok
p1.v_+1v2        | 1.18 Volts        | ok
p1.v_+1v25core   | 1.28 Volts        | ok
p2.cardok        | 0x01              | ok
p2.fail          | 0x01              | ok
p2.d0.fail       | 0x01              | ok
p2.d1.fail       | 0x01              | ok
p2.d2.fail       | 0x01              | ok
p2.d3.fail       | 0x01              | ok
p2.t_amb         | 17 degrees C      | ok
p2.v_+2v5        | 2.56 Volts        | ok
p2.v_core        | 1.36 Volts        | ok
p2.v_+3v3aux_r   | 3.22 Volts        | ok
p2.v_+12v        | 12.10 Volts       | ok
p2.v_+3v3led     | 3.09 Volts        | ok
p2.v_+1v2        | 1.19 Volts        | ok
p2.v_+1v25core   | 1.28 Volts        | ok
p3.cardok        | 0x01              | ok
p3.fail          | 0x01              | ok
p3.d0.fail       | 0x01              | ok
p3.d1.fail       | 0x01              | ok
p3.d2.fail       | 0x01              | ok
p3.d3.fail       | 0x01              | ok
p3.t_amb         | 17 degrees C      | ok
p3.v_+2v5        | 2.59 Volts        | ok
p3.v_core        | 1.34 Volts        | ok
p3.v_+3v3aux_r   | 3.22 Volts        | ok
p3.v_+12v        | 12.16 Volts       | ok
p3.v_+3v3led     | 3.08 Volts        | ok
p3.v_+1v2        | 1.18 Volts        | ok
p3.v_+1v25core   | 1.30 Volts        | ok
p4.cardok        | 0x01              | ok
p4.fail          | 0x01              | ok
p4.d0.fail       | 0x01              | ok
p4.d1.fail       | 0x01              | ok
p4.d2.fail       | 0x01              | ok
p4.d3.fail       | 0x01              | ok
p4.t_amb         | 16 degrees C      | ok
p4.v_+2v5        | 2.57 Volts        | ok
p4.v_core        | 1.36 Volts        | ok
p4.v_+3v3aux_r   | 3.22 Volts        | ok
p4.v_+12v        | 12.10 Volts       | ok
p4.v_+3v3led     | 3.08 Volts        | ok
p4.v_+1v2        | 1.18 Volts        | ok
p4.v_+1v25core   | 1.28 Volts        | ok
p5.cardok        | 0x01              | ok
p5.d0.fail       | 0x01              | ok
p5.d1.fail       | 0x01              | ok
p5.d2.fail       | 0x01              | ok
p5.d3.fail       | 0x01              | ok
p5.fail          | 0x01              | ok
p5.t_amb         | 17 degrees C      | ok
p5.v_+2v5        | 2.57 Volts        | ok
p5.v_core        | 1.36 Volts        | ok
p5.v_+3v3aux_r   | 3.22 Volts        | ok
p5.v_+12v        | 12.10 Volts       | ok
p5.v_+3v3led     | 3.08 Volts        | ok
p5.v_+1v2        | 1.18 Volts        | ok
p5.v_+1v25core   | 1.28 Volts        | ok
p6.cardok        | 0x01              | ok
p6.fail          | 0x01              | ok
p6.d0.fail       | 0x01              | ok
p6.d1.fail       | 0x01              | ok
p6.d2.fail       | 0x01              | ok
p6.d3.fail       | 0x01              | ok
p6.t_amb         | 17 degrees C      | ok
p6.v_+2v5        | 2.57 Volts        | ok
p6.v_core        | 1.34 Volts        | ok
p6.v_+3v3aux_r   | 3.22 Volts        | ok
p6.v_+12v        | 12.10 Volts       | ok
p6.v_+3v3led     | 3.07 Volts        | ok
p6.v_+1v2        | 1.18 Volts        | ok
p6.v_+1v25core   | 1.28 Volts        | ok
p7.cardok        | 0x01              | ok
p7.fail          | 0x01              | ok
p7.d0.fail       | 0x01              | ok
p7.d1.fail       | 0x01              | ok


p7.d2.fail       | 0x01              | ok
p7.d3.fail       | 0x01              | ok
p7.t_amb         | 16 degrees C      | ok
p7.v_+2v5        | 2.57 Volts        | ok
p7.v_core        | 1.36 Volts        | ok
p7.v_+3v3aux_r   | 3.20 Volts        | ok
p7.v_+12v        | 12.16 Volts       | ok
p7.v_+3v3led     | 3.08 Volts        | ok
p7.v_+1v2        | 1.18 Volts        | ok
p7.v_+1v25core   | 1.28 Volts        | ok
p0.prsnt         | 0x02              | ok
p1.prsnt         | 0x02              | ok
p2.prsnt         | 0x02              | ok
p3.prsnt         | 0x02              | ok
p4.prsnt         | 0x02              | ok
p5.prsnt         | 0x02              | ok
p6.prsnt         | 0x02              | ok
p7.prsnt         | 0x02              | ok
p0.t_core        | 37 degrees C      | ok
p1.t_core        | 34 degrees C      | ok
p2.t_core        | 31 degrees C      | ok
p3.t_core        | 30 degrees C      | ok
p4.t_core        | 36 degrees C      | ok
p5.t_core        | 34 degrees C      | ok
p6.t_core        | 33 degrees C      | ok
p7.t_core        | 36 degrees C      | ok

Solaris - Basic Security Module, Default Setting

In Solaris, the default setting for the Basic Security Model (a.k.a C2 level auditing) is turned off. To turn it on, you need to run /etc/security/bsmconv followed by a reboot. The default auditing is controlled by a few files, namely

• /etc/security/audit_startup which set the policy

  #! /bin/sh
  #
  # Copyright 2004 Sun Microsystems, Inc.  All rights reserved.
  # Use is subject to license terms.
  #
  # ident "@(#)audit_startup.txt  1.1     04/06/04 SMI"
  
  /usr/bin/echo "Starting BSM services."
  /usr/sbin/auditconfig -setpolicy +cnt
  /usr/sbin/auditconfig -conf
  /usr/sbin/auditconfig -aconf
  
  
       -aconf
           Set   the   non-attributable   audit   mask   from   the
           audit_control(4) file.
       -conf
           Configure kernel audit event to class mappings.  Runtime
           class  mappings  are changed to match those in the audit
           event to class database file.
       -setpolicy
           cnt      Do not suspend processes when  audit  resources
                    are  exhausted. Instead, drop audit records and
                    keep a count of the number of records  dropped.
                    By  default,  process are suspended until audit
                    resources become available.
  

• /etc/security/audit_user - level of logging per use

  #
  # Copyright (c) 1988 by Sun Microsystems, Inc.
  #
  # ident "@(#)audit_user.txt     1.6     00/07/17 SMI"
  #
  #
  # User Level Audit User File
  #
  # File Format
  #
  #       username:always:never
  #
  root:lo:no
  

• /etc/security/audit_control

  #
  # Copyright (c) 1988 by Sun Microsystems, Inc.
  #
  # ident "@(#)audit_control.txt  1.4     00/07/17 SMI"
  #
  dir:/var/audit
  flags:
  minfree:20
  naflags:lo
  

In the above, the default is to always audit the login/logout (lo) of root and never audit "invalid class" (no). Here is the tabulated audit class (/etc/security/audit_class)

Class	Description	Default
no	invalid class
fr	file read
fw	file write
fa	file attribute access
fm	file attribute modify
fc	file create
fd	file delete
cl	file close
nt	network
ip	ipc
na	non-attribute
lo	login or logout	root
ap	application
ss	change system state
as	system-wide administration
ua	user administration
am	administrative (meta-class)
aa	audit utilization
ad	old administrative (meta-class)
ps	process start/stop
pm	process modify
pc	process (meta-class)
xp	X - privileged/administrative operations
xc	X - object create/destroy
xs	X - operations that always silently fail, if bad
xx	X - all X events (meta-class)
io	ioctl
ex	exec
ot	other
all	all classes (meta-class)

In my previsous blog, I mentioned how I configured BSM to keep track of user commands (with arguments) executed in a login session.

For details of a specific class, you need to refer to the /etc/security/audit_event event. For example, the login/logout ("lo") class, it consists of the following events. Basically it covers all kinds of login and logout

# egrep ':lo$' /etc/security/audit_event
6152:AUE_login:login - local:lo
6153:AUE_logout:logout:lo
6154:AUE_telnet:login - telnet:lo
6155:AUE_rlogin:login - rlogin:lo
6158:AUE_rshd:rsh access:lo
6159:AUE_su:su:lo
6162:AUE_rexecd:rexecd:lo
6163:AUE_passwd:passwd:lo
6164:AUE_rexd:rexd:lo
6165:AUE_ftpd:ftp access:lo
6171:AUE_ftpd_logout:ftp logout:lo
6172:AUE_ssh:login - ssh:lo
6173:AUE_role_login:role login:lo
6212:AUE_newgrp_login:newgrp login:lo
6213:AUE_admin_authenticate:admin login:lo
6221:AUE_screenlock:screenlock - lock:lo
6222:AUE_screenunlock:screenlock - unlock:lo
6227:AUE_zlogin:login - zlogin:lo

Utility to Add/Del/Mod Configuration File

In UNIX, root (or administrator) can delegrate some of the commands to non-privilege users to carry out system administrative tasks by using sudo or RBAC (available only in Solaris

Configuation files like /etc/passwd and /etc/group have equivalent utilities (useradd, usermod, userdel, groupadd, groupmod, groupdel) to handle them properly. However, other configuration files have to rely on vi to do the editing. If someone is given the sudo to vi any file, he/she can sudo vi /etc/sudoers to temporarily include him/her to run any command.

Below script is a 'proof-of-concept' script that you may be able to allow sudo to run to add/del/mod entry in any ascii configuration file. Therefore, no need to allow vi in the sudo. BTW, only pre-defined files can be modified.

#! /bin/sh


usage()
{
        echo "Usage: $0 <config-file> add <line#> <new-data>"
        echo "       if line# > total # of lines, append new-data to end of file"
        echo "Usage: $0 <config-file> del <line#>"
        echo "Usage: $0 <config-file> mod <line#> <new-data>"
}


if [ $# -lt 3 -o $# -gt 4 ]; then
        usage
        exit 1
fi
config="$1"
action="$2"
lineno="$3"


if [ ! -f $config ]; then
        echo "Error. \"$config\" does not exist"
        exit 2
fi


#
# only allow to edit the following files in the list
#
allow="hosts defaultrouter services"
found=0
for cfg in $allow
do
        if [ `basename $config` = "$cfg" ]; then
                found=1
                break
        fi
done
if [ $found -eq 0 ]; then
        echo "Error. You can only edit \"$allow\" files"
        exit 2
fi


#
# lineno <= total
#
total=`wc -l $config | awk '{print $1}'`
if [ $lineno -gt $total ]; then
        lineno=$total
fi


#
# case - add/del/mod
#
case $action in
add)
        newdata="$4"
ex -s "$config" << EOF
${lineno}a
$newdata
.
wq!
EOF
        ;;

del)
ex -s "$config" << EOF
${lineno}d
wq!
EOF
        ;;

mod)
        newdata="$4"
ex -s "$config" << EOF
${lineno}c
$newdata
.
wq!
EOF
        ;;

*)
        usage
        exit 1
        ;;
esac

In order to accomplish the above task, we need to work with primitive command like ex. It may not be easy to find out what are the commands to feed to "ex". In UNIX Power Tool 3rd Ed Chapter 20, it showed that you can use diff -e to find out all the "ex" commands. Here is an illustration. Suppose you want to add 10.0.11.3 monitor to line 5 in the 'hosts' file, all you have to do is to a diff between the new and original file.

$ cat /etc/hosts
127.0.0.1      localhost
10.0.1.11      master loghost
10.0.11.1      gateway
10.0.11.2      storage
10.0.11.12     peter
10.0.11.13     bruce
10.0.11.14     elvis
10.0.11.15     alan
10.0.11.16     chris
10.0.11.17     andy
10.0.11.18     tony
10.0.11.19     cliff
10.0.11.20     richard

$ cat /tmp/hosts.modified
127.0.0.1      localhost
10.0.1.11      master loghost
10.0.11.1      gateway
10.0.11.2      storage
10.0.11.3      monitor
10.0.11.12     peter
10.0.11.13     bruce
10.0.11.14     elvis
10.0.11.15     alan
10.0.11.16     chris
10.0.11.17     andy
10.0.11.18     tony
10.0.11.19     cliff
10.0.11.20     richard

$ diff -e /etc/hosts /tmp/hosts.modified
5a
10.0.11.3      monitor
.

In this example, all you need to program in your script to include 10.0.11.3 monitor using ex will be:

ex -s /etc/hosts << EOF
5a
10.9.8.7   mon
.
wq!
EOF

You may want to program in the script to backup the configuration file before any modification.

Power of 2, Part 3

In Part 1 and Part 2 of the problem of "power of 2", I finally found the 2^n with 10 consecutive zeros. It took 703 CPU-hours to find that 2^559940 with 10 consecutive zeros in this number with 168559 digits.

$ python
Python 2.5.2 (r252:60911, Oct  2 2008, 09:47:11) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> n=2**559940
>>> '0'*10 in str(n)
True
>>> len(str(n))
168559

If you fit these numbers (n) (ie, 559940 100823 14007 6801 1492 1491 377 242 53 10) into Google search, you should be able to locate the rest of the number from AT&T site. They managed to find 2^n with 17 consecutive zeros. So, what's the use of it, I don't know.