Change Your Data To Suit Your Program

You may have some proven utilities developed in Solaris 8 that are based on pmap output and would love to port them to Solaris 10. Or you may just want to maintain a single version to handle both Solaris 8 and Solaris 10. However, the output from pmap is different from the two Solaris versions. So, shall we modify the scripts ?

Do you know that you can have another alternative. Yes, it is to change the output to suit your existing program. In this case, you do not have to go through all the testings to ensure the correctness of your scripts.

What you can do is to run a customised function (_pmap) to convert the pmap output for Solaris 10, else run the original pmap command for Solaris 8.

Here are the sample outputs and the pmap function (_pmap). The rest of your script can remain intact.

solaris8$ pmap $$
29778:  sh -i
00010000    200K read/exec         /usr/bin/ksh
00052000      8K read/write/exec   /usr/bin/ksh
00054000     40K read/write/exec     [ heap ]
FF180000    688K read/exec         /usr/lib/libc.so.1
FF23C000     32K read/write/exec   /usr/lib/libc.so.1
FF280000    576K read/exec         /usr/lib/libnsl.so.1
FF310000     40K read/write/exec   /usr/lib/libnsl.so.1
FF31A000     24K read/write/exec   /usr/lib/libnsl.so.1
FF340000     16K read/exec         /usr/lib/libmp.so.2
FF354000      8K read/write/exec   /usr/lib/libmp.so.2
FF360000      8K read/write/exec   /usr/lib/libdl.so.1
FF370000      8K read/write/exec     [ anon ]
FF380000     40K read/exec         /usr/lib/libsocket.so.1
FF39A000      8K read/write/exec   /usr/lib/libsocket.so.1
FF3A0000      8K read/exec         /usr/platform/sun4u-us3/lib/libc_psr.so.1
FF3B0000    192K read/exec         /usr/lib/ld.so.1
FF3E0000      8K read/write/exec   /usr/lib/ld.so.1
FF3E2000      8K read/write/exec   /usr/lib/ld.so.1
FFBEC000     16K read/write          [ stack ]
 total     1928K


solaris10$ pmap $$
1742:   bash
00010000     648K r-x--  /usr/bin/bash
000C0000      80K rwx--  /usr/bin/bash
000D4000     184K rwx--    [ heap ]
FF100000     864K r-x--  /lib/libc.so.1
FF1E8000      32K rwx--  /lib/libc.so.1
FF1F0000       8K rwx--  /lib/libc.so.1
FF200000     584K r-x--  /lib/libnsl.so.1
FF2A2000      40K rwx--  /lib/libnsl.so.1
FF2AC000      24K rwx--  /lib/libnsl.so.1
FF2BE000       8K rwxs-    [ anon ]
FF2D0000       8K r-x--  /platform/sun4u-us3/lib/libc_psr.so.1
FF2E0000       8K r-x--  /lib/libdl.so.1
FF2F2000       8K rwx--  /lib/libdl.so.1
FF300000       8K rwx--    [ anon ]
FF310000      48K r-x--  /lib/libsocket.so.1
FF32C000       8K rwx--  /lib/libsocket.so.1
FF340000     168K r-x--  /lib/libcurses.so.1
FF37A000      32K rwx--  /lib/libcurses.so.1
FF382000       8K rwx--  /lib/libcurses.so.1
FF390000      24K rwx--    [ anon ]
FF3A0000       8K r--s-  dev:85,0 ino:42159
FF3B0000     208K r-x--  /lib/ld.so.1
FF3F4000       8K rwx--  /lib/ld.so.1
FF3F6000       8K rwx--  /lib/ld.so.1
FFBFC000      16K rw---    [ stack ]
 total      3040K



_pmap()
{
        /usr/bin/pmap $1 | awk '
        NF>=4 {
                printf("%-10s %8s ", $1, $2)
                mode=""
                if ( substr($3,1,1) == "r" ) { mode=sprintf("read") }
                if ( substr($3,2,1) == "w" ) { mode=sprintf("%s/write", mode) }
                if ( substr($3,3,1) == "x" ) { mode=sprintf("%s/exec", mode) }
                if ( substr($3,4,1) == "s" ) { mode=sprintf("%s/shared", mode) }

                printf("%-18s ",mode)
                for(i=4;i<=NF;++i) {
                        printf("%s ",$i)
                }
                printf("\n")
                next
        }
        {
                print
        }'
}

case `uname -r` in
5.8)
        PMAP="/usr/proc/bin/pmap"
        ;;
5.10)
        PMAP=_pmap
        ;;
*)
        echo "Unknown solaris version"
        exit 1
        ;;
esac


$PMAP ....

Sed is Slow for Very Long Stream in Solaris

Today, I realised that my customer is running sed for more than an hour and the strange thing is that the input file is no more than a few MB. Also the pattern in sed is pretty straightforward doing global substitution. BTW, it is running on Solaris 10

Is this the natural of the problem that takes sed to run that long or sed is inefficient in certain circumstances?

In this exercise, I created a file with 2000 lines. The first line has 12 characters and all subsequent lines are having an increment of 12 characters with the last line of 24000 characters.

sed 's/\\\\/@/g;s/\\/@/g' took 35+ minutes on my Sun Fire V440. That's really inefficient. Okay, sed is definitely not the right tool for his job. Let's take a look at the other alternative.

Perl has this "-p" flag that allow your in-line code to be wrap around a
while (<>) { ... # your script } loop so that you can write a one-liner. Guess what, Perl took only 5 seconds to finish that substitution. Hey, that's a lot of CPU cycles saved!

Here is the code and the run time info:

$ cat run.sh
#! /bin/bash

comma()
{
        perl -e 'print "c:\\\\a\\\\b\\\\c,"x'${1:-1}
        echo ""
}

n=1
while [ $n -le $1 ]
do
        comma $n
        ((++n))
done


$ ./run.sh 2000 > run2000.txt


$ wc run2000.txt
    2000    2000 24014000 run2000.txt


$ time sed 's/\\\\/@/g;s/\\/@/g' run2000.txt > run1.txt

real    35m6.692s
user    35m5.559s
sys     0m0.430s



$ time perl -pe 's/\\\\/@/g;s/\\/@/g' run2000.txt > run2.txt

real    0m4.948s
user    0m4.491s
sys     0m0.145s


$ digest -a md5 run1.txt run2.txt
(run1.txt) = 8820c914e0e038cec9da6f0883b6d964
(run2.txt) = 8820c914e0e038cec9da6f0883b6d964


$ uname -a
SunOS chihung 5.10 Generic_118822-11 sun4u sparc SUNW,Sun-Fire-V440


$ psrinfo -v
Status of virtual processor 0 as of: 02/25/2009 00:14:28
  on-line since 12/13/2008 00:37:43.
  The sparcv9 processor operates at 1281 MHz,
        and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 02/25/2009 00:14:28
  on-line since 12/13/2008 00:37:43.
  The sparcv9 processor operates at 1281 MHz,
        and has a sparcv9 floating point processor.
Status of virtual processor 2 as of: 02/25/2009 00:14:28
  on-line since 12/13/2008 00:37:43.
  The sparcv9 processor operates at 1281 MHz,
        and has a sparcv9 floating point processor.
Status of virtual processor 3 as of: 02/25/2009 00:14:28
  on-line since 12/13/2008 00:37:41.
  The sparcv9 processor operates at 1281 MHz,
        and has a sparcv9 floating point processor.

Press Any Key To Continue

If you need to prompt the user to "Press any key to continue", the user would just expect to hit a single key to continue instead of any key + a carriage return. If you use:
echo -e "Press any key to continue. \c"; read any,
the user will need to hit the "Return key" to complete the sequence. With stty, you can achieve this without the user hitting an additional carriage return key. The below function anykey will do this trick.

The steps in the function:

1. save original terminal setting
2. print the message (without a carriage return)
3. turn off the echo input character to hide your key press
4. set terminal into a raw mode
5. use dd to a single character from terminal
6. revert back the original terminal setting

anykey ()
{
    orig=`stty -g`;
    echo -e "Press any key to continue. \c";
    stty -echo;
    stty raw;
    any=`dd if=/dev/tty bs=1 count=1 2>/dev/null`;
    stty $orig
}

The Power of Bash Shell

Recently I borrowed Shell Scripting Recipes: A Problem-Solution Approach from our National Library and found the book extremely useful. Unlike other shell scripting books which started off with a lot basic stuff, it goes straight to solving problems. Also, the author really spent a lot of time trying to explore builtin capabilities of shells ( bash, ksh, ...). By using builtin features, you can avoid running separate processes to handle a particular operation. This will improve your shell script performance tremendously.

In this blog, I will show you some of them and it's equivalent in sh

fname="/some/where/dir/file.txt"
str="to be or not to be"

bash/ksh	sh	Output
$((9*8+7))	expr 9 \* 8 + 7	79
${fname%/*}	dirname $fname	/some/where/dir
${fname##*/}	basename $fname	file.txt
${#str}	echo -n $str | wc -c	18
${str%be*}	echo $str | perl -pe 's/(.*)be.*?/\1/'	to be or not to
${str%%be*}	echo $str | sed -e 's/be.*$//'	to
${str#to*}	echo $str | perl -pe 's/to.*?//'	be or not to be
${str##to*}	echo $str | sed -e 's/^to.*//'
${str:7}	echo $str | awk '{print substr($0,8)}'	r not to be
${str:0:7}	echo $str | awk '{print substr($0,0,7)}'	to be o
${str/be/BE}	echo $str | sed -e 's/be/BE/'	to BE or not to be
${str//be/BE}	echo $str | sed -e 's/be/BE/g'	to BE or not to BE

With the above builtin features in bash, you can do quite a few things in shell scripting as shown in the book. BTW, here is a bash script demonstrates all of the above features and some practical usages:

$ cat a.sh
#! /bin/bash


fname="/some/where/dir/file.txt"
echo -e '$fname'        "\t"    $fname
echo -e '${fname%/*}'   "\t"    ${fname%/*}
echo -e '${fname%%/*}'  "\t"    ${fname%%/*}
echo -e '${fname#*/}'   "\t"    ${fname#*/}
echo -e '${fname##*/}'  "\t"    ${fname##*/}

echo

# number calaculation
echo -e 'i=$((21 + 32))'        "\t"    ; i=$((21+32))
echo -e '$((++i))'              "\t"    $((++i))
echo -e 'i*7+3 $((i*7+3))'      "\t"    $((i*7+3))

echo

# string manipulation
str="to be or not to be"
echo -e '$str'          "\t"    \"$str\"
echo -e '${#str}'       "\t"    \"${#str}\"
echo -e '${str%be*}'    "\t"    \"${str%be*}\"
echo -e '${str%%be*}'   "\t"    \"${str%%be*}\"
echo -e '${str#to*}'    "\t"    \"${str#to*}\"
echo -e '${str##to*}'   "\t"    \"${str##to*}\"
echo -e '${str:7}'      "\t"    \"${str:7}\"
echo -e '${str:0:7}'    "\t"    \"${str:0:7}\"
echo -e '${str/be/BE}'  "\t"    \"${str/be/BE}\"
echo -e '${str//be/BE}' "\t"    \"${str//be/BE}\"


echo -e "\n\nPractical use:\n"

# underline
echo $str
echo ${str//?/=}

# remove leading 0s
echo -e month=03        "\t"    ; month=03
echo -e '${month#0}'    "\t"    ${month#0}

$ ./a.sh
$fname   /some/where/dir/file.txt
${fname%/*}      /some/where/dir
${fname%%/*}
${fname#*/}      some/where/dir/file.txt
${fname##*/}     file.txt

i=$((21 + 32))
$((++i))         54
i*7+3 $((i*7+3))         381

$str     "to be or not to be"
${#str}          "18"
${str%be*}       "to be or not to "
${str%%be*}      "to "
${str#to*}       " be or not to be"
${str##to*}      ""
${str:7}         "r not to be"
${str:0:7}       "to be o"
${str/be/BE}     "to BE or not to be"
${str//be/BE}    "to BE or not to BE"


Practical use:

to be or not to be
==================
month=03
${month#0}       3

How to Remove Files ?

You may have files created with non-printable characters in the file name and have hard time removing them. Although ls has the flag -q to help you to display non-graphic characters as ?, the question mark is not the actual character. It is only a representation of the non-printable character. In order to find out the character, you can use od (octal dump) to dump them out in ASCII characters or backslash escapes format. Simply run ls -1 (minus one) and pipe the output to od -c

To remove these type of files, you can get ls -li to list out all the inode number and use the inode number (inum) in find to locate the file for removal.

# ls -li
123848 -rw-r--r-- 1 user staff 0 Feb 11 21:13 some funny file

# find . -inum 123848 -exec rm -i {} \;
rm: remove regular file `./some funny file'? y

Alternatively, you can do a rm -i ./* and say no to all except the file you want to remove. That may sound a bit dangerous.

OpenSolaris Bible - Free Chapters

Couple of free chapters from OpenSolaris Bible:

• What Is OpenSolaris
• OpenSolaris Crash Course
• ZFS

Understanding the ZFS Technology

Stumbled upon these two links:

• Video: Understanding the ZFS Technology
• ZFS Discovery Day from Sun UK events in Sep 2008