Finding Failed Logins in Solaris using C2 Log

In Linux, you have lastb - last bad logins, to log failed logins. In Solaris, you can register it using loginlog(4). but it is not turn on by default.

If your environment enables auditing (aka BSM - Basic Security Module, or C2), you can extract the login information. Default setting for BSM includes login / logout. With auditreduce, you can extrace the lo (login/logout) class from the audit log. Below one-liner will help you to extract those logins.

root@chihung# auditreduce -a 20101025000000 -u chihung \
 -c lo /var/audit/20101024164715.not_terminated.chihung | praudit -l
header,85,2,ftp access,,chihung7,2010-10-25 11:37:59.141+08:00,subject,chihung,chihung,chihung,root,root,4675,4675,633 131093 host1.chihung.chan,text,bad password,return,failure,1
header,69,2,login - ssh,,chihung4,2010-10-25 11:52:19.537+08:00,subject,chihung,chihung,chihung,chihung,chihung,157,4071257189,37622 host2.chihung.chan,return,failure,Maximum number of attempts exceeded
header,69,2,login - ssh,,chihung2,2010-10-25 11:52:43.658+08:00,subject,chihung,chihung,chihung,chihung,chihung,327,372387060,376131094 host3.chihung.chan,return,failure,Permission denied

House Keep Script Not Doing Its Job

Recently I found one of my customer's scripts is not doing what it suppose to do, which is to house keep old log files. Here is how I locate the culprit.

I have been dumping out those zero link files daily for all my managed servers and realised that one of the application accounts has been choking up 2GB+ worth of memory. Since these files are having zero link, there is no way to find out the original file name. With the help of DTrace in Solaris 10, I am able to tell when this particular user trying to delete (unlink in syscall) a file. Below is a simple dtrace script that I run for a couple of days.

#! /usr/sbin/dtrace -s

#pragma D option quiet

syscall::unlink:entry
/uid==1234/
{
        printf("%-20Y pid=%d ppid=%d execname=%s args=%s\n", walltimestamp, pid, ppid, execname, copyinstr(arg0));
}

Sample output:

2010 Oct 21 20:01:15 pid=1667 ppid=29587 execname=gzip args=/opt/chihung/log/Err-s_ac.txt.20101021064218
2010 Oct 21 20:01:15 pid=29599 ppid=29587 execname=gzip args=/opt/chihung/log/ChanSecurityLog211010.log
2010 Oct 21 20:01:48 pid=1668 ppid=29587 execname=gzip args=/opt/chihung/log/ChanControlServicesLog211010.log
2010 Oct 21 20:02:21 pid=1916 ppid=29587 execname=gzip args=/opt/chihung/log/ChanBusinessServicesLog211010.log
2010 Oct 21 20:02:27 pid=2160 ppid=29587 execname=gzip args=/opt/chihung/log/ChanFixedexpenseLog211010.log
2010 Oct 21 20:02:47 pid=2290 ppid=29587 execname=gzip args=/opt/chihung/log/aclist.txt.20101021084119
2010 Oct 21 20:02:47 pid=2294 ppid=29587 execname=gzip args=/opt/chihung/log/Chan_20101020_error_13360_batch
2010 Oct 21 20:02:47 pid=2296 ppid=2295 execname=rm args=/opt/chihung/log/ChanControlServicesLog211010.log.gz
2010 Oct 21 20:02:47 pid=2297 ppid=2295 execname=rm args=/opt/chihung/log/ChanFixedexpenseLog211010.log.gz
2010 Oct 21 20:02:47 pid=2299 ppid=2295 execname=rm args=/opt/chihung/log/ChanUserLog211010.log.gz
2010 Oct 21 20:02:47 pid=2300 ppid=2295 execname=rm args=/opt/chihung/log/aclist.txt.20101021084119.gz
2010 Oct 21 20:02:47 pid=2301 ppid=2295 execname=rm args=/opt/chihung/log/Chan003_20101020_error_13359_adhoc.gz
2010 Oct 21 20:02:47 pid=2303 ppid=2295 execname=rm args=/opt/chihung/log/ChanBusinessServicesLog211010.log.gz
2010 Oct 21 20:02:47 pid=2304 ppid=2295 execname=rm args=/opt/chihung/log/ChanDepositLog211010.log.gz
2010 Oct 21 20:02:47 pid=2305 ppid=2295 execname=rm args=/opt/chihung/log/ChanNdfLog211010.log.gz
2010 Oct 21 20:02:47 pid=2307 ppid=2295 execname=rm args=/opt/chihung/log/Err-S001_20101020_error_13358_adhoc.gz

At 20:00 daily, gzip is compressing a couple of log files and that followed by rm on the gzipped files. That's wierd. Why would someone want to compress log files and remove it after that. When the application rotate log file daily, it is unable to locate the original log file handler to release the memory. Therefore, the number of zero link files increases and hence more used memory. By cross examining the scheduled jobs at 20:00, I found the following script

#!/bin/ksh -x

APPDIR=/opt/$1/log
if [[ -d $APPDIR ]]; then
        find $APPDIR \( -name "*.out" -o -name "*.gz" \) -prune -o -type f -mtime -1 -exec gzip {} \;
        find $APPDIR -name "*.gz" -mtime -1 -exec rm -f {} \;
        exit 0
fi

If you look at the man page for find(1), -mtime -1 means modified time less than 1 day! Shouldn't it be -mtime +1 - modified time older than 1 day.

Key-Value Pair in Multi-line Records

If you have to deal with multi-line records, AWK is definitely your friend. Sometimes the records come in the form of first record is the key and the next record is the value. To make matter worse, the value record can be optional, ie it may not exist at all.

For example, the following sample data set have the above characteristics. What we need is a script to list down the values for a particular pool name. The first 3 lines are headers which can we going to ignore.

$ cat testdata.txt
media media robot robot robot side/ ret    size status
 ID type type   # slot face level  KBytes
----------------------------------------------------------------------------
One pool

A00001 HCART    TLD   0   26  -  10     101568 AVAILABLE

Two pool

B00001 HCART    NONE   -   -  -  -     - AVAILABLE
B00002 HCART    NONE   -   -  -  -     - AVAILABLE

Three pool

C00001 HCART    NONE   -   -  -  -     - AVAILABLE
C00002 HCART    NONE   -   -  -  -     - AVAILABLE
C00003 HCART    NONE   -   -  -  -     - AVAILABLE

Zero pool

Four pool

D00001 HCART2   NONE   -   -  -  -     - AVAILABLE
D00002 HCART3   NONE   -   -  -  -     - AVAILABLE
D00003 HCART3   NONE   -   -  -  -     - AVAILABLE
D00004 HCART3   NONE   -   -  -  -     - AVAILABLE

By changing the default AWK FS (field separator) and RS (record separator) to be newline and blank line repsectively, we are able to handle the above multi-line records. In order to accommodate zero 'value' in a record, we need to put in the logical in the AWK code.

$ cat pool.sh
#! /bin/sh

PATH=/usr/bin:/bin:/usr/sbin:/usr/local/bin
LD_LIBRARY_PATH=/usr/lib:/lib:/usr/local/lib


if [ $# -ne 2 ]; then
 echo "Usage: $0  \n"
 exit 1
fi
txtfile=$1
pool=$2
if [ ! -f $txtfile ]; then
 echo "Error. $txtfile does not exist\n"
 exit 2
fi


sed -n '4,$p' $txtfile | awk -v pool="$pool" '
BEGIN {
 FS="\n"
 RS=""
 poolname=sprintf("%s pool", pool)
}
$1 == poolname {
 start=1
 next
}
start==1 && $0 ~ /pool/ {
 exit
}
start==1 {
 for (i=1;i<=NF;++i ) {
         print $i
 }
 exit
}'

See script in action.

$ ./pool.sh testdata.txt One
A00001 HCART    TLD   0   26  -  10     101568 AVAILABLE

$ ./pool.sh testdata.txt Two
B00001 HCART    NONE   -   -  -  -     - AVAILABLE
B00002 HCART    NONE   -   -  -  -     - AVAILABLE

$ ./pool.sh testdata.txt Three
C00001 HCART    NONE   -   -  -  -     - AVAILABLE
C00002 HCART    NONE   -   -  -  -     - AVAILABLE
C00003 HCART    NONE   -   -  -  -     - AVAILABLE

$ ./pool.sh testdata.txt Four
D00001 HCART2   NONE   -   -  -  -     - AVAILABLE
D00002 HCART3   NONE   -   -  -  -     - AVAILABLE
D00003 HCART3   NONE   -   -  -  -     - AVAILABLE
D00004 HCART3   NONE   -   -  -  -     - AVAILABLE

$ ./pool.sh testdata.txt Unknown

$ ./pool.sh testdata.txt Zero

$ 

So what is the big deal ? This is the kind of output you will get from NetBackup /usr/openv/netbackup/bin/goodies/available_media. With this script, you can handle the available_media output with ease.

Data Visualization

Stumbled upon this article - Videos and visualization - Data viz for journalism, student career paths, multi-dimensional data, and the future.. It featured this video

Journalism in the Age of Data from Geoff McGhee on Vimeo.

With the exponential increase in data, data visualization will be the next wave. The big names are working on this area: IBM is working on Many Eyes and Google is on Public Data Explorer