/var/log/messages and Security

Yesterday my colleague was asking me how to summarise the /var/log/messages in Linux. Below is a typical output of /var/log/messages:

Apr 13 04:06:41 myserver syslogd 1.4.1: restart.
Apr 13 04:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed 2 Celsius to 31 Celsius since last report
Apr 13 13:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 29 Celsius since last report
Apr 15 04:02:42 myserver webalizer: gethostby*.getanswer: asked for "194.216.80.218.in-addr.arpa IN PTR", got type "A"
Apr 15 04:37:23 myserver auditd[1899]: Audit daemon rotating log files
Apr 15 12:00:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed 2 Celsius to 31 Celsius since last report
Apr 15 14:00:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 29 Celsius since last report
Apr 15 14:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 27 Celsius since last report
Apr 15 14:30:23 myserver smartd[2483]: Device: /dev/sdb, Temperature changed -5 Celsius to 24 Celsius since last report
Apr 15 16:30:23 myserver smartd[2483]: Device: /dev/sdb, Temperature changed 3 Celsius to 27 Celsius since last report

Basically the fields are month, day, time, hostname, process-name, message. We can summarise the process count base on date and time. Also, we can launch the script just before mid-night to summarise that day's messages and have it email to your account. Include the below script in your crontab to run at 23:59 everyday

#! /bin/sh


month=`date '+%b'`
day=`date '+%d'`

# cron at 23:59 and sleep for 1 min in order to collect a complete messages for the day
sleep 60

awk -v m=$month -v d=$day '
$1==m && $2==d {
        split($3,t,":")
        split($5,p,"[:\(\[]")
        ind=sprintf("%s %s:00:00",p[1],t[1])
        ++s[ind]
}
END {
        for(i in s) {
                print i, s[i]
        }
}' /var/log/messages 2>/dev/null | sort | \
mailx -s "Summary of /var/log/messages" yourname@yourcompany.com

You should get an email everyday in this format:

auditd 04:00:00 1
smartd 12:00:00 1
smartd 14:00:00 3
smartd 16:00:00 1
smartd 17:00:00 2
smartd 19:00:00 1
smartd 23:00:00 1
webalizer 04:00:00 1

After running this script, my colleague realised that there were lots of sshd processes with deny access. In Linux, you can track down the bad login using lastb command. However, this is not setup by default and you need to touch /var/log/btmp to activate that. Once that is done, you will get output from lastb like below: (Note, ip and usernames are masked for secuity reason)

# lastb | head
xxuser    ssh:notty    xxx.166.139.yyy   Wed Apr 16 07:50 - 07:50  (00:00)
rxxot     ssh:notty    xxx.124.185.yyy   Wed Apr 16 04:14 - 04:14  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)

What we can do is to find out the top 10 'hacker' ip addresses so that we can include that in the /etc/hosts.deny file to block off the bad guys. We can also find out what username hackers like to try and ensure these accounts do not exist or protected with strong password.

# lastb | awk '/Apr/{++s[$3]}END{for(i in s){print i,s[i]}}' | sort -n -k 2 -r | head -10
xxx.191.254.yyy 1178
xxx.208.50.yyy 1106
xxx.108.131.yyy 1043
xxx.29.152.yyy 957
xxx.220.217.yyy 295
xxx.68.36.yyy 295
xxx.196.13.yyy 295
xxx.194.55.yyy 295
xxx.118.166.yyy 295
xxx.130.201.yyy 265
# lastb | awk '{++s[$1]}END{for(i in s){print i,s[i]}}' | sort -n -k 2 -r | head -10
root 19230
admin 3704
test 3290
guest 1442
webmaste 1146
user 1014
oracle 894
info 716
postgres 618
ftpuser 604