Memory Utilisation In Linux, Part 2

Wanna see the symptom of memory shortage ? Here is a Python script to 'swallow' 10MB of memory every second:

#! /usr/bin/python

import time

a=[]
while 1:
        a.append('a'*10*1024*1024)
        time.sleep(1)

While the above script is running in the background, here is the output from sar -r 5 100:

# sar -r 5 100
Linux 2.6.18-8.1.8.el5 (chihung)        09/09/2008

07:14:29 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
07:14:34 PM     56396   2019244     97.28    128784   1545336   2031524        84      0.00         0
07:14:39 PM     54876   2020764     97.36    128608   1495564   2031524        84      0.00         0
07:14:44 PM     54704   2020936     97.36    127972   1445768   2031524        84      0.00         0
07:14:49 PM     55540   2020100     97.32    127496   1394408   2031524        84      0.00         0
07:14:54 PM     55600   2020040     97.32    126932   1343752   2031524        84      0.00         0
07:14:59 PM     54452   2021188     97.38    126624   1294228   2031524        84      0.00         0
07:15:04 PM     53116   2022524     97.44    126304   1245076   2031524        84      0.00         0
07:15:09 PM     54484   2021156     97.38    125884   1193316   2031524        84      0.00         0

07:15:09 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
07:15:14 PM     52784   2022856     97.46    125300   1144816   2031524        84      0.00         0
07:15:19 PM     54800   2020840     97.36    124684   1092436   2031524        84      0.00         0
07:15:24 PM     54920   2020720     97.35    124200   1052136   2031524        84      0.00         0
07:15:29 PM     51652   2023988     97.51    123724   1004592   2031524        84      0.00         0
07:15:34 PM     52280   2023360     97.48    123368    964236   2031524        84      0.00         0
07:15:39 PM     54624   2021016     97.37    122944    912644   2031524        84      0.00         0
07:15:44 PM     53756   2021884     97.41    122576    874024   2031524        84      0.00         0

07:15:44 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
07:15:49 PM     52632   2023008     97.46    122440    825380   2031524        84      0.00         0
07:15:54 PM     53488   2022152     97.42    122288    775016   2031524        84      0.00         0
07:15:59 PM     52632   2023008     97.46    122068    727192   2031524        84      0.00         0
07:16:04 PM     53220   2022420     97.44    121928    678084   2031524        84      0.00         0
07:16:09 PM     54432   2021208     97.38    121792    627572   2031524        84      0.00         0
07:16:14 PM     52196   2023444     97.49    121436    581396   2031524        84      0.00         0
07:16:19 PM     53572   2022068     97.42    121068    531860   2031524        84      0.00         0

07:16:19 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
07:16:24 PM     52192   2023448     97.49    120800    485084   2031524        84      0.00         0
07:16:29 PM     51588   2024052     97.51    109316    448756   2031524        84      0.00         0
07:16:34 PM     52528   2023112     97.47    103308    414500   2031524        84      0.00         0
07:16:39 PM     55024   2020616     97.35     96268    369416   2031524        84      0.00         0
07:16:44 PM     54404   2021236     97.38     87060    330008   2031524        84      0.00         0
07:16:49 PM     51752   2023888     97.51     79648    290004   2031524        84      0.00         0
07:16:54 PM     53224   2022416     97.44     68392    250512   2031524        84      0.00         0
07:16:59 PM     54384   2021256     97.38     59184    208384   2031524        84      0.00         0

07:16:59 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
07:17:04 PM     55544   2020096     97.32     46008    171400   2031524        84      0.00         0
07:17:09 PM     54532   2021108     97.37     28476    141884   2031396       212      0.01       128
07:17:14 PM     54372   2021268     97.38     12612    120860   2030756       852      0.04       128
07:17:19 PM     52628   2023012     97.46      2252     85396   2030116      1492      0.07       128
07:17:24 PM     51716   2023924     97.51       144     43988   2009848     21760      1.07     16244
07:17:29 PM     60688   2014952     97.08       208     44044   1956132     75476      3.72     14236
07:17:34 PM     54932   2020708     97.35       216     44068   1915068    116540      5.74     12644

07:17:34 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
07:17:39 PM     53980   2021660     97.40       216     44160   1878736    152872      7.52      1340
07:17:44 PM     55072   2020568     97.35       212     44208   1823232    208376     10.26     16704
07:17:49 PM     51976   2023664     97.50       220     44276   1779172    252436     12.43     13948
07:17:54 PM     52052   2023588     97.49       228     44176   1734720    296888     14.61      8260
07:17:59 PM     53116   2022524     97.44       236     44104   1671496    360112     17.73     19380
07:18:04 PM     51824   2023816     97.50       268     44012   1628964    402644     19.82     13436
07:18:09 PM     56040   2019600     97.30       276     44396   1557816    473792     23.32     40076

07:18:09 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
07:18:14 PM     53260   2022380     97.43       288     44364   1544164    487444     23.99      6492
07:18:19 PM     52592   2023048     97.47       288     42052   1473880    557728     27.45     27956
07:18:24 PM   1878472    197168      9.50       372     40140   1943788     87820      4.32      2520
07:18:29 PM   1878596    197044      9.49       384     40108   1943788     87820      4.32      2520
07:18:34 PM   1878844    196796      9.48       384     40108   1943788     87820      4.32      2520
07:18:39 PM   1878968    196672      9.48       392     40108   1943788     87820      4.32      2520
07:18:44 PM   1878968    196672      9.48       396     40168   1943788     87820      4.32      2520
07:18:49 PM   1878968    196672      9.48       404     40184   1943788     87820      4.32      2520

I terminated the Python script at 07:18:24. As you can see, although the %memused started at 97%, it is until 2.5 minutes later we can see swap kicked in. That is about about 1.5GB of memory allocated to the Python program. During this 2.5 minute, the system will page out whatever not necessary in the memory cache to give memory space to the Python program. That's why you do not see any memory shortage or swapping for the first two and a half minutes. However, when the Python program still keep on 'swallowing' more memory, the system has no more memory cache to page out. At this stage, the system has to resolve to swapping out memory pages to swap partition (that's will be in disk). As you can see, the 'kbswpused' and '%swpused' keep going up and this is a clear indication of memory shortage.

When we terminate our Python program, all memory previously allocated in the heap will be returned back to the system. During the entire run of the Python program, it 'swallowed' almost 1.8GB of memory and that's why you will see the %memused dropped down to less than 10%.

Memory Utilisation In Linux

Yesterday one of my customers asked about why Linux memory utilisation is always remain high. In order to convince everyone (including myself), I conducted the below experiment in my Linux box (with 2GB memory). Basically the system is reporting a very high %memused of 96.92% with only 63,916 KBytes of free memory. So the question is: Is this system really running low in memory, Do I need to add more memory, What will be the best indicator for memory shortage?

If you look at the man page of proc (man proc), you can disable caching in the virtual memory:

       /proc/sys/vm/drop_caches (since Linux 2.6.16)
              Writing  to  this  file  causes the kernel to drop clean caches,
              dentries and inodes from memory, causing that memory  to  become
              free.

              To  free  pagecache,  use  echo 1 > /proc/sys/vm/drop_caches; to
              free dentries and inodes, use echo 2 > /proc/sys/vm/drop_caches;
              to   free   pagecache,   dentries  and  inodes,  use  echo  3  >
              /proc/sys/vm/drop_caches.

              Because this is a non-destructive operation  and  dirty  objects
              are not freeable, the user should run sync(8) first.

Step 1: Set the system to drop all cache in VM, sync the system and set it back to enable caching. You will see the %memused dropped from 96.62 to 22.73. Also the kbbuffers/kbcached dropped as well. This will be the baseline for the rest of my tests.

# grep MemTotal /proc/meminfo
MemTotal:      2075640 kB


# sar -r 1
Linux 2.6.18-8.1.8.el5 (chihung)        09/09/2008

03:00:06 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:00:07 PM     63916   2011724     96.92    133656   1455556   2031524        84      0.00         0
Average:        63916   2011724     96.92    133656   1455556   2031524        84      0.00         0


# cat /proc/sys/vm/drop_caches
0


# echo 3 > /proc/sys/vm/drop_caches


# sync


# sar -r 1 1
Linux 2.6.18-8.1.8.el5 (chihung)        09/09/2008

03:03:32 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:03:33 PM   1603800    471840     22.73       488     49512   2031524        84      0.00         0
Average:      1603800    471840     22.73       488     49512   2031524        84      0.00         0

# echo 0 > drop_caches


# cat drop_caches
0

Step 2: Travserse the / partition; the system will cache all the necessary file information and of course these data has to be stored somewhere in the memory; hence the %memused goes up. Also, you can see that the second invocation of the same 'find' command is extremely fast because it does not have to go back to disk to get the file information (from 94.395 seconds to 1.169 second)

# time find / -mount -print > /dev/null

real    1m34.395s
user    0m0.391s
sys     0m1.256s


# sar -r 1 1
Linux 2.6.18-8.1.8.el5 (chihung)        09/09/2008

03:06:17 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:06:18 PM   1476196    599444     28.88    114676     62212   2031524        84      0.00         0
Average:      1476196    599444     28.88    114676     62212   2031524        84      0.00         0


# time find / -print > /dev/null

real    0m1.169s
user    0m0.368s
sys     0m0.802s

Step 3: This is to show that my newly created file (200MB) content is memory mapped to the VM (%memused increased by 10%). Read man mmap for details.

# dd if=/dev/urandom of=/tmp/newfile bs=20k count=10000
10000+0 records in
10000+0 records out
204800000 bytes (205 MB) copied, 58.9314 seconds, 3.5 MB/s


# sar -r 1 2
Linux 2.6.18-8.1.8.el5 (chihung)        09/09/2008

03:08:54 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:08:55 PM   1274512    801128     38.60    115148    262336   2031524        84      0.00         0
Average:      1274512    801128     38.60    115148    262336   2031524        84      0.00         0


# rm /tmp/newfile
rm: remove regular file `/tmp/newfile'? y


# sar -r 1 2
Linux 2.6.18-8.1.8.el5 (chihung)        09/09/2008

03:09:54 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:09:55 PM   1475024    600616     28.94    115216     62376   2031524        84      0.00         0
Average:      1475024    600616     28.94    115216     62376   2031524        84      0.00         0

Step 4: I tried to work on (doing an MD5) every single file in the /usr/share in the background and monitor (running sar -r) in the foreground. As you can see, I did not ask the system to store any of the file content, it is Linux's default behaviour to 'cache' the content

# du -sh /usr/share
2.2G    /usr/share


# at now
at> find /usr/share -type f -exec md5sum {} \; > /dev/null
at> <EOT>
job 11 at 2008-09-09 15:10


# sar -r 5 100
Linux 2.6.18-8.1.8.el5 (chihung)        09/09/2008

03:10:35 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:10:40 PM   1393712    681928     32.85     46092    224460   2031524        84      0.00         0
03:10:45 PM   1380204    695436     33.50     46252    237752   2031524        84      0.00         0
03:10:50 PM   1361832    713808     34.39     46544    255736   2031524        84      0.00         0
03:10:55 PM   1356128    719512     34.66     46624    261420   2031524        84      0.00         0
03:11:00 PM   1315144    760496     36.64     47060    302112   2031524        84      0.00         0
03:11:05 PM   1284696    790944     38.11     47668    331500   2031524        84      0.00         0
03:11:10 PM   1258344    817296     39.38     48084    357852   2031524        84      0.00         0

03:11:10 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:11:15 PM   1233656    841984     40.57     48752    381428   2031524        84      0.00         0
03:11:20 PM   1214500    861140     41.49     49196    400216   2031524        84      0.00         0
03:11:25 PM   1177320    898320     43.28     49764    436032   2031524        84      0.00         0
03:11:30 PM   1150792    924848     44.56     50276    462236   2031524        84      0.00         0
03:11:35 PM   1116972    958668     46.19     50768    495060   2031524        84      0.00         0
03:11:40 PM   1098172    977468     47.09     51268    513144   2031524        84      0.00         0
03:11:45 PM   1087184    988456     47.62     51332    523948   2031524        84      0.00         0

03:11:45 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:11:50 PM   1071824   1003816     48.36     51508    539348   2031524        84      0.00         0
03:11:55 PM   1038660   1036980     49.96     52020    571084   2031524        84      0.00         0
03:12:00 PM   1009128   1066512     51.38     52476    600048   2031524        84      0.00         0
03:12:05 PM    983304   1092336     52.63     52836    625224   2031524        84      0.00         0
03:12:10 PM    956756   1118884     53.91     53372    650948   2031524        84      0.00         0
03:12:15 PM    940372   1135268     54.69     53744    667116   2031524        84      0.00         0
03:12:20 PM    925048   1150592     55.43     54220    681732   2031524        84      0.00         0

03:12:20 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:12:25 PM    913396   1162244     55.99     54648    693080   2031524        84      0.00         0
03:12:30 PM    909124   1166516     56.20     54944    696932   2031524        84      0.00         0
03:12:35 PM    904196   1171444     56.44     55044    701660   2031524        84      0.00         0
03:12:40 PM    898896   1176744     56.69     55328    706216   2031524        84      0.00         0
03:12:45 PM    895216   1180424     56.87     55748    709320   2031524        84      0.00         0
03:12:50 PM    892456   1183184     57.00     55816    712140   2031524        84      0.00         0
03:12:55 PM    889400   1186240     57.15     55844    715308   2031524        84      0.00         0

03:12:55 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:13:00 PM    885916   1189724     57.32     55856    718716   2031524        84      0.00         0
03:13:05 PM    882616   1193024     57.48     55872    722148   2031524        84      0.00         0
03:13:10 PM    879376   1196264     57.63     55884    725236   2031524        84      0.00         0
03:13:15 PM    875596   1200044     57.82     55956    728996   2031524        84      0.00         0
03:13:20 PM    870316   1205324     58.07     56136    734288   2031524        84      0.00         0
03:13:25 PM    846048   1229592     59.24     56328    757780   2031524        84      0.00         0
03:13:30 PM    810196   1265444     60.97     57048    792428   2031524        84      0.00         0

03:13:30 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:13:35 PM    767804   1307836     63.01     58012    833768   2031524        84      0.00         0
03:13:40 PM    726556   1349084     65.00     58744    874292   2031524        84      0.00         0
03:13:45 PM    694428   1381212     66.54     59116    905580   2031524        84      0.00         0
03:13:50 PM    649688   1425952     68.70     60144    948956   2031524        84      0.00         0
03:13:55 PM    611792   1463848     70.53     60884    985812   2031524        84      0.00         0
03:14:00 PM    568976   1506664     72.59     61592   1027748   2031524        84      0.00         0
03:14:05 PM    540652   1534988     73.95     61992   1054012   2031524        84      0.00         0

03:14:05 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:14:10 PM    509420   1566220     75.46     62756   1084120   2031524        84      0.00         0
03:14:15 PM    446928   1628712     78.47     63676   1145888   2031524        84      0.00         0
03:14:20 PM    424768   1650872     79.54     64092   1167432   2031524        84      0.00         0
03:14:25 PM    369460   1706180     82.20     65052   1221272   2031524        84      0.00         0
03:14:30 PM    343220   1732420     83.46     65544   1246748   2031524        84      0.00         0
03:14:35 PM    335232   1740408     83.85     65664   1254504   2031524        84      0.00         0
03:14:40 PM    312048   1763592     84.97     65984   1276860   2031524        84      0.00         0

03:14:40 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:14:45 PM    277980   1797660     86.61     66816   1309520   2031524        84      0.00         0
03:14:50 PM    258992   1816648     87.52     67696   1327724   2031524        84      0.00         0
03:14:55 PM    232688   1842952     88.79     68372   1353356   2031524        84      0.00         0
03:15:00 PM    211608   1864032     89.81     68876   1373732   2031524        84      0.00         0
03:15:05 PM    189792   1885848     90.86     69360   1394508   2031524        84      0.00         0
03:15:10 PM    163984   1911656     92.10     69900   1419764   2031524        84      0.00         0
03:15:15 PM    143396   1932244     93.09     70384   1440224   2031524        84      0.00         0

03:15:15 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:15:20 PM    126700   1948940     93.90     70776   1456592   2031524        84      0.00         0
03:15:25 PM    101996   1973644     95.09     71372   1479824   2031524        84      0.00         0
03:15:30 PM     70808   2004832     96.59     72040   1510020   2031524        84      0.00         0
03:15:35 PM     51624   2024016     97.51     69880   1530840   2031524        84      0.00         0
03:15:40 PM     52176   2023464     97.49     64064   1536432   2031524        84      0.00         0
03:15:45 PM     53048   2022592     97.44     59980   1540968   2031524        84      0.00         0
03:15:50 PM     52100   2023540     97.49     55668   1546316   2031524        84      0.00         0

03:15:50 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
03:15:55 PM     52396   2023244     97.48     53908   1547744   2031524        84      0.00         0
03:16:00 PM     52264   2023376     97.48     54648   1547500   2031524        84      0.00         0

In this exercise, I did not explicitly consume or allocate memory. It is the default behaviour of Linux that buffer and cache content and meta content. IMHO, you do not have to worry about high %memused, it is the non-zero and constantly rising in kbswpused, %swpused indicate swapping, that means memory shortage.

While I was driving home, I thought I may as well show my audience what swapping is like. See part 2

Understand Linux Memory

My colleague is managing a set of Redhat Linux servers for customer. An ex-colleague of ours gave him a script to run as a cron job to monitor free memory and email will be sent out if it is below certain threshold. What the script does is basically work out the percentage of free memory from /proc/meminfo based on this equation - 100*MemFree/MemTotal. Guess what happen, tonnes of email alerts will flood your Inbox and you will start to worry about not having enough memory.

However, the author of the script basically do not have a deep understanding of how memory is allocated, used, buffered, cached and released.

If you have an hour to spare, I strongly recommend you to watch the video in this blog - The Answer to Free Memory, Swap, Oracle, and Everything. If you want to convince your customer that the server has enough memory to run whatever application, just ask them to watch that video.

Ubuntu on my old notebook

Downloaded the latest Ubuntu 8.04 this morning and installed on my old Toshiba notebook. Now my notebook has Windows XP, Fedora Core 8 and Ubuntu 8.04. Actually I wanted to reserve a partition to install Solaris, but it just refused to recognise my keyboard. How can one install an operating system without a keyboard.

/var/log/messages and Security, Part 2

This morning I blogged about the how to find out what ip addresses the hackers are trying to ssh into your server. In order to be more proactive, I implemented this auto-include script to run as a cron job so that I can automatically include those ip addresses into the /etc/hosts.deny if they are not already there.

#! /bin/sh
# automatically include the top 10 ip addressess that deny ssh login logged in lastb


for i in `lastb -i | awk '$2~/^ssh:/{++s[$3]}END{for(i in s){print i,s[i]}}' | sort -n -k 2 -r | head | cut -f1 -d" "`
do
        egrep ":$i$|:$i,|,$i,|,$i$" /etc/hosts.deny > /dev/null 2>&1 || echo "sshd:$i" >> /etc/hosts.deny
done

/var/log/messages and Security

Yesterday my colleague was asking me how to summarise the /var/log/messages in Linux. Below is a typical output of /var/log/messages:

Apr 13 04:06:41 myserver syslogd 1.4.1: restart.
Apr 13 04:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed 2 Celsius to 31 Celsius since last report
Apr 13 13:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 29 Celsius since last report
Apr 15 04:02:42 myserver webalizer: gethostby*.getanswer: asked for "194.216.80.218.in-addr.arpa IN PTR", got type "A"
Apr 15 04:37:23 myserver auditd[1899]: Audit daemon rotating log files
Apr 15 12:00:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed 2 Celsius to 31 Celsius since last report
Apr 15 14:00:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 29 Celsius since last report
Apr 15 14:30:23 myserver smartd[2483]: Device: /dev/sda, Temperature changed -2 Celsius to 27 Celsius since last report
Apr 15 14:30:23 myserver smartd[2483]: Device: /dev/sdb, Temperature changed -5 Celsius to 24 Celsius since last report
Apr 15 16:30:23 myserver smartd[2483]: Device: /dev/sdb, Temperature changed 3 Celsius to 27 Celsius since last report

Basically the fields are month, day, time, hostname, process-name, message. We can summarise the process count base on date and time. Also, we can launch the script just before mid-night to summarise that day's messages and have it email to your account. Include the below script in your crontab to run at 23:59 everyday

#! /bin/sh


month=`date '+%b'`
day=`date '+%d'`

# cron at 23:59 and sleep for 1 min in order to collect a complete messages for the day
sleep 60

awk -v m=$month -v d=$day '
$1==m && $2==d {
        split($3,t,":")
        split($5,p,"[:\(\[]")
        ind=sprintf("%s %s:00:00",p[1],t[1])
        ++s[ind]
}
END {
        for(i in s) {
                print i, s[i]
        }
}' /var/log/messages 2>/dev/null | sort | \
mailx -s "Summary of /var/log/messages" yourname@yourcompany.com

You should get an email everyday in this format:

auditd 04:00:00 1
smartd 12:00:00 1
smartd 14:00:00 3
smartd 16:00:00 1
smartd 17:00:00 2
smartd 19:00:00 1
smartd 23:00:00 1
webalizer 04:00:00 1

After running this script, my colleague realised that there were lots of sshd processes with deny access. In Linux, you can track down the bad login using lastb command. However, this is not setup by default and you need to touch /var/log/btmp to activate that. Once that is done, you will get output from lastb like below: (Note, ip and usernames are masked for secuity reason)

# lastb | head
xxuser    ssh:notty    xxx.166.139.yyy   Wed Apr 16 07:50 - 07:50  (00:00)
rxxot     ssh:notty    xxx.124.185.yyy   Wed Apr 16 04:14 - 04:14  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:17 - 01:17  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
xxisha    ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)
axxisha   ssh:notty    xxx.212.57.yyy    Wed Apr 16 01:16 - 01:16  (00:00)

What we can do is to find out the top 10 'hacker' ip addresses so that we can include that in the /etc/hosts.deny file to block off the bad guys. We can also find out what username hackers like to try and ensure these accounts do not exist or protected with strong password.

# lastb | awk '/Apr/{++s[$3]}END{for(i in s){print i,s[i]}}' | sort -n -k 2 -r | head -10
xxx.191.254.yyy 1178
xxx.208.50.yyy 1106
xxx.108.131.yyy 1043
xxx.29.152.yyy 957
xxx.220.217.yyy 295
xxx.68.36.yyy 295
xxx.196.13.yyy 295
xxx.194.55.yyy 295
xxx.118.166.yyy 295
xxx.130.201.yyy 265
# lastb | awk '{++s[$1]}END{for(i in s){print i,s[i]}}' | sort -n -k 2 -r | head -10
root 19230
admin 3704
test 3290
guest 1442
webmaste 1146
user 1014
oracle 894
info 716
postgres 618
ftpuser 604

Is it 32-bit or 64-bit

How to find whether an executable or a running process is 32-bit or 64-bit. I will show you in Solaris as well as in Linux. It will be based on this simple C program.

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
        unsigned int t;
        t=(unsigned int)atoi(argv[1]);
        sleep(t);
        exit(0);
}

In Solaris, you can differentiate it from either the dynamic dependencies of the executable or the address space of a running process

$ uname -a
SunOS myserver 5.10 Generic_118833-36 sun4u sparc SUNW,UltraSPARC-IIi-cEngine

$ isainfo -v
64-bit sparcv9 applications
        vis
32-bit sparc applications
        vis v8plus div32 mul32

$ which gcc
/usr/sfw/bin/gcc

$ gcc --version
gcc (GCC) 3.4.3 (csl-sol210-3_4-branch+sol_rpath)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ gcc -o a64 -m64 a.c

$ gcc -o a32 -m32 a.c

$ ldd a32
        libc.so.1 =>     /lib/libc.so.1
        libm.so.2 =>     /lib/libm.so.2

$ ldd a64
        libc.so.1 =>     /lib/64/libc.so.1
        libm.so.2 =>     /lib/64/libm.so.2

$ ./a32 100 &
[1] 3392

$ pmap 3392
3392:   ./a32 100
00010000       8K r-x--  /export/home/chihung/tmp/a32
00020000       8K rwx--  /export/home/chihung/tmp/a32
FF280000     864K r-x--  /lib/libc.so.1
FF368000      32K rwx--  /lib/libc.so.1
FF370000       8K rwx--  /lib/libc.so.1
FF3A0000      24K rwx--    [ anon ]
FF3B0000     184K r-x--  /lib/ld.so.1
FF3EE000       8K rwx--  /lib/ld.so.1
FF3F0000       8K rwx--  /lib/ld.so.1
FFBFE000       8K rwx--    [ stack ]
 total      1152K

$ ./a64 100 &
[2] 3394

$ pmap 3394
3394:   ./a64 100
0000000100000000          8K r-x--  /export/home/chihung/tmp/a64
0000000100100000          8K rwx--  /export/home/chihung/tmp/a64
FFFFFFFF7F000000          8K rwx--    [ anon ]
FFFFFFFF7F100000         24K rwx--    [ anon ]
FFFFFFFF7F200000        920K r-x--  /lib/sparcv9/libc.so.1
FFFFFFFF7F3E6000         64K rwx--  /lib/sparcv9/libc.so.1
FFFFFFFF7F3F6000          8K rwx--  /lib/sparcv9/libc.so.1
FFFFFFFF7F500000          8K rwx--    [ anon ]
FFFFFFFF7F600000        176K r-x--  /lib/sparcv9/ld.so.1
FFFFFFFF7F72C000         16K rwx--  /lib/sparcv9/ld.so.1
FFFFFFFF7FFFE000          8K rw---    [ stack ]
         total         1248K

In Linux, you can do the same as what we did for Solaris.
CentOS-5 32-bit

$ uname -a
Linux myserver 2.6.18-8.1.8.el5 #1 SMP Tue Jul 10 06:50:22 EDT 2007 i686 i686 i386 GNU/Linux

$ gcc --version
gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-52)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ gcc -m32 -o a32 a.c

$ ./a32 100 &
[1] 4305

$ ldd a32
        linux-gate.so.1 =>  (0x008ef000)
        libc.so.6 => /lib/libc.so.6 (0x0070a000)
        /lib/ld-linux.so.2 (0x006e8000)

$ cat /proc/4305/maps
00477000-00478000 r-xp 00477000 00:00 0          [vdso]
006e8000-00701000 r-xp 00000000 fd:00 7307266    /lib/ld-2.5.so
00701000-00702000 r-xp 00018000 fd:00 7307266    /lib/ld-2.5.so
00702000-00703000 rwxp 00019000 fd:00 7307266    /lib/ld-2.5.so
0070a000-00841000 r-xp 00000000 fd:00 7310310    /lib/libc-2.5.so
00841000-00843000 r-xp 00137000 fd:00 7310310    /lib/libc-2.5.so
00843000-00844000 rwxp 00139000 fd:00 7310310    /lib/libc-2.5.so
00844000-00847000 rwxp 00844000 00:00 0
08048000-08049000 r-xp 00000000 fd:00 1444299    /home/chihung/a32
08049000-0804a000 rw-p 00000000 fd:00 1444299    /home/chihung/a32
b7fa6000-b7fa7000 rw-p b7fa6000 00:00 0
b7fbf000-b7fc0000 rw-p b7fbf000 00:00 0
bfb33000-bfb48000 rw-p bfb33000 00:00 0          [stack]

RHEL3 64-bit

$ gcc --version
gcc (GCC) 3.2.3 20030502 (Red Hat Linux 3.2.3-52)
Copyright (C) 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ uname -a
Linux myserver 2.4.21-32.ELsmp #1 SMP Fri Apr 15 21:03:28 EDT 2005 x86_64 x86_64 x86_64 GNU/Linux

$ gcc -m64 -o a64 a.c

$ ./a64 100 &
[1] 21632

$ ldd a64
        libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a9567c000)
        /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x0000002a95556000)

$ cat /proc/21632/maps
0000000000400000-0000000000401000 r-xp 0000000000000000 08:02 1255888  /home/chihung/a64
0000000000500000-0000000000501000 rw-p 0000000000000000 08:02 1255888  /home/chihung/a64
0000002a95556000-0000002a9566b000 r-xp 0000000000000000 08:02 537723  /lib64/ld-2.3.2.so
0000002a9566b000-0000002a9566c000 rw-p 0000000000015000 08:02 537723  /lib64/ld-2.3.2.so
0000002a9566c000-0000002a9566d000 rw-p 0000000000000000 00:00 0
0000002a9567c000-0000002a957b7000 r-xp 0000000000000000 08:02 944733  /lib64/tls/libc-2.3.2.so
0000002a957b7000-0000002a958b7000 ---p 000000000013b000 08:02 944733  /lib64/tls/libc-2.3.2.so
0000002a958b7000-0000002a958bc000 rw-p 000000000013b000 08:02 944733  /lib64/tls/libc-2.3.2.so
0000002a958bc000-0000002a958c1000 rw-p 0000000000000000 00:00 0
0000007fbfff9000-0000007fc0000000 rw-p ffffffffffffb000 00:00 0