Memory Utilisation In Linux
Yesterday one of my customers asked about why Linux memory utilisation is always remain high. In order to convince everyone (including myself), I conducted the below experiment in my Linux box (with 2GB memory).
Basically the system is reporting a very high %memused of 96.92% with only 63,916 KBytes of free memory. So the question is: Is this system really running low in memory, Do I need to add more memory, What will be the best indicator for memory shortage?
If you look at the man page of proc (man proc), you can disable caching in the virtual memory:
/proc/sys/vm/drop_caches (since Linux 2.6.16)
Writing to this file causes the kernel to drop clean caches,
dentries and inodes from memory, causing that memory to become
free.
To free pagecache, use echo 1 > /proc/sys/vm/drop_caches; to
free dentries and inodes, use echo 2 > /proc/sys/vm/drop_caches;
to free pagecache, dentries and inodes, use echo 3 >
/proc/sys/vm/drop_caches.
Because this is a non-destructive operation and dirty objects
are not freeable, the user should run sync(8) first.
Step 1: Set the system to drop all cache in VM, sync the system and set it back to enable caching. You will see the %memused dropped from 96.62 to 22.73. Also the kbbuffers/kbcached dropped as well. This will be the baseline for the rest of my tests.
#grep MemTotal /proc/meminfo MemTotal: 2075640 kB #sar -r 1 Linux 2.6.18-8.1.8.el5 (chihung) 09/09/2008 03:00:06 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:00:07 PM 63916 2011724 96.92 133656 1455556 2031524 84 0.00 0 Average: 63916 2011724 96.92 133656 1455556 2031524 84 0.00 0 #cat /proc/sys/vm/drop_caches 0 #echo 3 > /proc/sys/vm/drop_caches #sync #sar -r 1 1 Linux 2.6.18-8.1.8.el5 (chihung) 09/09/2008 03:03:32 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:03:33 PM 1603800 471840 22.73 488 49512 2031524 84 0.00 0 Average: 1603800 471840 22.73 488 49512 2031524 84 0.00 0 #echo 0 > drop_caches #cat drop_caches 0
Step 2: Travserse the / partition; the system will cache all the necessary file information and of course these data has to be stored somewhere in the memory; hence the %memused goes up. Also, you can see that the second invocation of the same 'find' command is extremely fast because it does not have to go back to disk to get the file information (from 94.395 seconds to 1.169 second)
#time find / -mount -print > /dev/null real 1m34.395s user 0m0.391s sys 0m1.256s #sar -r 1 1 Linux 2.6.18-8.1.8.el5 (chihung) 09/09/2008 03:06:17 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:06:18 PM 1476196 599444 28.88 114676 62212 2031524 84 0.00 0 Average: 1476196 599444 28.88 114676 62212 2031524 84 0.00 0 #time find / -print > /dev/null real 0m1.169s user 0m0.368s sys 0m0.802s
Step 3: This is to show that my newly created file (200MB) content is memory mapped to the VM (%memused increased by 10%). Read man mmap for details.
#dd if=/dev/urandom of=/tmp/newfile bs=20k count=10000 10000+0 records in 10000+0 records out 204800000 bytes (205 MB) copied, 58.9314 seconds, 3.5 MB/s #sar -r 1 2 Linux 2.6.18-8.1.8.el5 (chihung) 09/09/2008 03:08:54 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:08:55 PM 1274512 801128 38.60 115148 262336 2031524 84 0.00 0 Average: 1274512 801128 38.60 115148 262336 2031524 84 0.00 0 #rm /tmp/newfile rm: remove regular file `/tmp/newfile'?y #sar -r 1 2 Linux 2.6.18-8.1.8.el5 (chihung) 09/09/2008 03:09:54 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:09:55 PM 1475024 600616 28.94 115216 62376 2031524 84 0.00 0 Average: 1475024 600616 28.94 115216 62376 2031524 84 0.00 0
Step 4: I tried to work on (doing an MD5) every single file in the /usr/share in the background and monitor (running sar -r) in the foreground. As you can see, I did not ask the system to store any of the file content, it is Linux's default behaviour to 'cache' the content
#du -sh /usr/share 2.2G /usr/share #at now at> find /usr/share -type f -exec md5sum {} \; > /dev/null at> <EOT> job 11 at 2008-09-09 15:10 #sar -r 5 100 Linux 2.6.18-8.1.8.el5 (chihung) 09/09/2008 03:10:35 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:10:40 PM 1393712 681928 32.85 46092 224460 2031524 84 0.00 0 03:10:45 PM 1380204 695436 33.50 46252 237752 2031524 84 0.00 0 03:10:50 PM 1361832 713808 34.39 46544 255736 2031524 84 0.00 0 03:10:55 PM 1356128 719512 34.66 46624 261420 2031524 84 0.00 0 03:11:00 PM 1315144 760496 36.64 47060 302112 2031524 84 0.00 0 03:11:05 PM 1284696 790944 38.11 47668 331500 2031524 84 0.00 0 03:11:10 PM 1258344 817296 39.38 48084 357852 2031524 84 0.00 0 03:11:10 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:11:15 PM 1233656 841984 40.57 48752 381428 2031524 84 0.00 0 03:11:20 PM 1214500 861140 41.49 49196 400216 2031524 84 0.00 0 03:11:25 PM 1177320 898320 43.28 49764 436032 2031524 84 0.00 0 03:11:30 PM 1150792 924848 44.56 50276 462236 2031524 84 0.00 0 03:11:35 PM 1116972 958668 46.19 50768 495060 2031524 84 0.00 0 03:11:40 PM 1098172 977468 47.09 51268 513144 2031524 84 0.00 0 03:11:45 PM 1087184 988456 47.62 51332 523948 2031524 84 0.00 0 03:11:45 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:11:50 PM 1071824 1003816 48.36 51508 539348 2031524 84 0.00 0 03:11:55 PM 1038660 1036980 49.96 52020 571084 2031524 84 0.00 0 03:12:00 PM 1009128 1066512 51.38 52476 600048 2031524 84 0.00 0 03:12:05 PM 983304 1092336 52.63 52836 625224 2031524 84 0.00 0 03:12:10 PM 956756 1118884 53.91 53372 650948 2031524 84 0.00 0 03:12:15 PM 940372 1135268 54.69 53744 667116 2031524 84 0.00 0 03:12:20 PM 925048 1150592 55.43 54220 681732 2031524 84 0.00 0 03:12:20 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:12:25 PM 913396 1162244 55.99 54648 693080 2031524 84 0.00 0 03:12:30 PM 909124 1166516 56.20 54944 696932 2031524 84 0.00 0 03:12:35 PM 904196 1171444 56.44 55044 701660 2031524 84 0.00 0 03:12:40 PM 898896 1176744 56.69 55328 706216 2031524 84 0.00 0 03:12:45 PM 895216 1180424 56.87 55748 709320 2031524 84 0.00 0 03:12:50 PM 892456 1183184 57.00 55816 712140 2031524 84 0.00 0 03:12:55 PM 889400 1186240 57.15 55844 715308 2031524 84 0.00 0 03:12:55 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:13:00 PM 885916 1189724 57.32 55856 718716 2031524 84 0.00 0 03:13:05 PM 882616 1193024 57.48 55872 722148 2031524 84 0.00 0 03:13:10 PM 879376 1196264 57.63 55884 725236 2031524 84 0.00 0 03:13:15 PM 875596 1200044 57.82 55956 728996 2031524 84 0.00 0 03:13:20 PM 870316 1205324 58.07 56136 734288 2031524 84 0.00 0 03:13:25 PM 846048 1229592 59.24 56328 757780 2031524 84 0.00 0 03:13:30 PM 810196 1265444 60.97 57048 792428 2031524 84 0.00 0 03:13:30 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:13:35 PM 767804 1307836 63.01 58012 833768 2031524 84 0.00 0 03:13:40 PM 726556 1349084 65.00 58744 874292 2031524 84 0.00 0 03:13:45 PM 694428 1381212 66.54 59116 905580 2031524 84 0.00 0 03:13:50 PM 649688 1425952 68.70 60144 948956 2031524 84 0.00 0 03:13:55 PM 611792 1463848 70.53 60884 985812 2031524 84 0.00 0 03:14:00 PM 568976 1506664 72.59 61592 1027748 2031524 84 0.00 0 03:14:05 PM 540652 1534988 73.95 61992 1054012 2031524 84 0.00 0 03:14:05 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:14:10 PM 509420 1566220 75.46 62756 1084120 2031524 84 0.00 0 03:14:15 PM 446928 1628712 78.47 63676 1145888 2031524 84 0.00 0 03:14:20 PM 424768 1650872 79.54 64092 1167432 2031524 84 0.00 0 03:14:25 PM 369460 1706180 82.20 65052 1221272 2031524 84 0.00 0 03:14:30 PM 343220 1732420 83.46 65544 1246748 2031524 84 0.00 0 03:14:35 PM 335232 1740408 83.85 65664 1254504 2031524 84 0.00 0 03:14:40 PM 312048 1763592 84.97 65984 1276860 2031524 84 0.00 0 03:14:40 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:14:45 PM 277980 1797660 86.61 66816 1309520 2031524 84 0.00 0 03:14:50 PM 258992 1816648 87.52 67696 1327724 2031524 84 0.00 0 03:14:55 PM 232688 1842952 88.79 68372 1353356 2031524 84 0.00 0 03:15:00 PM 211608 1864032 89.81 68876 1373732 2031524 84 0.00 0 03:15:05 PM 189792 1885848 90.86 69360 1394508 2031524 84 0.00 0 03:15:10 PM 163984 1911656 92.10 69900 1419764 2031524 84 0.00 0 03:15:15 PM 143396 1932244 93.09 70384 1440224 2031524 84 0.00 0 03:15:15 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:15:20 PM 126700 1948940 93.90 70776 1456592 2031524 84 0.00 0 03:15:25 PM 101996 1973644 95.09 71372 1479824 2031524 84 0.00 0 03:15:30 PM 70808 2004832 96.59 72040 1510020 2031524 84 0.00 0 03:15:35 PM 51624 2024016 97.51 69880 1530840 2031524 84 0.00 0 03:15:40 PM 52176 2023464 97.49 64064 1536432 2031524 84 0.00 0 03:15:45 PM 53048 2022592 97.44 59980 1540968 2031524 84 0.00 0 03:15:50 PM 52100 2023540 97.49 55668 1546316 2031524 84 0.00 0 03:15:50 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 03:15:55 PM 52396 2023244 97.48 53908 1547744 2031524 84 0.00 0 03:16:00 PM 52264 2023376 97.48 54648 1547500 2031524 84 0.00 0
In this exercise, I did not explicitly consume or allocate memory. It is the default behaviour of Linux that buffer and cache content and meta content. IMHO, you do not have to worry about high %memused, it is the non-zero and constantly rising in kbswpused, %swpused indicate swapping, that means memory shortage.
While I was driving home, I thought I may as well show my audience what swapping is like. See part 2
Labels: Linux
posted by chihungchan at 5:32 PM
3 Comments:
i think this shows that people have a primitive understanding of how modern day linux memory management works.
From one source I come to know the way to calculate the "actual" memory utilization in the past using "sar" file. Below is the way to do the same.
tch memory utilization data from sar files using -r switch.
sar -r -f filename
It will display data in this format
04:40:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
04:50:01 PM 258788 16372260 98.44 12696 13912224 12484624 94144 0.75 420
Information being displayed here, is somewhat misleading. According to it, at 04:50PM, 98.44% memory was utilized (As its merely calculated by formula kbmemused/sum(kbmemused, kbmemfree)).
But in actual that was not the case. In order to get actual memory utilization, subtract kbbuffers and kbcached from kbmemused, and then used the above formula. In this case,
Memory utilization = (16372260-12696-13912224)/(258788+16372260) = 14.71%
The reason behind this is Linux treats unused memory as a wasted resource and so uses as much RAM as it can to cache process/kernel information.
Here Buffers= amount of physical memory used as buffers for disk writes
Cached = amount of physical memory used as cache for disk reads
Please let me know if this is a correct approach for the same.
Thanks,
Suraj Sharma
Yes, you are right.
BTW, you may want to read Understand Linux Memory. In this blog, I refer to an article (with a video) that I think is definitely worth reading and watching.
Post a Comment
<< Home