ZFS Adaptive Replacement Cache (ARC) relinquishes memory

One of my customers has a SunFire X4500 (a.k.a Thumper) serving as a NAS for all the Galaxy type of servers running Windows 2003 via Samba.

All servers' CPU/Memory/Disks/Network are monitored using SNMP. Something strange happens in the X4500 server 'cos the memory utilisation keep going up. Although the server has 16GB memory, it is definitely not a good sign for a production server. This seems to indicate some form of memory leak in the system. However, we are not delopying any application except Samba which comes with Solaris 10 anyway. I don't think Samba or Solaris itself has some serious memory leaking issue, so where does the memory go ?

After some searching in the Internet, I found the answer to the problem. ZFS Best Practices Guide explained that "The ZFS adaptive replacement cache (ARC) tries to use most of a system's available memory to cache file system data. The default is to use all of physical memory except 1 Gbyte. As memory pressure increases, the ARC relinquishes memory.". So, how true is this ? Let's find out.

I am lucky to have an identical X4500 for testing to verify that the Solaris kernel indeed do free up memory. I wrote a script to create a 200MB file (dd if=/dev/urandom of=/zfs/`perl -e 'print time()'` bs=1024 count=204800 > /dev/null 2>&1) in the ZFS file systems and run that for every minute as a cron job. SAR (system activity reporter) is enabled and configured to collect SAR info every 5 minutes.

sar -r of the freemem (average pages available to user processes) is plotted using Gnuplot. BTW, the size of a page in Solaris 10 x86 is 4096 (run pagesize to get the system page size). As you can see, it relinquishes memory when it is about 2GB freemem.

If we continue to run the test for hours, the freemem looks like this. BTW, the CPU utilisation is about 5% (0% usr, 5% sys) throughout.