The Short version:
If you use ZFS with TeamSite, you will want to limit the memory usage of the ARC cache otherwise TeamSite will constantly complain about running out of memory.
*
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache* To check values: kstat zfs
* fix the arc to 10GB (10737418240)
set zfs:zfs_arc_max=0x280000000
The Long Version
We recently deployed a new server with TeamSite 6.7.1 using ZFS as the filesystem. Almost immediately, we started getting these errors:
>>>
Your system is running low on virtual memory (130 Mb is available)
[Mon Sep 8 09:31:43 2008] Server will deactivate all stores if available memory falls below 100 Mb
<<<
Worse yet, we got a few of these:
>>>
Your system has run low on virtual memory (61 Mb is available)
[Wed Jul 30 11:03:53 2008] All stores are being deactivated
[Wed Jul 30 11:03:53 2008] Memory Monitor: releasing memory...
[Wed Jul 30 11:03:53 2008] Memory Monitor: deactivating stores...
<<<
That's odd. We have 64GB of ram and a very limited number of apps on the box.
ZFS uses a very aggressive "ARC" cache that will consume all of the free memory on the box. In theory, Solaris will release this memory if/when an application needs it. Since TeamSite tries to be proactive about memory monitoring, this can set up false positives for a low memory situation.
Some options for getting around this include:
1. Limit the ARC cache with a /etc/system setting
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_CacheWe set our ARC cache to 10GB and have not had an issue since. This is probably too low, but reduced performance is better than restarts due to deactivated stores. We will gradually increase this over time.
2. Add a large swap file
By default, we do not deploy our machines with a swap file. In theory, adding a large swap file will also reduce the likelihood of this issue. Solaris seems to release the ARC/kernel memory before resorting to a swap file. The swap file will give TeamSite the needed allusion of lots of memory. If you go with this approach, you should monitor things and make sure the swap file is not actually used as this will hurt performance.
Hope this helps someone else...
~jed
