Unable to populate iwadmin config branch.

zed0901

We began experiencing a problem with one of our Teamsite servers where the TS repository is readable but not writeable. Upon startup we see errors such as "Unable to populate iwadmin config branch". It appears that all branches are read-only due to insufficient space. We have confirmed that the filesystem where our backing store is located is only 50% full (50 G total / 25 G free). We ran iwfsck (no arguments) and it returned no errors.

When we startup TS we see the following errors in servletd_out.log

[html]
2010-02-12 16:35:49,135 [main] INFO com.interwoven.ui.teamsite.population.IwadminConfigBranch () - population step 1/7: checking the existence of iwadmin store...
2010-02-12 16:35:49,344 [main] ERROR com.interwoven.ui.teamsite.util.TeamSiteServletContextListener () - Unable to populate iwadmin config branch.
com.interwoven.ui.teamsite.population.UnableToPopulateException: iwadmin store is not running.
at com.interwoven.ui.teamsite.population.IwadminConfigBranch.populate(IwadminConfigBranch.java:141)
at com.interwoven.ui.teamsite.population.IwadminConfigBranch.populate(IwadminConfigBranch.java:113)
at com.interwoven.ui.teamsite.util.TeamSiteServletContextListener.contextInitialized(TeamSiteServletContextListener.java:41)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3764)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4216)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:736)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
at org.apache.catalina.core.StandardService.start(StandardService.java:448)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
2010-02-12 16:35:49,351 [main] INFO
[/html]


We also see the following message regarding the inodes count in iwtrace.log:

[html]

Fri Feb 12 16:34:26 2010] TEnvironment: fd limit 65536 max 65536
[Fri Feb 12 16:34:26 2010] Warning: disk low "/opt/appdata/iwstore/workflow" Less than 10 percent (259369 free of 2594009 inodes).
[Fri Feb 12 16:34:26 2010] Initialized store 0x1 at "/opt/appdata/iwstore/workflow" (version=300009, next id=0x2cbb00).
[Fri Feb 12 16:34:26 2010] TEnvironment: fd limit 65536 max 65536
[Fri Feb 12 16:34:26 2010] Warning: disk low "/opt/appdata/iwstore/default" Less than 10 percent (259369 free of 2594009 inodes).
[Fri Feb 12 16:34:26 2010] Initialized store 0x64 at "/opt/appdata/iwstore/default" (version=300009, next id=0x492800).
[Fri Feb 12 16:34:28 2010] TFSEnvironment: inode cache 30000
[Fri Feb 12 16:34:28 2010] TEnvironment: fd limit 65536 max 65536
[Fri Feb 12 16:34:28 2010] Warning: disk low "/opt/appdata/iwstore/iwadmin" Less than 10 percent (259369 free of 2594009 inodes).
[Fri Feb 12 16:34:28 2010] Initialized store 0x65 at "/opt/appdata/iwstore/iwadmin" (version=300009, next id=0x58300).
note that it's saying the /opt/appdata stores are low

[/html]


These errors are preventing the Teamsite UI from starting and are also preventing write operations to the repository.

Has anyone else experienced this problem? What are our suggestions to fix it? We opened a support case with IWOV yesterday but hopefully one of you IWOV wizards has some handy feedback. Thanks for your help!

Zed

ISCBorisB

You are running low (less than 10%, TS Default) on inodes. In such case the actual amount of the FS free space is irrelevant.
AFAIK once the inodes are allocated, you cannot change the number without re-creating the file system (not on Solaris anyway).

Talk to your Sytem Administrator on how to increase the number of inodes. You have not indicated your Unix/Linux Platform but
frankly, I think you are out of luck. You'd have to backup the data, make a new file system and restore the data. Note that TS
Backing store consists of many relatively small files referencing Content/MD Versions. Thus you can perhaps decrease inode size
in your new FS (newfs' -i switch on Solaris) to increase total number of allocated inodes, etc

If needed, you can buy some time by lowering "Disk Low" threshold to, oh say 5% instead of default 10%. iw.cfg parameters are
'disklowpercent" and "disklowwait". It looks however that you would have to recreate FS sooner rather than later