LSDS 7.3.1 Runtime Cache (JCS) Errors | The file is corrupt.

Kapil_sape

HI All,

TS/LSDS: 7.3.1 running within the shipped Autonomy version of Tomcat
RED HAT ENTERPRISE LINUX 5.1

We are noticing errors as below in the catalina.out & livesite_runtime.log related to errors in the JCS runtime cache for LSDS (following a LSDS restart).
=========================================================
2013-05-03 08:38:38,094 WARN [org.apache.jcs.auxiliary.disk.indexed.IndexedDisk]
The file is corrupt:
Record [DED: pos = 2840504 len = 131947] does not match data length on disk (131945)
2013-05-03 08:38:38,094 ERROR [org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache] Region [page13] IO Exception, Problem reading object from file
java.io.IOException: The File Is Corrupt, need to reset
at org.apache.jcs.auxiliary.disk.indexed.IndexedDisk.readObject(IndexedDisk.java:113)
at org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache.readElement(IndexedDiskCache.java:716)
at org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache.processGet(IndexedDiskCache.java:627)
at org.apache.jcs.auxiliary.AbstractAuxiliaryCacheEventLogging.getWithEventLogging(AbstractAuxiliaryCacheEventLogging.java:91)
at org.apache.jcs.auxiliary.disk.AbstractDiskCache.doGet(AbstractDiskCache.java:795)
.........................................................
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
2013-05-03 08:38:38,095 ERROR [org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache] Region [page13] Failure getting from disk, key = destinations-en:mauritius/flights-to-mauritius
java.io.IOException: The File Is Corrupt, need to reset
at org.apache.jcs.auxiliary.disk.indexed.IndexedDisk.readObject(IndexedDisk.java:113)
at org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache.readElement(IndexedDiskCache.java:716)
at org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache.processGet(IndexedDiskCache.java:627)
at org.apache.jcs.auxiliary.AbstractAuxiliaryCacheEventLogging.getWithEventLogging(AbstractAuxiliaryCacheEventLogging.java:91)
at org.apache.jcs.auxiliary.disk.AbstractDiskCache.doGet(AbstractDiskCache.java:795)
at org.apache.jcs.auxiliary.disk.AbstractDiskCache.get(AbstractDiskCache.java:299)
..............................................................................
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
2013-05-03 08:38:38,095 WARN [org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache] Region [page13] Reseting cache
2013-05-03 08:39:44,130 ERROR [com.ba.livesite.runtime.session.SharedSessionBridge] Error from SharedSessionBridge query service [Timeout waiting for connection]
2013-05-03 08:39:44,297 ERROR [com.ba.livesite.runtime.session.SharedSessionBridge] Error from SharedSessionBridge query service [Timeout waiting for connection]
2013-05-03 08:39:49,438 ERROR [com.ba.livesite.runtime.session.SharedSessionBridge] Error from SharedSessionBridge query service [Timeout waiting for connection]
2013-05-03 08:39:52,889 ERROR [com.ba.livesite.runtime.session.SharedSessionBridge] Error from SharedSessionBridge query service [Timeout waiting for connection]
2013-05-03 08:39:54,633 WARN [org.apache.jcs.auxiliary.disk.indexed.IndexedDisk]
The file is corrupt:
Record [DED: pos = 3663822 len = 120531] does not match data length on disk (46866)
2013-05-03 08:39:54,633 ERROR [org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache] Region [page3] IO Exception, Problem reading object from file
java.io.IOException: The File Is Corrupt, need to reset

==========================================================
This is happening on across 6 servers in both preLive and Live production environments. It happens for different pages and components at different times.
This seems to be affecting the performance of pages particularly after an LSDS restart.

We have not noticed any CPU spikes or Tomcat out of memory issues. WE deliberately over sized both the 32GB RAM servers and RAM allocations for Tomcat (2GB heap)

Have 3 sites in Production:
1) destinations content - receives the most traffic across 497 pages
2) Information content - only 2 pages Live at the moment
3) Offers site - 124 pages live but receives less traffic than destinations site.

They have 51 components in our /iwadmin/main/livesite/component/STAGING/ area not all of them will be used, but most are in use across the above pages.
The Cache time for most components is "Until Published", other than a few that are either "No Cache" or set to a few minutes/hours. The majority should be "Until Published".

In addition I checked the size of /runtime/web/WEB-INF/livesite.cache dir. On the 4 live servers the size of the livesite.cache dirs are:

4.5G livesite.cache

5.1G livesite.cache

3.9G livesite.cache

3.4G livesite.cache

These servers are behind a load balancer so they do receive different levels of traffic.

What might be the reason for these errors?

Is this okay to clear contents of /runtime/web/WEB-INF/livesite.cache and /runtime/tomcat/temp?
How does the above make difference if I clear the cache through JCS.jsp admin page?

FYI: cache.ccf has not been modified and is the OOTB version.

Any inputs will be appreciated.

Thanks,
Kaps

Rick Poulin

What might be the reason for these errors?

Your cache size on disk seems excessive for that little content. I suggest opening a support case and reviewing whether the default cache config is suitable for your use cases. The only thing I can pick out in your config, is that you're using the ootb tomcat (x32) and allocating a max heap of 2GB. On Windows, the 32-bit JVM will only be able to allocate about 1.4GB of memory, and then it should **** out or behave really weirdly. This might be the source of the corruption. Either switch to 64-bit Tomcat, or cap your heap at 1.2GB.

Is this okay to clear contents of /runtime/web/WEB-INF/livesite.cache and /runtime/tomcat/temp? How does the above make difference if I clear the cache through JCS.jsp admin page?

Deleting this folder is fine, but Tomcat can't be running when you do it. I'd recommend doing so to fix your current corruption. The JCS interface clears the cache in memory but may not completely clean up the disk, so when you restart Tomcat it may reload the same corrupted files.