too many open files.

i keep running into the below issue frequently, may be once in a month or two.

see the below log from the servletd out log:

Dec 28, 2011 11:47:39 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Reinitializing ServerSocket
Dec 28, 2011 11:47:47 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Dec 28, 2011 11:47:47 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Reinitializing ServerSocket

Dec 28, 2011 11:47:47 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
WARNING: Reinitializing ServerSocket
getaddrinfo failed with error err=24 [No more file descriptors available for process] for addr localhost
2011-12-28 23:59:39,085 [http-8080-Processor10] ERROR com.interwoven.ui.teamsite.auth.CSClientContext () - Unexpected authentication failure
com.interwoven.cssdk.common.CSException: (Unable to establish connection with server)
at com.interwoven.cssdk.common.jni.CSOtherJNIServer.beginSessionUsingSessionString(Native Method)
at com.interwoven.cssdk.factory.CSLocalFactory.getClient(CSLocalFactory.java:178)
at com.interwoven.ui.teamsite.auth.CSClientContext.createFromSessionString(CSClientContext.java:506)
at com.interwoven.ui.teamsite.auth.CSClientContext.create(CSClientContext.java:341)
at com.interwoven.ui.base.impl.context.ContextFactory.createContext(ContextFactory.java:67)
at com.interwoven.ui.base.impl.context.ContextSetImpl.doCreateContext(ContextSetImpl.java:109)
at com.interwoven.ui.base.impl.context.AbstractContextSet.doGetContext(AbstractContextSet.java:331)
at com.interwoven.ui.base.impl.context.AbstractContextSet.getContext(AbstractContextSet.java:295)
at com.interwoven.ui.base.impl.context.AbstractContextSet.getContext(AbstractContextSet.java:255)
at com.interwoven.ui.base.impl.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:189)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.interwoven.ui.base.util.SetRequestEncodingFilter.doFilter(SetRequestEncodingFilter.java:105)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java

TeamSite UI is up and running fine but, wouldn't let users to login (we have both OS and AD users in the system)

to resolve this: i had to reset the ui service
there isn't any other helpful log entry in any of the other log files.
i guess, TS leaves file handlers open /improperly closed.
has anyone seen anything like this before?
is there any monitoring i can implement for this issue /has anyone identified the root cause for this,
I'm thinking about using lsof/and other native comamnds to figure out what/why/how so many files are open!

any pointers highly appreciated. thanks! S

Find more posts tagged with

Comments

ISCBorisB

Have you tried to identify what Files are actually open when this happens?
It is possible that some of your Client custom code does not handle Files
properly thus causing the problem.

sivaramiyer

Thanks Boris.
looks like iwserver has most of the files open on the server -

[root@dctlap-ts02 ~]# ps -ef | grep iwserver.linux
root 17006 13406 0 10:21 pts/0 00:00:00 grep iwserver.linux
iwts 27259 1 0 Dec18 ? 00:00:00 bash -c (echo 'STARTING: \c'; date; iwserver.linux -e "/var/log/Interwoven/TeamSite/iwevents.log" /data/Interwoven/iw-store; echo 'EXITED: \c'; date) 1>>"/var/log/Interwoven/TeamSite/iwtrace.log" 2>&1 &
iwts 27263 27259 10 Dec18 ? 1-02:34:06 iwserver.linux -e /var/log/Interwoven/TeamSite/iwevents.log /data/Interwoven/iw-store
[root@dctlap-ts02 ~]#
[root@dctlap-ts02 ~]#
[root@dctlap-ts02 ~]# lsof -p 27263 | wc -l
711
[root@dctlap-ts02 ~]#

StudentMarks_PR.rptdesign

Crsb

I have seen this issue before and it is related to Eventsubsystem and a significant amount of events/activities in teamsite such as a major migration with a lot of changes to files and submits.

ISCBorisB

Thanks Boris.
looks like iwserver has most of the files open on the server -

[root@dctlap-ts02 ~]# ps -ef | grep iwserver.linux
root 17006 13406 0 10:21 pts/0 00:00:00 grep iwserver.linux
iwts 27259 1 0 Dec18 ? 00:00:00 bash -c (echo 'STARTING: \c'; date; iwserver.linux -e "/var/log/Interwoven/TeamSite/iwevents.log" /data/Interwoven/iw-store; echo 'EXITED: \c'; date) 1>>"/var/log/Interwoven/TeamSite/iwtrace.log" 2>&1 &
iwts 27263 27259 10 Dec18 ? 1-02:34:06 iwserver.linux -e /var/log/Interwoven/TeamSite/iwevents.log /data/Interwoven/iw-store
[root@dctlap-ts02 ~]#
[root@dctlap-ts02 ~]#
[root@dctlap-ts02 ~]# lsof -p 27263 | wc -l
711
[root@dctlap-ts02 ~]#

700+ open Files/Pipes does seem a bit excessive.

Check Support Site, there are several KB articles addressing various aspects of "Too many open files"
issue. They *may* give you some troubleshooting ideas. Articles are pretty old though. You may want to
open up a case with support.

Adam Stoller

The next time this happens - try getting a full 'ps' listing to see if there appear to be a bunch of zombie and/or "hung" processes around. This often happens when iwperl processes stack up and don't properly exit - and since the iwperl processes are invoked from the iwserver process, it is possible that those iwperl processes are considered open file handles (just a guess).

sivaramiyer

@ghoti -
thank you! very good point (as always!).
i will get a full ps list next time this happens.

Siv

KoenDeSmet

Check if messages table of your eventsubsystem db has too many entries. In combination with teamsite search this could cause such phenomena. If it's the case truncate them.

check table :
select count(*) from messages;

truncate them:
TRUNCATE TABLE messages;
TRUNCATE TABLE message_handles;

Kind regards,

Koen

Teamsite 6.7.2 SP2 Solaris 10