Stuck Thread while running a read query from DFC
Hi everyone,
We are facing a problem on production environment when trying to run a query from a custom dfc application.
The stacktrace is as follows:
-- Blocked trying to get lock: com.documentum.fc.client.impl.connection.docbase.DocbaseConnection@d1b8f07[fat lock]
com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServer(DocbaseConnection.java:1692)
com.documentum.fc.client.impl.session.Session.getServer(Session.java:3151)
com.documentum.fc.client.impl.session.Session.getServerVersion(Session.java:731)
com.documentum.fc.client.impl.session.SessionHandle.getServerVersion(SessionHandle.java:281)
com.documentum.fc.client.DfQuery.runQuery(DfQuery.java:136)
com.documentum.fc.client.DfQuery.execute(DfQuery.java:203)
com.doxbit.documentum.viewer.controller.DocSearch.getDocuments(DocSearch.java:42)
Calling code:
IDfSession dfSession = null;
IDfCollection col = null;
IDfSessionManager sessionManager = getSessionManager();
try {
dfSession = sessionManager.getSession(docbase);
DfQuery query = new DfQuery(dql.toString());
col = query.execute(dfSession, IDfQuery.READ_QUERY);
serializeResults(ow, col);
} finally {
if (dfSession != null) {
sessionManager.release(dfSession);
}
if (col != null) {
col.close();
}
}
Did anyone have such issues?
Any help will be highly appreciated,
Thanks,
Alex.
Comments
-
Please consider moving this question as-is (no need to recreate) to the proper forum for maximum visibility. Questions written to the users' own "Discussions" space don't get the same amount of attention and can go unanswered for a long time.
You can do so by selecting "Move" under ACTIONS along the upper-right. Then search for: "Documentum" and select the very first result that comes up. This will relocate it to the Developer Network subspace of the Documentum community.
For further guidance on engaging with the communities for Documentum and related products, please refer to the
0 -
Typically at least two threads take part in lock, why do not show full thread dump?
0 -
All threads get stuck at the same line, as an example:
Thread-1651 "[STUCK] ExecuteThread: '51' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: com.documentum.fc.client.impl.connection.docbase.DocbaseConnection@d1b8f07[fat lock]
com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServer(DocbaseConnection.java:1692)
com.documentum.fc.client.impl.session.Session.getServer(Session.java:3151)
com.documentum.fc.client.impl.session.Session.getServerVersion(Session.java:731)
com.documentum.fc.client.impl.session.SessionHandle.getServerVersion(SessionHandle.java:281)
com.documentum.fc.client.DfQuery.runQuery(DfQuery.java:136)
com.documentum.fc.client.DfQuery.execute(DfQuery.java:203)
com.doxbit.documentum.viewer.controller.DocSearch.getDocuments(DocSearch.java:42)
sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:141)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:420)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:393)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:741)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:691)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:624)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:95)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:82)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:93)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:56)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:62)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:32)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:86)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:52)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:150)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:154)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
Thread-1718 "[STUCK] ExecuteThread: '56' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: com.documentum.fc.client.impl.connection.docbase.DocbaseConnection@d1b8f07[fat lock]
com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServer(DocbaseConnection.java:1692)
com.documentum.fc.client.impl.session.Session.getServer(Session.java:3151)
com.documentum.fc.client.impl.session.Session.getServerVersion(Session.java:731)
com.documentum.fc.client.impl.session.SessionHandle.getServerVersion(SessionHandle.java:281)
com.documentum.fc.client.DfQuery.runQuery(DfQuery.java:136)
com.documentum.fc.client.DfQuery.execute(DfQuery.java:203)
com.doxbit.documentum.viewer.controller.DocSearch.getDocuments(DocSearch.java:42)
sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:141)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:420)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:393)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:741)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:691)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:624)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:207)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:95)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:82)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:93)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:56)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:62)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:32)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:86)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:52)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:360)
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:150)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:154)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3702)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2209)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2131)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1457)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:250)
weblogic.work.ExecuteThread.run(ExecuteThread.java:213)
You can see this two execution threads get stuck at the same line,
Thanks.
0 -
You have shown threads trying to acquire lock for DocbaseConnection@d1b8f07 object, but there should be a thread that already holds this lock and executes long running operation. I bet you are sharing the same session manager and hence the same session and the same docbaseconnection across multiple threads, that is typically wrong.
0 -
Ok, you are right we were using a shared session manager. we were using spring to inject the manager and we have some long running queries on that, first we changed from using getSession() to newSession() in order to get a private session it didn't help an occasionally we were getting this stuck thread and because response times became larger(for session construction) we reverted to getSession and we changed the scope of the session manager bean to prototype so we have private session mangers but the issue still persists. Also I notice from the heapdump that we are getting more SessionHandles than DocbaseConnections. As a example for one particular execution we were getting 23 instances of com.documentum.fc.client.impl.session.StrongSessionHandle and only 3 of com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.
0 -
first we changed from using getSession() to newSession() in order to get a private session it didn't help an occasionally we were getting this stuck thread and because response times became larger(for session construction) we reverted to getSession and we changed the scope of the session manager bean to prototype so we have private session mangers but the issue still persists.
It's hard to say what is wrong without viewing code. But I can definitely say that sharing sessionManager/session across multiple threads is not a good idea. Also I would like to suggest check these topics:
0
Categories
- All Categories
- 109 Developer Announcements
- 49 Articles
- 100 General Questions
- 122 IM Services
- 40 OpenText Hackathon
- 31 Developer Tools
- 20.6K Analytics
- 4.1K AppWorks
- 8.9K Extended ECM
- 897 Cloud Fax and Notifications
- 77 Digital Asset Management
- 9.3K Documentum
- 29 eDOCS
- 122 Exstream
- 39.8K TeamSite
- 1.7K Web Experience Management