Dmp files getting created in Documentum Content server

Hi,

I am seeing dmp files getting created in documentum content server 7.2 with the below error in docbase log. Can you please let me know if anything is wrong. this started happening more often recently. I am seeing this more often recently and some times the Docbase service is going down. Can you please assist me here

Current Session ID: 0107a12181bd0b37
Heap information:
Current Heap:Local
Last SQL statement executed by DB was:

TID(6580) Exception Stack Dump:
Exception 00000000C0000005 (ACCESS_VIOLATION) occurred at 00007FFFC246DE0E.
RtlLockHeap+13eRtlLockHeap+13e 00007FFFC246B7E4 / 00000000197EE400
RtlEnterCriticalSection+e4RtlEnterCriticalSection+e4 00007FFF8707253C / 00000000197EE440
sltsmna+1csltsmna+1c 00007FFF86F36597 / 00000000197EE470
kpufhndl0+1dc7kpufhndl0+1dc7 00007FFF86F347AE / 00000000197EE740
kpughndl0+11fekpughndl0+11fe 00007FFF87DCC3F5 / 00000000197EE770
OCIHandleFree+15OCIHandleFree+15 00007FFFA12533E2 / 00000000197EE7A0
OCIHandleFree+6aOCIHandleFree+6a 0000000180745B11 / 00000000197EE7E0
DBCursorImpPerRPC::FreeOCIHandles+81 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\dbxx\oracle\src\dbicursperrpc.cxx:363 0000000180743374 / 00000000197EE840
DBCursorImp::~DBCursorImp+34 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\dbxx\oracle\src\dbicurs.cxx:146 00000001807434D4 / 00000000197EE880
DBCursorImp::scalar deleting destructor'+14DBCursorImp::scalar deleting destructor'+14 0000000180726B11 / 00000000197EE8B0
DBCursor::~DBCursor+91 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\dbxx\common\src\dbcursor.cxx:99 00000001801711DD / 00000000197EE9F0
dmSession::~dmSession+2dd f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\server\session\dmsess.cxx:3637 0000000180173434 / 00000000197EEA90
dmSession::scalar deleting destructor'+14dmSession::scalar deleting destructor'+14 0000000180122C37 / 00000000197EEAC0
dmSessionContext::~dmSessionContext+27 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\dmrpc.cxx:4262 00000001801232C4 / 00000000197EEAF0
dmSessionContext::scalar deleting destructor'+14dmSessionContext::scalar deleting destructor'+14 00000001801A74F5 / 00000000197EEB20
dmSessionThread::Terminate+85 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\dmsthrd.cxx:226 00000001801A72F0 / 00000000197EEB50
dmSessionThread::ShutdownForException+10 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\dmsthrd.cxx:268 00000001805FDAE2 / 00000000197EEB80
dmExtCallFailureHandler+52 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\dmapply.cxx:111 00000001805F892B / 00000000197EEBB0
ApplyForObject_string+2ab f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\dmapply.cxx:2033 00000001805F5CDD / 00000000197EEE40
iIXApplyForObject_string+3d f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\stubs\sstubs.c:8578 000000018011A260 / 00000000197EEE90
nwvss_+230nwvss_+230 00000001805F19EB / 00000000197EEF90
S_ApplyForObject_string+eb f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\stubs\sstubs.c:8635 000000018057C879 / 00000000197EF0D0
driver_Proc+ad9 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\stubs\bstubs.c:1052 000000018026F144 / 00000000197EF6F0
dmNetwiseConnection::Dispatch+134 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\dmnwisew.cxx:806 000000018013126B / 00000000197EF7C0
dmSessionThreadStart+42b f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\rpc\netwise\server\dmconn.cxx:221 0000000180121236 / 00000000197EFB90
threadMain+1b6 f:\perforce\depot\platform\contentserver\hfx\hfx_7.2_rc\dmnew\server\compat\dmthread.cxx:76 00007FFFA65B4F7F / 00000000197EFEF0
beginthreadex+107beginthreadex+107 00007FFFA65B5126 / 00000000197EFF20
endthreadex+192endthreadex+192 00007FFFC07413D2 / 00000000197EFF50
BaseThreadInitThunk+22BaseThreadInitThunk+22 00007FFFC24454F4 / 00000000197EFF80

2020-03-18T15:07:26.058000 9760[6580] 0107a12181bd0b37 Error: dm_bear_trap: Unexpected exception, (Wed Mar 18 15:07:26 2020 NT Exception: Exception Address: c246de0e, ACCESS_VIOLATION at (Connection Failure)), during new session creation in module ........\rpc\netwise\server\dmapply.cxx after line 542. Process exiting.

Comments

  • I would submit a ticket with Support. This looks like a memory leak and may be caused by windows update/patch.

  • Thanks DCTM_Guru. I will raise it. Just wanted to add, sometimes I see some ORA error too, do you think there is any issue with the DB too. I am also seeing this quite often, but this message does not come along with the above heap issue. I see these two errors together however.

    2020-03-18T08:38:51.101000 156[7476] 0107a12181bd0259 [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info: "The following error was encountered trying to get a database connection: ORA-12170: TNS:Connect timeout occurred
    The server will retry 9 more time(s) (every 30 seconds)."

  • Might be. I don't recall what happens when CS cant connect to DB after tenth time. Core dmp is not ideal way to shutdown :)

  • Typically your server would recover if there is a connection issue with the DB. You may then see warnings querying the dm_docbase_config but functionality would resume.
    In the stack, it does appear to be directly related to the database connection. Are there any errors reported in the oracle alert.log at the same time? Also, you may want to connect using sqlplus when you face an issue to ensure you can get a connection from the Documentum Server.

    Russell Kavanagh
    Documentum SME | Opentext

  • Thanks for the response DCTM_Guru and rkavanagh. Most of the times the server is recovering, however few times I noticed docbase going down. I will check again with my Oracle DBA about the alert log.

  • Any update/workaround found over this issue? @Vamsirk.masanam @rkavanagh the server recovers but it impacts the workflows in application