RPC 116 Error

Hi All,

We have been using DCTM content server 5.3.sp5 for quite a long time.

We have created a point of service layer using Java - a webservice - that has some basic services like insert/delete/search.

any java client can send request to this POS service and get the results.

We have been getting this error lately a lot - "RPC 116 error". - [DM_SESSION_E_CANT_CONNECT]error: "Could not connect to docbase (stlprod2) at host (stpdmv00) because ( errno: 0, message: Success). Network address: (INET_ADDR: family: 2, port: 5020, host: stpdmv00 (10.30.32.224, 0a1e20e0))

[DM_SESSION_E_RPC_ERROR]error: "RPC error 116 occurred: (116) Error performing send/receive.

We have diagnosed almost everything here but couldnt get it reproduced in our DEV or TEST environment.

Pl. pour your ideas as in how can I reproduce it.

Vishal.

Find more posts tagged with

Comments

mszurap

Hi,

RPC Error 116 is the most difficult thing to reproduce or to get it fixed. Basically it means that the DMCL layer is trying to communicate with the server but the network connectivity has been dropped. Maybe there is some network outages, or the socket has been closed, maybe you reached the maximum number of open sockets (maybe max open files in UNIX environment)?

If I remember well, in the old days (DFC 4.2 era) we got this error because of a DFC version difference of client and server, but nowadays this shouldn't be a problem.

Learner

Miklos,

Thanks for the information. We are still facing the issue and are not able to reproduce it in Dev or Test. Any ideas on how we can achieve it?

Thanks

vishal.goswami

Hi All,

Here is what I have tried so far to reproduce this issue but all in vain.

Hope this will help you to carry out your tests.

I decided to go by this approach based on the various possible cause of this issue.

Possible causes:

1) Content Server is not responding or going in delayed mode intermittently.

Test:

You can run java test harness program to execute the DFC calls in multiple suites which will keep runing againt CS for many hrs say 72 hrs. Take a note of response time pattern using code.

In case there is any delayed response from CS, it will get wrapped up in DFC exception.

2) The dfc client from where the requests have been issued is going in delayed mode intermittently.

Test:

You can deploy the test harness program on client machine and run java test harness program to execute the DFC calls in multiple suites and in multiple threads to increase the load which will keep runing againt CS for many hrs . Take a note of response time pattern using code.

In case there is any delayed response at client side, it will get wrapped up in DFC exception. Try to identify the response time pattern here.

3) There is some issues with the dfc native libraries installed on client machine.

Test:

This is really tuogh to test. You need to really load test libdmcl40.so (unix or solaris) / dmcl40.dll (windows). For this, pl write a thin java class to execute any DFC call againt CS. Run that class using Jmeter or self coded test harness at various load for sometime

Now to debug the library, you need to download some lib debugger or process watcher software.

4) Intermittent Network delay issue - tcp packets not able to reach to destinations or are getting delayed or lost.

Test:

this one is tricky. the default timeout of a tcp socket is 5 min on HP-UX.

whenever a new requests come to CS, it tries to communicate to the same opened tcp socket but if that sockect is ESTABLISHED, it will open new one or will use any from TIME_WAIT state.

Design you test case to issue request after every 5 mins for a much longer period of time. Put some hueristic logic to send requests like 4 mins, next requst 4min,59 secos, next 5 mins, next 5 min 20 secs....Basically they should be boderlined on 5 min period.

5) client side and content server side - reaching max open socket limit and because of this all the subsequent tcp socket open request gets timed out. DFC wraps this error as RPC 116.

Test: You can easily reproduce RPC 116 by issues thousands of dfc requests in a very short period of time. But this is very unlikely to happen in PRODUCTION.

6) Oracle DB issues.

No clue here..our DBA guys always say...- "Everything is under control"

7) Client DFC code - not releasing sessions properly. Even if one is using session manager, there should always be singleton class for session maneger.

This is very important. It would definitly resolve issue related to RPC error.

Let me know if any of these above works for you.

Vishal.