Performance issue with DFS using Kerberos authentication

Hi,

I'm currently migrating a client application to use Kerberos but I'm experiencing huge performance issues.

The response time of a service using Kerberos authentification is 3.3 times longer than using the standard authentification (BasicIdentity).

Am I missing something or are these performances usually expected ?

Please find attached the snippet i'm using to mesure my performances.

DFS 6.7sp1 on Websphere 7

Content server 6.7sp1 on Windows Server 2008 R2 x64

krb5.conf

ServiceTicketGenerator.java

Find more posts tagged with

Comments

Jeremy Saumen

Further investigations ...

I tried to watch the response time between the DFS server and the KDC using wireshark and it seems the response time is between 1 and 2 ms so that's not it.

If I activate the dfs-runtime logs (see attached file), the time seems to be always lost between these 2 lines :


10:35:31,977 [WebContainer : 0] DEBUG com.emc.documentum.fs.tracing.MethodInfo - ...getDecodedValue [finished] 
10:35:32,201 [WebContainer : 0] DEBUG com.emc.documentum.fs.tracing.MethodInfo - ..authenticate [finished]


10:35:32,433 [WebContainer : 0] DEBUG com.emc.documentum.fs.tracing.MethodInfo - ...getDecodedValue [finished] 
10:35:32,653 [WebContainer : 0] DEBUG com.emc.documentum.fs.tracing.MethodInfo - ..authenticate [finished]


10:35:33,022 [WebContainer : 0] DEBUG com.emc.documentum.fs.tracing.MethodInfo - ...getDecodedValue [finished] 
10:35:33,245 [WebContainer : 0] DEBUG com.emc.documentum.fs.tracing.MethodInfo - ..authenticate [finished]

Is there a way to have even more informations so i can know what happened between these lines ?

Thanks in advance

abc123

Though it is not seen much network overhead with the wireshark result, it is recommended to use keytab file at DFS server side so that the communication between DFS server and KDC can be minimized. The thing happened between two lines you mentioned is the Kerberos service ticket accept and delegation using JGSS. It is all about Kerberos. DCTM related authentcation hasn't happened at that line.

If your issue is not resolved with keytab file, please open a consult request.

William

Jeremy Saumen

Hi Wei Zhou,

Thanks for your answer.

Do you mean there is another way to use DFS and Kerberos without a keytab ?

Because the documentation seems to indicate that the keytab is mandatory on the DFS Server.

EMC Documentum Foundation Services Version 6.7 Deployment GuideKerberos and JAAS configurationTo enable Kerberos support on the DFS server, the following files are required:• Kerberos configuration file• Kerberos keytab file• JAAS configuration

I have just re-checked my jaas-config and krb5.ini. It seems I had forgot to add the line "default_keytab_name = C:\APP\kerberos\EcmDevKbs.keytab" in section [libdefaults] ... but no luck, it doesn't work better.

abc123

No, I was meant to use keytab. It is OK if you have used it. DFS Kerberos auth does use session manager cache, and it is confirmed from your log. If you were talking about DFC level2 session pool, I neither saw any unexpected info in logs.

William

Jeremy Saumen

Hi Wei Zhou,

You are right, I misinterpreted the logs and by activating the krb classes logs, it seems I also misinterpreted the logs wireshark was sending me back because they contradict what I'm finding now :


[1/2/12 18:42:20:621 CET] 00000019 SystemOut     O [KRB_DBG_AS] KrbAsReq:WebContainer : 0: >>> KrbAsReq calling createMessage
[1/2/12 18:42:20:621 CET] 00000019 SystemOut     O [KRB_DBG_AS] KrbAsReq:WebContainer : 0: >>> KrbAsReq in createMessage
[1/2/12 18:42:20:621 CET] 00000019 SystemOut     O [KRB_DBG_AS] KrbAsReq:WebContainer : 0: request etypes: [1] rc4-hmac[2] des-cbc-md5
[1/2/12 18:42:20:621 CET] 00000019 SystemOut     O [KRB_DBG_KDC] KrbKdcReq:WebContainer : 0: >>> KrbKdcReq send: kdc=AD.DUMMY.COM TCP:88, timeout=30000, number of retries =3, #bytes=312
[1/2/12 18:42:20:833 CET] 00000019 SystemOut     O [KRB_DBG_KDC] KrbKdcReq:WebContainer : 0: >>> KrbKdcReq receive: #bytes read=1634
[1/2/12 18:42:20:833 CET] 00000019 SystemOut     O [KRB_DBG_KDC] KrbKdcReq:WebContainer : 0: >>> KrbKdcReq receive: #bytes read=1634
[1/2/12 18:42:20:834 CET] 00000019 SystemOut     O [KRB_DBG_KDC] KrbKdcReq:WebContainer : 0: >>> KrbKdcRep Part: 0000: 6b 82 06 5e 30 82 06 5a  a0 03 02 01 05 a1 03 02  k...0..Z........

Between the send and receive operations, I'm losing around 210ms and that's a big part of the difference I'm experiencing between using DFS without kerberos and DFS with kerberos


[main] INFO  com.pwc.lu.business.plugin.ecm.dynamic.tools.kerberos.ServiceTicketGenerator - 00h 01min 21sec : 3364.0 object found - response time : 96ms. (current), 74ms. (average)


[main] INFO  com.pwc.lu.business.plugin.ecm.dynamic.tools.kerberos.ServiceTicketGenerator - 00h 05min 18sec : 3366.0 object found - response time : 357ms. (current), 311ms. (average)

Maheshwari

Hi Jeremy,

Would you please share the instuctions to configure Kerberos SSO for DFS. I did follow all the given instructions in the deployment guide but its not working. Thanks for your help

Harish

Jeremy Saumen

Hi,

I'm sorry but it's been a few month since I didn't touch Kerberos ; I

was put on other more urgent projects and Kerberos is still not used

today.

I'm unable to help you for the moment (because I forgot a lot of what I have

learnt) but I can provide few examples I have made to test Kerberos.

You can also check this website, i found it very helpfull to obtain my

first kerberos ticket (especially the part dedicated to the windows

registry modifications) ; once you have the ticket, it's nearly won :

http://www.javaactivedirectory.com/

I hope it will help you

liaoch

Hi - Have you identified the root cause? Was UDP or TCP used? (It seems that TCP was used but just in case ...). Was the DFS remote to KDC?

Thanks!

Jeremy Saumen

Sorry, I never found the issue. We are thinking to come back to kerberos in a few month, may be then i'll be able to find more.

About your question if DFS was remote to KDC, the answer is yes they are on a different server but if your question was about the distance between the servers, no, they are in the same building and on the same vlan.

liaoch

The described behavior looks to be caused by Nagle's algorithm. You might want to try to disable it (TCP_NODELAY) to see if it will resolve the issue.

For technical detail, refer to http://support.microsoft.com/kb/214397

Jeremy Saumen

Thanks, i'll keep in mind to try that