Latency + DFC = pain

After moving the Content Server to a different country it now takes me about 50 ms to reach it from my workstation. It seems like this might be a killer for a DFC application because of its chattiness. Does anyone know if there's some settings I could look at to improve the performance? It's kinda pain to develop with these performances. I'm running 6.7SP2 with patch 16 on both the server and client side.

For reference I've created a very simple console application to illustrate the difference of running with 50 ms latency (from my workstation) vs 1 ms latency (from another server in the same network). They both use identical libraries and dfc.properties.

This is the application:

This is the application


public class Main {


    public static void main(String[] args) {
        IDfClientX clientX = new DfClientX();
        System.out.println("Starting measurement. Please wait...");
        long timestamp = currentTimeInMs();
        try {
            clientX.getLocalClient();
        } catch (DfException e) {
            System.exit(1);
        }
        System.out.println("Runtime in millis " + (currentTimeInMs() - timestamp));
    }


    private static long currentTimeInMs() {
        return System.nanoTime() / 1000000;
    }
}

Running locally with 50 ms results in:

Runtime in millis 27846

While running on the application server with 1 ms results in:

Runtime in millis 4803

Other operations like getSession also takes forever some times (probably when there's no session in the session pool).

Find more posts tagged with

Note:

If there are Accepted Answers, those will be shown by default. You can switch to 'All Replies' by selecting the tab below.

Accepted answers

PanfilovAB

integers come from com.documentum.fc.client.impl.connection.docbase.netwise.NetwiseDocbaseRpcClientFactory#makeNativeConnection: 3 means there was no try to connect to docbase using IPV6, 2 means there was a try to connect to docbase using IPV6 but it got failed. DFC decides that it is able to use IPV6 if docbroker returns IPV6 address for content server and client's (host where DFC resides) hostname resolves into IPV6 address - for windows this means that host has IPV6 address. M.b. adding -Djava.net.preferIPv4Stack=true to java options will help

All comments

klvs2001

Can your application use Restful Services instead? DFC is a "chatty" protocol and will exhibit this type of behavior over a WAN.

Francois Dauberlieu

Have you looked into use Branche Office Cache Services (BOCS).

ape

It's a big task at hand if I were to rewrite the code to use the REST services, but still something I would consider. However it looks to me, correct me if I'm wrong, like they don't support aspects.

ape

Nope, but I will take a look at it.

DubrovskyD

What kind of end-user-app do you use?

ape

It's a html/js application with Documentum as a backend using DFC. Running the webapplication in the same physical environment as the Documentum server is working just fine, the problem is with developing and running unit tests locally.

lastnitescurry

get a local repository for dev?

DubrovskyD

I agree with . Also you can put some intermediate server near your environment to run your test on it and even to develop; it depends on whether RDP would be faster than DFC (I`ll not be surprised if it would really be so).

For me, it would be the easiest way.

If you have time and interest you can dig into DFC protocol for some hidden undocumented options (but I`m very uncertain about this) - had already done protocol parsing and you can look through his python client somewhere on GitHub.

ape

Aye, the problem is that the corp decided to move all our servers. Getting some servers up locally at my location seems to be a tough sell. At the moment I'm looking into two separate steps simultaneously. Creating docker images with necessary Documentum components for developers to run locally, and extracting code into our own REST services running on the Content Server. Just need to find a way to get approval;)

PanfilovAB

If getLocalClient() is the only problem call, I bet adding dfc.verify_registration=false into dfc.properties will help

ape

Thank you kindly. It's not the only issue but at least it cuts down the initial time taken.

PanfilovAB

You are doing something wrong, let's perform some basic calculations.

For 50ms network latency expected TCP throughput is 10Mbps (64 * 1024 * 8 / 0.05), this is good enough for downloading files, but what happens with DFC? Documentum session establishment and disconnecting with cold start of client (what happens during session establishment you can read in my blog: http://blog.documentum.pro/2014/12/21/session-management-beginning/) looks like:

TCP handshaking with docbroker
request servers for docbase
disconnecting from docbroker
TCP handshaking with docbase
new session RPC
ENTRY_POINTS RPC
AUTHENTICATE_USER RPC
GET_ERRORS RPC
fetching messages
disconnecting RPC
disconnecting from docbase

For my network latency (350-400ms) the following code:


public class Test {

    public static void main(String[] argv) throws Exception {
        long start = System.currentTimeMillis();
        IDfSession session = null;
        try {
            session = new DfClient().newSession("ssc_bee", new DfLoginInfo(
                    "test01", "test01"));
        } finally {
            if (session != null) {
                session.disconnect();
            }
        }
        System.out.println(System.currentTimeMillis() - start);
    }

}

prints 5000 - 5200, which corresponds with network interactions mentioned above. Your code:


public class Test {

    public static void main(String[] argv) throws Exception {
        long start = System.currentTimeMillis();
        IDfClientX clientX = new DfClientX();
        clientX.getLocalClient();
        System.out.println(System.currentTimeMillis() - start);
    }

}

prints 27504 for dfc.verify_registration=true and 1322 for dfc.verify_registration=false. dfc.verify_registration=true causes a lot of delays because DFC tries to check it's registration in global registry and if you have a dedicated GR it sucks to be you, because as far DFC starts fetching objects from docbase (checking client registration does fetch dm_client_registration and dm_public_key_certificate objects) it retrieves a lot of auxiliary information like information about certain types, information about available TBO and modules, etc. But your result for 50ms network latency is weird.

ape

There could be other consequences from the relocation of the server aswell. Looking into the dfc trace it seems to stuck for a while discovering Content Servers.

This trace is obtained running


session = new DfClient().newSession(repository, new DfLoginInfo(username, password));
session.disconnect();

When dfc.verify_registration is true:

1,948 N/A N/A [main] .DEBUG: Using explicit identity "dm_bof_registry" for docbase "n5d_utv"

2,258 N/A N/A [main] .DEBUG: Updated Server Choices for 'n5d_utv'. 1 servers found. 0 content servers found

23,322 N/A N/A [main] .DEBUG: 2:IPV6: true address: /172.28.20.159:47625

And when it's false:

1,777 0,000 N/A [main] com.documentum.fc.client.DfClient@6cbe68e9.<init>() ==> <void>

1,795 22,867 N/A [main] com.documentum.fc.client.DfClient@6cbe68e9.newSession("n5d_utv",DfLoginInfo{user=peder.lie, forceAuth=true}) ==> com.documentum.fc.client.IDfSession@7b2bf745

1,880 N/A N/A [main] .DEBUG: Using explicit identity "peder.lie" for docbase "n5d_utv"

2,178 N/A N/A [main] .DEBUG: Updated Server Choices for 'n5d_utv'. 1 servers found. 0 content servers found

23,243 N/A N/A [main] .DEBUG: 2:IPV6: true address: /172.28.20.159:47625

Running from one of the servers in the same data center with default dfc.verify_registration:

3.964 N/A N/A [main] .DEBUG: Using explicit identity "dm_bof_registry" for docbase "n5d_utv"

4.187 N/A N/A [main] .DEBUG: Updated Server Choices for 'n5d_utv'. 1 servers found. 0 content servers found

4.222 N/A N/A [main] .DEBUG: 3:IPV6: false address: /172.28.20.159:47625

I don't know what the int in the ipv6 debug statement is, but my issue might be with the network setup I guess.

PanfilovAB