Can anyone state guidelines/ best practices while checking out/ in large number of documents

 Correct me if I am wrong but number of parallel threads would be driven by the fact how many IDfSession objects I could instantiate in one run of the application without reaching max number of allowed sessions

Number of threads wouldn't be driven by max session, it would be rather controlled by your program how many threads you'd want to launch, say 10/15/20 etc... However, it will depend on the max number of session setup at your content server/dfc client. You can't launch 500 threads, if your max concurrent session is set to 100, for example.

You would need to use one session per thread, alright. The max number of concurrent session is set at the server level (in server.ini) as well at your client dfc level (in dfc.properties), which is configurable. Again, the hardware resources i.e. cpu/memory/speed will have to be considered. If connection pooling is enabled, session creation and re-use of sessions will perform better -- to be obtained from IDfSessionManager, as opposed to non-managed session.

I agree that you need to try different scenarios to find the optimal settings.

Can you gives us a bit more background on your requirement as why you need to perform check-in/out operation on these documents? are you performing other operations too, besides library services? how many documents are we talking about? etc...

Message was edited by: Ahmad Sakhi

All comments

Are you checking out/in from the content server? Is your utility multi-threaded? How big are your DITA files?

Hi Johnny,

I am checking out/in from the content server using java application built on dfc api.

The application is not multi-threaded and the dita files range from 4KB to 1.07MB.

Thank you for taking the time to look into this discussion.

Files are not too large and since you are running on content server, its not a network bandwidth issue. My guess is the performance is file I/O issue at the OS level. Which OS are you using? I know in Linux you can change the number of open files the OS can support at one time - dont know how this would impact performance.

Hi Johnny,

Let me correct myself. I am running application on windows 7 client machine on the network and not on content server directly.

When I said I am checking out/in from content server I meant via a dfc based application and not wdk based application like Webtop.

My apologies for the confusion.

Try running the DFC application on the content server - this eliminates any network latency that can contribute to performance problems.

pettitk

We also once created a multi-threaded dfc app to do bulk importing. The same principal would hold for checkout/in. As Johnny stated, running anything like this on the Content Server avoids latency. -Kyle

Running the dfc application on the content server would definetly eliminate network latency but I would like to avoid running any other services on same machine as content server.

Are there any other suggestions or options I could look into.

I would like to learn more about the multi-threaded dfc app Kyle recomended. How can I find the optimal number of files to check out/in in a single thread. How many threads could I run simultaneously performing check out/in operation. Would this mean having n number of Idfsessions under use by n number of parallel threads for the application.

pettitk

There are lots of multi threading java examples out there… just google it, you will find a lot.

Hi Kyle,

I have read and written multi threaded applications before.

Let me rephrase my question:

Is there optimal number of files each thread should consider to check out/in rather than simply dividing the total number of files among available threads.

Thank you for taking the time to reply.

Is there optimal number of files  each thread should consider to check out/in rather than simply dividing the total number of files among available threads.

I think the question might be "how many threads are ideal to run in parallel" instead. The reason is because each thread will process n number of documents sequentially. So, at the end, it wouldn't matter how many documents you dedicate per thread.

Also, consider senchronizing your program, in such a way not to process one file multiple times, since with threads you tend to face race situation.

pettitk

I don’t think there is a concrete one size fits all answer… I think it depends (size of docs, resources, etc)… You will likely have to test the scenarios.

Hi Ahmad,

I think the question might be "how many threads are ideal to run in parallel" instead.

That would be my second question. Correct me if I am wrong but number of parallel threads would be driven by the fact how many IDfSession objects I could instantiate in one run of the application without reaching max number of allowed sessions.

I am still interested in finding the optimal number of documents I can perform checkin/ out in a single thread. For example if I had 5 documents and 5 threads it woudn't make sense to checkout each document on each thread. I guess to Kyle's point this is something I would have to figure out by experimenting under different scenarios.

Are there any other recommendations or settings I could change on client or code to reduce time for performing check out/in.

 Correct me if I am wrong but number of parallel threads would be driven by the fact how many IDfSession objects I could instantiate in one run of the application without reaching max number of allowed sessions

I agree that you need to try different scenarios to find the optimal settings.

Message was edited by: Ahmad Sakhi

One such service we have is project that works with 5600 documents. Every single document is a dita xml file and ranges from 4KB to 1.07MB in size. Our content server stores all xml files in xDB.

The service runs once every week where it checks out all documents. Updates the content via xsl transformations and checks in updated documents. This worked fine before as the number of documents increased, the service would take longer to checkout documents and even more longer to check it back in. The approximate duration of checkin is more than 2 hours.

I hope this information helps.

Have you looked at XML transformation services product? Its built on CTS architecture, which uses queue model to process documents. This way you can add more queue processors to handle ever increase document load.

Just an idea: If it's the number of documents that degrades the performance, you could run your program in two chunks, say 2800 documents per run?

That is what I have been doing as of now. I am running fewer documents at one run but wanted to know if there are other improvements I could make.

That is why running multiple threads with fewer documents seems to be right direction to go.

Hi Johnny,

My project is updating an existing document in the system. It checks out, makes changes to content, versions it by checking in document back into the system.

Unless I am missing something I was under the impression that this cannot be done via XTS. We do own license to XTS product and it is used to publish our content out of the system.

Can XTS change existing xml document within the system ?

I never used XTS before, but I'm guessing it should be able to generate a rendition post transformation. If that's the case, you may be able to create a TBO to override the doAddRendition() method and instead create a new version of the document.

Hi Johnny,

 I'm guessing it should be able to generate a rendition post transformation.  If that's the case, you may be able to create a TBO to override the doAddRendition() method and instead create a new version of the document.

What you have mentioned could work for a xml document that is not linked to another document in any way but in our case this seems extremely difficult since every xml document in system conrefs and references other xml documents and files. To get to these relations at TBO for rendition would be difficult and overkill for what I am trying to do. This is definetly a great suggestion but I might consider for other cases in our team.

For this discussion I am accepting multi-threaded option and consider running application on content server as an helpful answer.

Thank you all for your time, this was my first post ever and it was nice to get your valuable inputs.

I meant to say "...create a TBO to override the doAddRendition() method and create a new version of the document instead."

Basically, taking the xml generated from XTS and checking it as new version via TBO.

I understood what you meant earlier. Say for example the document conrefed another xml document within the system. The generated xml from xts would have conrefed content resolved into the xml itself at that point. Getting back to that information would require customization at transformation level that is to embed that information within the xml content. Theoretically it sounds possible but I would have to learn more about XTS and try it out to see if it works. I may give it a try later when needed. Thanks again.