Document upload performance using REST API

Ugis

Hello,

In one CS 16.2 isntallation we are using Streamserve 16 to archive documents to Content Server. Streamserve has out-of-the-box CS connector that is using REST API to upload documents (including meta-data) to CS.

With current configuration (1 CS instance for upload and 1 back-end just for indexing) we have achieved upload speed for 1 document around 0,45 seconds (14000 documents uploaded in 1h 45m). Average document has 1-2 pages and size range is between 10kb to 150kb, so in average it could be around 50kb or something like that.

Documents have meta-data, but again, nothing too complex - few attributes (average around 5 attributes) with short values.

When using REST API for upload, is there a chance to reduce the import time somehow? Some specific CS configuration? Adding more load-balanced CS instances? Anything else?

What is your experience when it comes to document mass import to Content Server vs speed?

Thanks,

Ugis

Nizar Ghazal

Quoted Ugis Godmanis on 10/12/2017 08:18 AM:

Hello,

In one CS 16.2 isntallation we are using Streamserve 16 to archive documents to Content Server. Streamserve has out-of-the-box CS connector that is using REST API to upload documents (including meta-data) to CS.

With current configuration (1 CS instance for upload and 1 back-end just for indexing) we have achieved upload speed for 1 document around 0,45 seconds (14000 documents uploaded in 1h 45m). Average document has 1-2 pages and size range is between 10kb to 150kb, so in average it could be around 50kb or something like that.

Documents have meta-data, but again, nothing too complex - few attributes (average around 5 attributes) with short values.

When using REST API for upload, is there a chance to reduce the import time somehow? Some specific CS configuration? Adding more load-balanced CS instances? Anything else?

What is your experience when it comes to document mass import to Content Server vs speed?

Thanks,

Ugis

Hi Ugis,

When load balancing, I believe you can setup more than one instance to act as the destination end-point for your upload code provided that you use the exact same cookie encryption key.

So, it becomes a fine balancing act for when designing any sort of mass ingestion, to have enough threads to service the multiple upload operations, but not too many that the database will begin to experience a level of lock escalation and subsequent blocking.

Were I in your position, I would measure if there are any improvements we can make in the following locations:

Networking

-Measure the time it takes for a full transmission of contents from origin to destination, is there any sort of congestion that would need to be worried about? Am I getting everything transmitted in a timely manner, am I seeing lots of packets being dropped, spurious retransmissions, etc? Wireshark it.

-Optionally, I suppose you could probably try tracing the request through the target web server, if you need to scale that up, too. I'm no IIS/Tomcat wizard, but I would imagine that they're very susceptible to memory pressure on the underlying OS.

Content Server

-Enable debug logging, take some aggregate timings to figure out where the most of the process for upload is being spent. If you have Performance Analyzer, it'll help you out with something like this.

Database

-Have your DBA gather/update statistics for all of the high-activity tables involved in your upload, ensure they're up-to-date and gathering at appropriate intervals based on the projected changes in volume of data. In an MS SQL world, you can probably setup an extended event to gather data on how much time the process is taking. If I recall, you should aim to have less than 12 ms per discrete operation. If it's getting to 15ms and above, you're having issues, with configuration, hardware, etc.

 

You could also inquire about a scaling engagement, with your account rep, to have someone externally do the above, and probably more I don't know enough to account for.

 

Hope that gives you some ideas.