Multiple concurrent calls to the same service returns failures half the time

RWLXOM

= Environment =====

• OTCS 16.2.8
• 8 Threads on the server in question

= Question =====

If we call an API service multiple times concurrently, we get an error stating {"error":"Authentication Required"} on about half of them. Is there a reason why this would happen and how should we go about troubleshooting it?

= Problem Background and Details =====

We have an external process that starts with a search query to OTCS, it returns the node name and ID. Since the search results count is about 40K, we run the search multiple times iterating through the pages.

http://{MyServer}/OTCS/cs.exe/api/v2/nodes/{folderID}/nodes/?fields=properties{id,name}&limit=100&page=1&where_type=144

If we make one call at a time, it works without an issue. If we make 10 concurrent calls (page=1...page=10) then about half of them fail. If we lower the number to 8 or 2 concurrent calls, we still get a failure rate of about 1 in 2.

We're making these calls from Azure Data Factory and the returned error is:

Failure happened on 'Source' side. ErrorCode=RestSourceCallFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 401 indicates failure. Request URL: http://{MyServer}/OTCS/cs.exe/api/v2/nodes/{folderID}/nodes/?fields=properties%7Bid,name%7D&limit=100&page=1&where_type=144 Response payload:{"error":"Authentication Required"},Source=Microsoft.DataTransfer.ClientLibrary,'

The "Authentication Required" would make sense to me if the other service calls failed. But that is not the case.

We have the same issue on the next step when we iterate through the node IDs to import the documents. About 1/2 of all the calls fail when we call this service concurrently.

http://{MyServer}/OTCS/cs.exe/api/v2/nodes/{documentID}/content