When Archiving documents into OTCS, getting OTAC error- couldn't be distributed to node

Vijaya Sankari

We are using OTCS 23.4 and OTAC 23.4. We are having 3 front ends and 1 back end. We are also having 3 OTAC servers on cluster.
When we are archiving documents using rest API, sometimes we are getting the below error. It navigates between all 3 OTAC servers
02/17/2025 09:19:06 ERROR EASP EnterpriseArchiveStorageProvider.os:ProviderFilePut(73) ProviderFilePut of documentname.pdf failed: HTTP error (Internal Server Error [500]): Internal error in archive server: (ds-error='3030: The content for docId 'xxxxxx' with filename 'DATA.;1' could not be distributed to the following node(s): OTAC2 hostname.emea.dc.corpintra.net (no response: Read timed out) - couldn't write to file') ('') URL='https://otac lb:443/archive?create&pVersion=0045&contRep=XXXArchive&compId=data&ixCheckSum=MC6hBAICE4gCAQIEFGrHKZXKe0NTIRiskCBw6H7EOoTTBgUrDgMCGhMGbGliZHNo&ixUser=OTCSFE2&ixAppl=Archive%20Storage%20Provider&accessMode=c&authId=SP_OTCSFE2&expiration=20250217091806&secKey=MIIBXwYJKoZIhvcNAQcCoIIBUDCCAUwCAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHATGCASswggEnAgEBMCYwGjEYMBYGA1UEAwwPU1Bfc2V3bTBwMDAyMzA1AghmPg5cYkKLizAJBgUrDgMCGgUAoF0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMjUwMjE3MDgxODA2WjAjBgkqhkiG9w0BCQQxFgQU1gMJLPdUhrvB+o3VsSHGCRQKRCIwDQYJKoZIhvcNAQEBBQAEgYAgSg9TzdR4GBa6eJKSYEVSuX96Bk2b67tlO8dP+%2ft6PjoDJ5I2y9ej8PsbhZvUTkCu7MASd%2ff9DIy%2fL78KxBAEuD3q1b%2fdusDK%2fS28IDnssEmz3nWAZk0A8j9ef2c36sfzILuZlJGUUgE2OF9a3b56VFj4mupgIoqMzIcVI0XdqA%3d%3d' (Error creating document in Archive Server)

appuq

This cluster node

OTAC2 hostname.emea.dc.corpintra.net (no response: Read timed out) - couldn't write to file') ('') URL='https://otac lb:443/archive?

wasn't responding why is it a VM cluster or a K8S cluster. AFAIK OTAC needs a 3-node hazelclast cluster do you know if the architecture is based on hazelcast…

in a properly set up OTAC the files should go to a buffer and the buffer's write job needs to get it into an archive , perhaps the OTAC is set up without a buffer is sit even possible?

appuq

This would be difficult to solve in this forum.If I were you I would engage OT support

Vijaya Sankari

Hi Appu,

Thanks for the suggestion. We have already involved OT team, they are yet to analyse the logs. but this is happening in production and customers are facing this issue frequently, so wanted to check to see if there is any way to resolve the issue.

If you have seen this issue anytime previously, can you please let us know what might be causing the issue and how to resolve this, that would help us a lot.

Regards,
VijayaSankari.

appuq

This cluster node

OTAC2 hostname.emea.dc.corpintra.net (no response: Read timed out) - couldn't write to file') ('') URL='https://otac lb:443/archive?

wasn't responding why is it a VM cluster or a K8S cluster. AFAIK OTAC needs a 3-node hazelclast cluster do you know if the architecture is based on hazelcast…

in a properly set up OTAC the files should go to a buffer and the buffer's write job needs to get it into an archive , perhaps the OTAC is set up without a buffer is sit even possible?

David Henshaw

Vijaya - you're original message contains an reference to an error code - ds-error 3030.

Search that on the OpenText Support site and you'll find this

Support articles, alerts & useful tools - Archive Center - Unable to create new documents from front end system due to "No space left on device"

Check the as.logs on OTAC nodes - do they show anything specific?

Vijaya Sankari

Hi David...Space is present on device. In the above error it was mentioned no space is present but in our error it was not mentioned anything about space.

Vijaya Sankari

Hi Appu,

I will check which cluster is it...but we do have buffers configured and documents initial come to buffer and then they go to final storage iCASFS through poolwrite jobs

Regards,
Vijaya Sankari

Vijaya Sankari

Hi Appu,

We checked and found that it is hazelcast cluster. and in the logs it is saying that it couldn't be distributed to node of OTAC.

Regards,
Vijaya Sankari.

Vijaya Sankari

Getting it like this:

IXOS5Repository::put IXOS5Repository.cpp-784 Got error 11012: HTTP error (Internal Server Error [500]): Internal error in archive server (ds-error='3030: The content for docId 'aaadcvoidvvgpmtaw3aacervrcb1i' with filename 'DATA.;1' could not be distributed to the following node(s):

appuq

in the end, what was the problem as @David Henshaw pointed out a inodes(https://askubuntu.com/questions/231585/running-out-of-inodes) problem or was it something to do with hazelcast?

OT probably runs K8S PODS so they are Linux OS so it could be inode related.

Vijaya Sankari

Hi Appu,

The issue didn't get resolved. We are still getting the errors
We are not able to find what is causing the issue. But why it's happening is 'since 2 copies of documents needs to be created when receiving node is trying to contact other node we are getting distributed node error. We are not sure what is causing this issue or how to resolve it.'

Regards,
Vijaya

Vijaya Sankari

Getting this error now
UTC+0100 ERROR [http-nio-8080-exec-2315] com.opentext.ecm.lea.protocol.al.ALOutput -- logException: Caught exception of type com.opentext.ecm.lea.kernel.bridge.ICSJDSException: (ICS:0110) The content repository returned an error: code=3030, message=couldn't write to file, description=Error during write of ATTRIB.ATR (reason: ',java.io.IOException: Can't distribute ATTRIB.ATR with path E:\Apps\OpenText\Buffer\BPVol\01\89\31\008801E8\ATTRIB.ATR to volume 135')

appuq

At some point, OT will find it :) If you don't have an escalation ticket with OT perhaps time to do that . That just means they can get the finest R&D of OTCS.OTAC and SQL Gurus together and work on a solution.I have not worked on the new hazelcast implementation earlier it used to be MSCS or whatever cluster technology HP-UX supported . I see a windows path so that may rule out your cluster being on *Nix at this point so Inodes may not be the issue.is Volume 135 locked or full or anything of that sort?

Vijaya Sankari

Hi Appu,

volume is not locked or full as of now. Yeah Hope the OT will solve this as soon as possible.
Thank you

Regards,
Vijaya Sankari.