REST Random 500 errors on auth

BrandonL

I have posted previously about similar issues with our usage of the REST API but I believe this may be different. We built a project in our development environment and it worked really well, but now that we've gone to our production OpenText environment, we are seeing completely random 500 errors on the auth call. Usually if we retry the call, it clears up. Each call we have done is using the same credentials and method of authenticating, yet roughly 10 - 20% (out of 1000s of calls) are failing intermittently.

One difference between OpenText DV and PD in our environment, is that PD has multiple front end servers, but we do NOT load balance. I have tried pointing to multiple front end servers and the behavior is the same, so I don't believe it is a problem isolated to one particular node.

The documentation indicates that the 500 error is triggered when credentials are not passed, but we are definitely passing them. We are hitting a virtual directory we set up called OTCSAPI and it only uses anonymous authentication. The top-level server setting only accepts anonymous as well.

Is there some type of throttling in Content Server that might be causing calls to fail when it is busy?

The anonymous user is the built-in IUSR account. Any ideas? The error we are getting is here - some of it is proprietary but it may help you:

Message: Method and/or Endpoint execution failed (Inner: 500 - 500 Internal Server Error); ServiceName: REST_otauth; ServiceGuid: 0cf4cde1-9414-44d6-8482-03a12fd1af97; InnerExceptionMessage: ;

We are passing in a connection string of username=usernamehere;password=passwordhere

BrandonL

Any ideas?

BrandonL

This support model is terrible. This is a production issue. Is there a place I can go to get actual help rather than posting on this forum and wait days for a response? Given OpenText's rather large size and customer base, this is an outrageous way to support companies trying to build on your platform.

Ferdinand Prantl

Did you look at the log files? Did you try debugging the error in the Builder/Oclipse?

As an OpenText customer, you can open a support case for the product you bought to help with the investigation of the problem. However, you cannot open a support case for a problem with the CS REST API today, AFAIK. You could contact OpenText and ask about the their strategy to support the CS REST API developers, except for running this forum.

BrandonL

I did look at the log files. I haven't found anything suspicious that would indicate a problem. I have done a wireshark trace and confirmed that good calls and bad calls both come to OpenText in the exact same way. I have reproduced the error with both a proprietary REST client and the Advanced REST Client extension for Chrome. Both send the exact same calls to the OpenText environment. I have confirmed that there should not be any IWA issues at play.

The only feedback I'm getting in the JSON is "Unexpected token: <".

I have opened a ticket with the log files provided. I am not an OpenText developer, which some additional instruction I would be willing to debug in Oclipse/Builder but I'm not familiar with those tools.

While this error is occurring during REST calls, it would seem it is not specifically a REST issue. 500 error is a server error. I don't think REST should have much to do with it, although I haven't been able to reproduce outside of rest.

BrandonL

I am continuing to troubleshoot this issue and will post updates here in parallel with my ticket as I don't know whose domain this will end up in.

I have been able to confirm that when this issue hits it occurs against all Content Server servers, and from multiple physical sources and accounts. It seems to be a Content Server farm issue rather than a specific web node.

With that in hand I decided to start looking at OTDS and found the following: it is failing to authentication my service user when these 500 errors occur. However, when I re-attempt the calls, (after a fewm inutes of not working), the calls start working again. I am seeing failures from both my proprietary client and from Advanced REST Client, so I don't think there is anything wrong with my calls (plus they work when I re-attempt them). Unless I have BOTH web servers somehow misconfigured, this must be something with OTDS.

Here is an example snippet of calls working, then failing, then working again:

015-05-12 15:26:49.058 INFO - ,2015/05/12 15:26:49 CDT,0,0,Authentication Service,Success Access,27,Initial authentication successful,serviceaccount@domain.com,,Authentication success: serviceaccount@domain.com from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:26:49.073 INFO - ,2015/05/12 15:26:49 CDT,0,0,Authentication Service,Success Access,56,Ticket Created,serviceaccount@domain.com,,User name [serviceaccount] in resource ID [071095c8-845b-40d1-abe7-dd670896a8db]
2015-05-12 15:27:47.886 WARN - ,2015/05/12 15:27:47 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:28:28.633 WARN - ,2015/05/12 15:28:28 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:28:42.315 WARN - ,2015/05/12 15:28:42 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:28:42.783 WARN - ,2015/05/12 15:28:42 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:28:47.213 WARN - ,2015/05/12 15:28:47 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:28:52.782 WARN - ,2015/05/12 15:28:52 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:29:26.884 WARN - ,2015/05/12 15:29:26 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:29:49.988 WARN - ,2015/05/12 15:29:49 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:30:02.015 WARN - ,2015/05/12 15:30:02 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:30:26.570 WARN - ,2015/05/12 15:30:26 CDT,0,0,Authentication Service,Failure Access,28,Initial authentication failed,domain\serviceaccount,,Authentication failure [INVALID_CREDENTIALS]: domain\serviceaccount from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:30:51.686 INFO - ,2015/05/12 15:30:51 CDT,0,0,Authentication Service,Success Access,27,Initial authentication successful,serviceaccount@domain.com,,Authentication success: serviceaccount@domain.com from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:30:51.686 INFO - ,2015/05/12 15:30:51 CDT,0,0,Authentication Service,Success Access,56,Ticket Created,serviceaccount@domain.com,,User name [serviceaccount] in resource ID [071095c8-845b-40d1-abe7-dd670896a8db]
2015-05-12 15:31:21.358 INFO - ,2015/05/12 15:31:21 CDT,0,0,Authentication Service,Success Access,27,Initial authentication successful,serviceaccount@domain.com,,Authentication success: serviceaccount@domain.com from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:31:21.373 INFO - ,2015/05/12 15:31:21 CDT,0,0,Authentication Service,Success Access,56,Ticket Created,serviceaccount@domain.com,,User name [serviceaccount] in resource ID [071095c8-845b-40d1-abe7-dd670896a8db]
2015-05-12 15:31:53.431 INFO - ,2015/05/12 15:31:53 CDT,0,0,Authentication Service,Success Access,27,Initial authentication successful,serviceaccount@domain.com,,Authentication success: serviceaccount@domain.com from host Unknown with address Unknown for resource 071095c8-845b-40d1-abe7-dd670896a8db
2015-05-12 15:31:53.431 INFO - ,2015/05/12 15:31:53 CDT,0,0,Authentication Service,Success Access,56,Ticket Created,serviceaccount@domain.com,,User name [serviceaccount] in resource ID [071095c8-845b-40d1-abe7-dd670896a8db]

Ferdinand Prantl

The “Unexpected token: <“ error most likely means that your web server blocks the error responses. Enable the error response body pass-through and you will get the actual error message from the response. If you use IIS, edit the web.config in the cgi directory and merge this into it:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <httpErrors errorMode="Detailed" existingResponse="PassThrough" />
  </system.webServer>
</configuration>

I saw a couple of posts that authenticating with a user imported from OTDS fails. I don't know why; OTDS integration should be transparent to the authentication request handler.

Dmitry_Popov

Hello *!

I receive such messages 2 times per minute in Tomkat logs (OTDS). I don't know why or how to disable this spam. Does anyone know how to turn this off?

localhost_access_log.2021-04-07.txt

10.56.21.241 - - [07/Apr/2021:11:13:19 +0400] "POST /otds-v2/services/authentication HTTP/1.1" 200 1520

10.56.21.241 - - [07/Apr/2021:11:13:19 +0400] "POST /otds-v2/services/authentication HTTP/1.1" 200 1285

directory-access.log:

2021-04-07 11:13:19.085 INFO   - ,2021/04/07 11:13:19 MSD,0,0,,Authentication Service,Success Access,27,Initial authentication successful,otadmin@otds.admin,,Authentication success: otadmin@otds.admin from host 0:0:0:0:0:0:0:1 with address 0:0:0:0:0:0:0:1 for resource 754c8f1d-440b-4e12-9dcb-686e0f6154db

2021-04-07 11:13:19.085 INFO   - ,2021/04/07 11:13:19 MSD,0,0,,Authentication Service,Success Access,56,Ticket Created,otadmin@otds.admin,,User name [Admin] in resource ID [754c8f1d-440b-4e12-9dcb-686e0f6154db]