Teamsite search: unknown error communicating with TeamSite Index Server

joaopeixoto

Hi,
I just installed Teamsite 16.4. I installed everything on the same environment.
My indexer is working when I submit/deploy stuff.
I can see the things on solr and when I access:
host:8080/lscs/v1/document$?q=* (lscs auth)
host:1876/lscs/v1/document$?q=* (lscs runtime)
I can see the query on the solr log.
The problem is when I open the search within the contentcenter professional, it opens the window: host/iw-cc/command/iw.search.ccpro.advanced_search
When I execute the search I got the error:
"An unknown error occurred communicating with the TeamSite Index Server"
On the logs I have:

Teamsite/local/logs/iwui/search.log

2018-09-13 14:40:36,052 [default task-2] ERROR com.interwoven.ui.search.AbstractSearchCommand () - Error while searching the documents
com.interwoven.wcm.lscs.LSCSException: Internal server error:Exceeded max retries for the task
    at com.interwoven.wcm.lscs.impl.JSONExtractionUtils.extractError(JSONExtractionUtils.java:242)
    at com.interwoven.wcm.lscs.impl.Context.fetchURIAsStream(Context.java:244)
    at com.interwoven.wcm.lscs.impl.JSONExtractionUtils.extractDocumentFacets(JSONExtractionUtils.java:149)
    at com.interwoven.wcm.lscs.impl.DocumentFacetIterator.fetchDocuments(DocumentFacetIterator.java:159)
    at com.interwoven.wcm.lscs.impl.DocumentFacetIterator.<init>(DocumentFacetIterator.java:122)
    at com.interwoven.wcm.lscs.impl.ClientImpl.getDocuments(ClientImpl.java:310)
    at com.interwoven.ui.search.AbstractSearchCommand.execute(AbstractSearchCommand.java:540)
    ...

LiveSiteCSRT/log/lscs.log

09-13-18 @ 14:40:36 [ERROR] MainController - error.internal_server
com.interwoven.wcm.iwrr.util.RetryException: Exceeded max retries for the task
    at com.interwoven.wcm.iwrr.util.RetryHandler.retry(RetryHandler.java:118)
    at com.interwoven.wcm.iwrr.util.RetryHandler.run(RetryHandler.java:78)
    at com.interwoven.wcm.iwrr.dao.solr.SolrSearchDaoRuntimeImpl.documentSearch(SolrSearchDaoRuntimeImpl.java:235)
    at com.interwoven.wcm.lscs.service.search.SearchServiceRuntimeImpl.documentSearch(SearchServiceRuntimeImpl.java:120)
    at com.interwoven.wcm.iwrr.request.handler.DocumentRequestHandler.handleGet(DocumentRequestHandler.java:181)
    ...
Caused by: java.lang.NullPointerException
    at com.interwoven.wcm.iwrr.dao.solr.SolrSearchDaoRuntimeImpl$1.task(SolrSearchDaoRuntimeImpl.java:212)
    at com.interwoven.wcm.iwrr.dao.solr.SolrSearchDaoRuntimeImpl$1.task(SolrSearchDaoRuntimeImpl.java:160)
    at com.interwoven.wcm.iwrr.util.RetryHandler.retry(RetryHandler.java:96)
    ... 79 more

Looking at this last exception and the code, it looks like a context problem?!?

Context cxt = SolrSearchDaoRuntimeImpl.this.mAssetDao.getContextFromPath(context);
searchDb = cxt.getSearchDbName();

Taken from rtserver.jar

I think it's strange that I have logging on the lscsRT and not on lscsAuth because, according to the administration guide, "Content Center Professional and EStudio then query LSCSAuth..."
I followed the install guides but probably there's something in conflict because I installed everything on the same environment. I did this because the install guide says it's possible and this is like a testing environment within our company.
Thanks for your help.

nipper

Have you installed the Indexer ? It is a separate install and needs to be configured to talk to SOLR

joaopeixoto

Yes, it is installed. Looking at the indexer.properties I think everything is ok.
When I access host:8983/solr I can query it and find content that I can see on the content center console.
On solr, I can also see content that I've deployed on different collections (with numbered names)
Accessing here: http://host:1876/lscs/v1/document$?q=*
I can see content from the first workarea that I deployed (I know that this is because of the defaultProject property)

Stopping and starting the indexer and looking at the /Interwoven/Indexer/log/indexer.log I don't see any errors.
I see that it checks my branches to see if they need to be indexed.

I only have my zookeeper address, localhost:2181, on the lscs-conf.properties inside: /LiveSiteCSRT/runtime/webapps/lscs/WEB-INF/classes
Do I need it on someplace else?

Thank you!

nipper

you may want to try manually adding branches to the indexer, iwndxaddbr, and can check out the status with iwndxstat -a

I am fighting issues with the Indexer myself, but when it is running I do get data.

BTW, there is a patch on the support site 16.4.0.1 is available under patches, Indexer can be applied.

For Solr I used Solr cloud which runs on different ports. (9983 for ZK)

joaopeixoto

All my branches have the status: 11 - INDEXED
My zookeeper is running on port 2181. My solr (cloud) on port 8983.
From what I've seen so far, the problem is not with the indexer but the connection between the content center (or any other console) with the indexer.
Thanks.

nipper

Hmm, that is odd, when I installed solr cloud, it put ZK on the solr cloud port+1000 (ended up at 9983).

You may need to ask support, I am not certain if LSCS/AT and LSCS/RT are supported on the same server/same solr.

Kaps_iwov

Hi Andy,

I finished TS and Indexer 16.4 installation. Both are on separate servers.
On TS box when clicking on Search button: getting below error:
An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server.

On Indexer box I see indexer is up and running but when I try to check the status using iwndxstatus -a; it looks like it is doing something but no result comes up.

Also I want to know where in TS server; Indexer server detail sits? Which file is that?

Thanks,
Kaps.

nipper

AFAIK, it is not in a file, but rather is it connected through the IR Tables in Application services.

When you install TeamSite (as well as Indexer), part of the installation is a DB connection which creates the tables for the various apps to talk to each other.

During the Index install you should have said use existing registry and point to the TS server.

On the Index server:

cat /opt/Interwoven/iwinstall/private/config/installer.properties

You should see (among other things)

IRHOST=TEAMISTE-SERVER
IWOVREGISTRY_STRATEGY=SPECIFY_EXISTING
INSTALLTYPE=INSTALL
IRPORT=8080

nipper

Also, my grep has finally completed and it is in this file as well:

./TeamSite/cssdk/cssdk.cfg

joaopeixoto

As Andy said, check the database on the Common Services schema, the table IR_SERVICEINFO
If something is wrong, you can update this with the script:
/CommonServicesClient/bin/iwirregister.sh

More info at the KB: KB5006899

Kaps_iwov

Hi,

I checked IR_SERVICEINFO table and entry for Search host is present.
On the Indexer server - indexer.properties has correct values of TS host & TeamSite JMS details. Both JMS port 3035 and Indexer port 6715 are open between the servers.

In cssdk.cfg search entries were commented but to test this I further made entries of my search server though I think it is not needed but no difference in the issue even after making the entries.

Getting below error in search.log in TS server
2018-10-03 05:10:22,547 [default task-62] ERROR com.interwoven.ui.search.ccpro.AdvancedSearchCommand () - An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server.

On the Indexer server I see below error in iw.tsindex.log though TS server is up and running..

[03 Oct 2018 07:26:05,992] FATAL com.interwoven.hunter.index.IndexServer (main) - Index server is unable to connect the TeamSite server
[03 Oct 2018 07:26:05,992] FATAL com.interwoven.hunter.index.IndexServer (main) - com.interwoven.hunter.common.ServiceUnavailableException: TeamSite server is not up

Also I noticed that indexer service is getting stopped by itself. And when I try to stop manually it is not getting stopped gracefully rather I have to kill it.
Another thing I noticed is Indexer clts are not returning any result like ./iwndxstatus -a; looks like as if it is doing some processing..
In between I see SolrCloud and zookeeper which I installed on Search server is up and running..

Not sure where is the issue.. Any help is appreciated.

Thanks.

nipper

There is an open issue when the indexer tries to index a binary file. It will stop working (still running but doesn't move forward).

In our case files were imported without an extension, so it would hit an image and stop.

What you should see happen is files get copied to /opt/Interwoven/Indexer/temp/content/content* and get indexed. Then they are deleted. You can look at those files, are there binary files there ? You can add the extensions to /opt/Interwoven/Indexer/etc/indexer.properties

iw.index.binaryextensions=exe,.......

ALso you can change

iw.bulkindex.failRetryCount= from 0, which will quit a branch on an error and up that.

Finally open a ticket with support, I've been working with support for a while to help clean this up.

Kaps_iwov

Hi Andy,

FYI this set up is on AWS cloud.
And I don't see anything under /opt/Interwoven/Indexer/temp/ folder. It is empty..

But when clicking on Search button in TeamSite I see error as:
"An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server. "

On TeamSite server; I see below error in search.log:

2018-10-05 02:33:08,750 [default task-47] ERROR com.interwoven.ui.search.ccpro.AdvancedSearchCommand () - An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server.

telnet works from TS to Indexer server:
[root@da2uvangapp01 iwui]# telnet da2uvangsql03.aws.cloud.epsilon 6715
Trying 10.226.192.107...
Connected to da2uvangsql03.aws.cloud.epsilon.
Escape character is '^]'.

On Indexer server:
[root@da2uvangsql03 bin]# netstat -ltnp | grep ':6715'
tcp 1 0 0.0.0.0:6715 0.0.0.0:* LISTEN 11080/java
[root@da2uvangsql03 bin]# ps -ef | grep index
iwts 11080 1 0 02:32 ? 00:00:01 /opt/Interwoven/Indexer/../FoundationLibraries/JDK/bin/java -server -Xms64m -Xmx512m -Diw.search.base.dir=/opt/Interwoven/Indexer -Djava.library.path=/opt/Interwoven/Indexer/lib -Dts.server.os=lin -jar /opt/Interwoven/Indexer/lib/hunter_index.jar
root 11578 10471 0 02:40 pts/0 00:00:00 grep --color=auto index

As I stated before indexer service is getting stopped by itself. And when I try to stop manually it is not getting stopped gracefully rather I have to kill it.
Another thing I noticed is Indexer clts are not returning any result like ./iwndxstatus -a; looks like as if it is doing some processing..

Search server entry is properly registered in common_services DB.
As a part of set up I did install zookeeper and SolrCLoud on Indexer host and both are up & running.
I have done SolrCloud connector configuration as well in the TeamSite end via AdminConsole.

Thanks.

nipper

Do you have a ticket open with support ? That really needs to be done.

Put TS JBOSS into debug mode. Just long enough to run a couple tests and get the connection string to the Index server. It seems like the index server is either on a different server or port (though 6715 is the standard) than TS is expecting.

nipper

Also search through your iwjboss log (on the TS server) for any references to the Index server hostname.