16.6 Content Indexer install/configuration issues (separate server)

Hello all - we finally have our 16.6 install in a decent state, but are now running into issues with the "Content Indexer" (search) install/configuration on a separate server (as recommended by OpenText). Any help would be greatly appreciated

1) followed the steps in this KB article to install the JRE 1.8, Zookeeper, SOLR & NSMM.
https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB10027579

everything installed successfully, and the test command returns the below results

D\Interwoven\solr\bin>solr status

Found Solr process 1956 running on port 8983
INFO - 2019-07-22 15:48:04.062; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
{
"solr_home":"D:\Interwoven\solr\cloud\node1\solr",
"version":"7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:47:52",
"startTime":"2019-07-19T18:39:51.758Z",
"uptime":"3 days, 1 hours, 8 minutes, 13 seconds",
"memory":"100.7 MB (%20.5) of 490.7 MB",
"cloud":{
"ZooKeeper":"dwsrxxxxxx:2181",
"liveNodes":"2",
"collections":"1"}}

Found Solr process 3452 running on port 8984
INFO - 2019-07-22 15:48:05.662; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
{
"solr_home":"D:\Interwoven\solr\cloud\node2\solr",
"version":"7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:47:52",
"startTime":"2019-07-19T18:39:47.539Z",
"uptime":"3 days, 1 hours, 8 minutes, 19 seconds",
"memory":"62.3 MB (%12.7) of 490.7 MB",
"cloud":{
"ZooKeeper":"dwsrxxxxxx:2181",
"liveNodes":"2",
"collections":"1"}}

2) installed Content Indexer only (with the 2 other required components) on a separate server.

First issue, the KB10027579 says install JRE 1.8 and set that as the JAVA_HOME environment variable C\Program Files\Java\jre1.8.0_221. However, you CANT run the TS installer without the JDK 11.0.2 (with this set as the JAVA_HOME). otherwise, the installer throws an error stating: JAVA_HOME cannot point a a JRE. so we updated the JAVA_HOME to point to the JDK vs the JRE, and the installer succeeded successfully. do we need to switch the JAVA_HOME back to the JRE per the KB article? or leave it pointing at the "C\Program Files\Java\jdk-11.0.2"?

First error when trying to search in TS is this: "An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server."

I opened up firewall rules between the servers for full connectivity, but that didn't fix the above.

After the install, the first thing i noticed are errors in the \SERVERNAME\Interwoven\Indexer\log\iw.tsindex.log about a missing TeamSite collection.

[22 Jul 2019 11:41:19,747] INFO com.interwoven.hunter.index.IndexServiceSolrImpl (BulkIndexer-0) - Indexing 145 assets for branch \DWAPP02IWCM\default\main\EStudioGlobal
[22 Jul 2019 11:41:19,888] ERROR com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Couldn't complete indexing due to exception
org.apache.solr.common.SolrException: Collection not found: TeamSite
at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1139)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:822)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
at com.interwoven.solrutils.impl.SolrServiceImpl.add(SolrServiceImpl.java:465)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexAsset(IndexServiceSolrImpl.java:1784)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexAssets(IndexServiceSolrImpl.java:1654)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexEditionBased(IndexServiceSolrImpl.java:758)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexBranchInBulk(IndexServiceSolrImpl.java:174)
at com.interwoven.hunter.index.BulkJobProcessor.processNextJob(BulkJobProcessor.java:91)
at com.interwoven.hunter.index.Indexer.run(Indexer.java:158)
at java.base/java.lang.Thread.run(Thread.java:834)

I couldn't find anything in the OpenText documentation, so i tried creating the collection via a command i found on the SOLR site. This just led to more errors...

D\Interwoven\solr\bin>solr create -c TeamSite

[22 Jul 2019 15:40:45,624] ERROR com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Couldn't complete indexing due to exception
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://dwsrxxxxxx:8983/solr/TeamSite_shard1_replica_n1: undefined field: "EDITIONCRTIME"
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
at com.interwoven.solrutils.impl.SolrServiceImpl.add(SolrServiceImpl.java:465)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexAsset(IndexServiceSolrImpl.java:1784)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexAssets(IndexServiceSolrImpl.java:1654)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexEditionBased(IndexServiceSolrImpl.java:758)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexBranchInBulk(IndexServiceSolrImpl.java:174)
at com.interwoven.hunter.index.BulkJobProcessor.processNextJob(BulkJobProcessor.java:91)
at com.interwoven.hunter.index.Indexer.run(Indexer.java:158)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://dwsrxxxxxx:8983/solr/TeamSite_shard1_replica_n1: undefined field: "EDITIONCRTIME"
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more

Find more posts tagged with

Comments

nipper

I've set up 16.4 Solr/Index not 16.6 (yet)

I also put Solr and the Indexer on the same (beefy) CPU.

I know I will be trying to install/configure 16.6 Index (I have TS/Reports in a lab so far as well as LSCS) but I am a couple weeks away. Hope you are working with support.

Kenzer

Try these steps:

Configure the Connector
1. Login to TeamSite in your browser.
2. Go to AdminConsole > Connectors.
3. Select “SolrCloud” on the left hand navigation.
4. Click on the “Edit” button to edit the configuration.
5. Enter dwsrxxxxxx:2181 (or whatever is the actual value for your ZooKeeper Url) in the zookeeper URL, and click validate.
6. Enter the desired shards and replicas or keep the defaults based on your business requirements.
7. You should receive a message saying that your configuration is complete. Click “Close”.

Then test the solr dashboard URL:
http://<hostname/ip>:8983
http://<hostname/ip>:8984

Please let us know or open a new Support ticket here: https://support.opentext.com at any time.

bmanilow13

Kenzer - thanks. Validation works. (see below) however, i don't see any options to configure shards/replicas? is that supposed to be in the GUI config? or in configuration files? I am a TS master. maybe because i tried to create the collection via the command line i posted above?

ON EDIT:
The SolrCloud connector has been successfully configured for TeamSite.
The configsets have been successfully uploaded to ZooKeeper.
Click Save to confirm these changes to the registry.

ON SAVE:
The SolrCloud connector has been successfully configured for TeamSite. The configsets have been successfully uploaded to ZooKeeper.

Despite the above success messages, clicking the search button still throws this error > "An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server."

do you have any suggestions? firewall rules are opened to allow both servers full access to each other (inbound and outband via IP)

I am going to try cranking up the debugging here:
\DWxxxxxxxxxxxx\Interwoven\ApplicationContainer\standalone\deployments\iw-cc.war\WEB-INF\conf\search\log4j.xml

I am able to access the SOLR dashboard UI from the TS server, i.e. http://192.168.xx.xx:8983/solr/. However, it has an error on the homepage (see screencap):

SolrCore Initialization Failures
{{core}}: {{error}}

Please check your logs for more information

Either way I will get a case going also - thanks Kenzer!

David Smith

I was able to install and configure the Indexer on a separate server (Linux) for 16.4.1.1, and it was not easy. I hope that OpenText improved the product docs for installing all these applications, because it sucked for 16.4.x. Between the Apache docs, the KB articles, and the other info I had to get from Support to fill in all the blanks, I finally got it configured. Clearly your SOLR install is hosed. Are you certain you have it configured exactly how the KB article(s) tell you? The Connector is really important to set up properly as well. If you didn't get the shards/replicas prompt, something is wrong. I would go through the KB article that tells you how to delete everything and start over from scratch and try again. It's definitely not intuitive and the docs are awful. They really need a single document that puts it all together in order and very detailed with screen shots.

Also - make sure you use the versions of Zookeeper and SOLR that are called out in the Release Notes or the Installation doc - I don't recall which one. I installed the latest versions the first time, and that was bad. You need to really follow the crumbs that are sprinkled throughout about 17 different docs to get it right. I am exaggerating, but not much.

soolara

Hey David! I hear you! Just want to let you know we heard and responded - in 16.6 (April 2019) we did do a single installation guide and have already gotten great feedback on that from customers! We also heard about challenges with Solr/Zookeeper and in 16.6.1 (October 2019) we are going to be offering to install/config Solr/Zookeeper for customers. Hope you're well!

David Smith

@soolara said:
Hey David! I hear you! Just want to let you know we heard and responded - in 16.6 (April 2019) we did do a single installation guide and have already gotten great feedback on that from customers! We also heard about challenges with Solr/Zookeeper and in 16.6.1 (October 2019) we are going to be offering to install/config Solr/Zookeeper for customers. Hope you're well!

This is definitely good news and steps in the right direction. I wonder if "offering to install SOLR/Zookeeper" infers Professional Services would do it and a customer would have to pay extra for that. It really isn't hard to do the install - there had just been too many fragmented docs all over the place when 16.4.x came out, and many bits of critical information was hidden away in KB articles, which is never an obvious place (for me anyway) to go look for help on a new version. I appreciate that you're taking steps to improve documentation - this is been a thorn in my side for years and years. I hope it expands to all the other legacy docs as well (TS Admin Guide for instance), which have so many outdated and flat out wrong information, it's a bit ridiculous. Now that Google is a customer, I would think there would be a nice push to solidify the whole doc set.

soolara

In this case we are building the Solr / Zookeeper install/config into the installation process - so not professional services, no additional cost. And yes, we are hoping to improve the whole doc set overall as we go forward.

David Smith

@soolara said:
In this case we are building the Solr / Zookeeper install/config into the installation process - so not professional services, no additional cost. And yes, we are hoping to improve the whole doc set overall as we go forward.

Even better - that's really good news. Thank you for the info!

bmanilow13

i am trying the steps in the KB "To completely clear out the collection and start from scratch"

These commands both error-ed (again with that ArchiveId field i mentioned previously)

iwndxpurgebr \DWAPP02IWCM\default\main\EStudioGlobal
iwndxpurgebr \DWAPP02IWCM\iwadmin\main\livesite\template

\Interwoven\Indexer\bin>iwndxpurgebr \DWAPP02IWCM\default\main\EStudioGlobal
Failure. Error: -1. Detail: Error from server at http://dwsrx02iwcm:8983/solr/TeamSite: Invalid Number: 640000000000000020200000 for field ArchiveId

\Interwoven\Indexer\bin>iwndxpurgebr \DWAPP02IWCM\iwadmin\main\livesite\template
Failure. Error: -1. Detail: Error from server at http://dwsrx02iwcm:8983/solr/TeamSite: Invalid Number: 650000000000000020200000 for field ArchiveId

So I deleted the collection via the below command line

solr delete -c TeamSite

now, when i run the below it says i have nothing indexed.

\Interwoven\Indexer\bin>iwndxlistbr
Indexed branches:

QUESTION: is the branches.cfg file i attached no longer used? does everything have to be setup via a command line now? i had those EStudioGlobal & livesite branches commented out in branches.cfg and restarted the services numerous times. we only want our branches indexed, i.e. /default/main/corporate_websites

on step 3 (Delete the contents of the nodes: Clear out folders) - these folders don't exist, we only have a solr.xml file in both node folders (see attached screencap)

\dwsrx02iwcm\Interwoven\solr\cloud\node1\solr\???
\dwsrx02iwcm\Interwoven\solr\cloud\node2\solr\???

I skipped this step and did the rest - everything complete successfully.

iwndxaddbr /default/main/corporate_websites

\Interwoven\Indexer\bin>iwndxaddbr /default/main/corporate_websites
Success.

However, the search functionality still isn't working and its throwing the same error (An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server). is there something i am missing on the TS side? like i said, all the firewall ports are opened. i can connect to the solr GUI from the the TS server, but its still the same error there also

SolrCore Initialization Failures
{{core}}: {{error}}

Please check your logs for more information

bmanilow13

1) all 4 services are up (zk, solr1, solr2, indexer)
2) i cannot access the localhost:8983 on the search server itself which is weird (they have these servers really locked down)

i googled a few things and it appears it might be an IE/javascript issue. i am going to talk to our sysadmins...
https://stackoverflow.com/questions/56262704/solr-solrcore-initialization-failures-core-error

does solr/zookeper/etc need internet access to external sites? as i mentioned these servers/firewall are locked down pretty tightly. we ran into an issue with the TS install was reaching out to the spring site and wouldnt install without us changing firewall rules (WCM-24695)

the weird thing is i can access is using the IP address from the TeamSite server.

http://192.168.xx.****:8983
http://192.168.xx.****:8984

bmanilow13

UPDATE: making some progress, i am now able to get the search window to come up, but anytime i search on some criteria i only get a blank page back. the content_center log has null pointer exception

2019-07-25 15:51:14,781 [default task-12] WARN com.interwoven.ui.base.impl.command.CommandHandler () - Execute com.interwoven.ui.search.ccpro.SearchCommand error.
java.lang.NullPointerException
at deployment.iw-cc.war//com.interwoven.ui.search.AbstractSearchCommand.execute(AbstractSearchCommand.java:438)
at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandHandler.doExecuteCommand(CommandHandler.java:1041)
at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandHandler.tryRunCommandDescriptor(CommandHandler.java:864)
at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandHandler.tryRunCommandID(CommandHandler.java:738)
at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandHandler.runCommandLoop(CommandHandler.java:330)
at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandServlet.doGet(CommandServlet.java:164)
at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandServlet.doPost(CommandServlet.java:263)
at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
at io.undertow.servlet@2.0.15.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet@2.0.15.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at deployment.i****
2019-07-25 15:51:14,782 [default task-12] WARN com.interwoven.ui.base.impl.command.CommandHandler () - runCommandLoop: command iw.search.ccpro.search returned null

Kenzer

@Eric Barker said:

Either way I will get a case going also - thanks Kenzer!

Please open a new Support ticket here if that is possible: https://support.opentext.com so Support can work with you to review items done and review any issues with current config.

nipper

As to your question about internet access, I can say that 16.4 does not require external access. We are using search heavily and also have pretty much all external access blocked. Now, there were 2 issues with JBoss in 16.6 that required external access to start the UI, so there may be an error.

David Smith

Have you seen this KB article for clearing out the entire index: https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB11841768

Once you delete or purge branches, you will have to add them back manually with the CLT. Then you need to wait until the branch(es) are indexed fully before you can search on them.

There is nothing trivial about moving to 16.4 and trying to get all the components of Search working. I had to open several support tickets before I had everything working properly. Don't be afraid to open tickets. And if you think you aren't getting good help or the person handling your ticket doesn't seem to understand the Indexer either - ask for your ticket to be reassigned to someone who knows it. We don't pay all this money for poor support engineers.

HeatherH

Dave, yes he has that KB and we're working on a support case currently. Thanks!

bmanilow13

Dave, we are making some progress - having a weird issue now where filename searches return a blank screen/null pointer exception. text / string searches work great. I am going to post the final details once we have everything fixed.

we did run into an issue where the indexer service wouldnt stay up. somehow the branches.db got corrupted - the fix was to delete branches.db and restart and service

[01 Aug 2019 14:11:10,622] DEBUG com.interwoven.hunter.index.Listener (main) - Starting subscription to: Interwoven
[01 Aug 2019 14:11:10,622] DEBUG com.interwoven.hunter.index.BranchMapping (main) - Error while calling BranchMapping.getInstance()
java.io.EOFException
at java.base/java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:3062)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1561)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at com.interwoven.hunter.index.BranchMapping.loadDB(BranchMapping.java:931)
at com.interwoven.hunter.index.BranchMapping.init(BranchMapping.java:104)
at com.interwoven.hunter.index.BranchMapping.getInstance(BranchMapping.java:72)
at com.interwoven.hunter.index.IndexManager.init(IndexManager.java:102)
at com.interwoven.hunter.index.IndexManager.(IndexManager.java:85)
at com.interwoven.hunter.index.IndexServer.start(IndexServer.java:384)
at com.interwoven.hunter.index.IndexServer.main(IndexServer.java:239)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.rzo.yajsw.app.WrapperJVMMain.executeMain(WrapperJVMMain.java:60)
at org.rzo.yajsw.app.WrapperJVMMain.main(WrapperJVMMain.java:43)
[01 Aug 2019 14:11:10,638] FATAL com.interwoven.hunter.index.IndexManager (main) - Error in initialization of the Index Manager
[01 Aug 2019 14:11:10,638] FATAL com.interwoven.hunter.index.IndexManager (main) -
[01 Aug 2019 14:11:10,638] DEBUG com.interwoven.hunter.index.Reaper (Thread-1) - Reaper thread started
[01 Aug 2019 14:11:10,638] INFO com.interwoven.hunter.index.Reaper (Thread-1) - Shutdown IndexServer
[01 Aug 2019 14:11:10,638] DEBUG com.interwoven.hunter.index.Listener (Thread-1) - Ending subscription to:Interwoven

\SERVERNAME\Interwoven\Indexer\etc\branches.db

David Smith

Eric - I have seen something similar, and I looked back at my support cases and my issue was that I had not completely applied all the patches that were added on top of 16.4.1 and 16.4.1.1 specifically for Indexer and Search. I assume all of the patches that went on top of 16.4.x were already built into 16.6, but you never know. I am certain that the Support folks can get to the bottom of it by looking at all your logs. It's hard to say if there's any overlap in issues since 16.6 came out pretty near when I was getting some patches made for me. Plus you're on Windows and I'm on Linux and I don't know if there are any quirks between those systems. I just know that all the moving parts have to be set up in the proper order and completely before everything works together. Good luck!

HeatherH

The blank screen issue is resolved, there was an issue with one of the CC Search debuggers being turned on. We've turned it off and no longer have issues with Search.

David Smith

Do you mean a log debug setting? How in the world does that impact the code?

HeatherH

Dave, it shouldn't, so we've opened an issue for engineering to fix it. It would also happen on your 16411 instance. Specifically it's com.interwoven.ui.search.AbstractSearchCommand

David Smith

Very strange - and good to know. I am pretty sure I had turned that one on at some point but I don't recall that it caused any problems. I can't be certain, though.

bmanilow13

Dave - agreed - really bizarre how a logger setting would only break certain parts the search UI...We had all the logging cranked up and there were no exceptions of note besides the null pointer exception shown earlier.

Either way i'm glad we were at least able to figure it out. There are so many kooky things and issues with this 16.6 version/install/setup process.

The OpenText Bug ID for the debug issue is WCM-25621.

Thanks all!