16.6 Content Indexer install/configuration issues (separate server)

Hello all - we finally have our 16.6 install in a decent state, but are now running into issues with the "Content Indexer" (search) install/configuration on a separate server (as recommended by OpenText). Any help would be greatly appreciated

1) followed the steps in this KB article to install the JRE 1.8, Zookeeper, SOLR & NSMM.
https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB10027579

everything installed successfully, and the test command returns the below results

D\Interwoven\solr\bin>solr status

Found Solr process 1956 running on port 8983
INFO - 2019-07-22 15:48:04.062; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
{
"solr_home":"D:\Interwoven\solr\cloud\node1\solr",
"version":"7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:47:52",
"startTime":"2019-07-19T18:39:51.758Z",
"uptime":"3 days, 1 hours, 8 minutes, 13 seconds",
"memory":"100.7 MB (%20.5) of 490.7 MB",
"cloud":{
"ZooKeeper":"dwsrxxxxxx:2181",
"liveNodes":"2",
"collections":"1"}}

Found Solr process 3452 running on port 8984
INFO - 2019-07-22 15:48:05.662; org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing SSL Credential Provider chain: env;sysprop
{
"solr_home":"D:\Interwoven\solr\cloud\node2\solr",
"version":"7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:47:52",
"startTime":"2019-07-19T18:39:47.539Z",
"uptime":"3 days, 1 hours, 8 minutes, 19 seconds",
"memory":"62.3 MB (%12.7) of 490.7 MB",
"cloud":{
"ZooKeeper":"dwsrxxxxxx:2181",
"liveNodes":"2",
"collections":"1"}}

2) installed Content Indexer only (with the 2 other required components) on a separate server.

First issue, the KB10027579 says install JRE 1.8 and set that as the JAVA_HOME environment variable C\Program Files\Java\jre1.8.0_221. However, you CANT run the TS installer without the JDK 11.0.2 (with this set as the JAVA_HOME). otherwise, the installer throws an error stating: JAVA_HOME cannot point a a JRE. so we updated the JAVA_HOME to point to the JDK vs the JRE, and the installer succeeded successfully. do we need to switch the JAVA_HOME back to the JRE per the KB article? or leave it pointing at the "C\Program Files\Java\jdk-11.0.2"?

First error when trying to search in TS is this: "An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server."

I opened up firewall rules between the servers for full connectivity, but that didn't fix the above.

After the install, the first thing i noticed are errors in the \SERVERNAME\Interwoven\Indexer\log\iw.tsindex.log about a missing TeamSite collection.

[22 Jul 2019 11:41:19,747] INFO com.interwoven.hunter.index.IndexServiceSolrImpl (BulkIndexer-0) - Indexing 145 assets for branch \DWAPP02IWCM\default\main\EStudioGlobal
[22 Jul 2019 11:41:19,888] ERROR com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Couldn't complete indexing due to exception
org.apache.solr.common.SolrException: Collection not found: TeamSite
at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1139)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:822)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
at com.interwoven.solrutils.impl.SolrServiceImpl.add(SolrServiceImpl.java:465)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexAsset(IndexServiceSolrImpl.java:1784)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexAssets(IndexServiceSolrImpl.java:1654)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexEditionBased(IndexServiceSolrImpl.java:758)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexBranchInBulk(IndexServiceSolrImpl.java:174)
at com.interwoven.hunter.index.BulkJobProcessor.processNextJob(BulkJobProcessor.java:91)
at com.interwoven.hunter.index.Indexer.run(Indexer.java:158)
at java.base/java.lang.Thread.run(Thread.java:834)

I couldn't find anything in the OpenText documentation, so i tried creating the collection via a command i found on the SOLR site. This just led to more errors...

D\Interwoven\solr\bin>solr create -c TeamSite

[22 Jul 2019 15:40:45,624] ERROR com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Couldn't complete indexing due to exception
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://dwsrxxxxxx:8983/solr/TeamSite_shard1_replica_n1: undefined field: "EDITIONCRTIME"
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
at com.interwoven.solrutils.impl.SolrServiceImpl.add(SolrServiceImpl.java:465)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexAsset(IndexServiceSolrImpl.java:1784)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexAssets(IndexServiceSolrImpl.java:1654)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexEditionBased(IndexServiceSolrImpl.java:758)
at com.interwoven.hunter.index.IndexServiceSolrImpl.indexBranchInBulk(IndexServiceSolrImpl.java:174)
at com.interwoven.hunter.index.BulkJobProcessor.processNextJob(BulkJobProcessor.java:91)
at com.interwoven.hunter.index.Indexer.run(Indexer.java:158)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://dwsrxxxxxx:8983/solr/TeamSite_shard1_replica_n1: undefined field: "EDITIONCRTIME"
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more

Comments

  • I've set up 16.4 Solr/Index not 16.6 (yet)

    I also put Solr and the Indexer on the same (beefy) CPU.

    I know I will be trying to install/configure 16.6 Index (I have TS/Reports in a lab so far as well as LSCS) but I am a couple weeks away. Hope you are working with support.

  • Try these steps:

    Configure the Connector
    1. Login to TeamSite in your browser.
    2. Go to AdminConsole > Connectors.
    3. Select “SolrCloud” on the left hand navigation.
    4. Click on the “Edit” button to edit the configuration.
    5. Enter dwsrxxxxxx:2181 (or whatever is the actual value for your ZooKeeper Url) in the zookeeper URL, and click validate.
    6. Enter the desired shards and replicas or keep the defaults based on your business requirements.
    7. You should receive a message saying that your configuration is complete. Click “Close”.

    Then test the solr dashboard URL:
    http://<hostname/ip>:8983
    http://<hostname/ip>:8984

    Please let us know or open a new Support ticket here: https://support.opentext.com at any time.

  • Kenzer - thanks. Validation works. (see below) however, i don't see any options to configure shards/replicas? is that supposed to be in the GUI config? or in configuration files? I am a TS master. maybe because i tried to create the collection via the command line i posted above?

    ON EDIT:
    The SolrCloud connector has been successfully configured for TeamSite.
    The configsets have been successfully uploaded to ZooKeeper.
    Click Save to confirm these changes to the registry.

    ON SAVE:
    The SolrCloud connector has been successfully configured for TeamSite. The configsets have been successfully uploaded to ZooKeeper.

    Despite the above success messages, clicking the search button still throws this error > "An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server."

    do you have any suggestions? firewall rules are opened to allow both servers full access to each other (inbound and outband via IP)

    I am going to try cranking up the debugging here:
    \DWxxxxxxxxxxxx\Interwoven\ApplicationContainer\standalone\deployments\iw-cc.war\WEB-INF\conf\search\log4j.xml

    I am able to access the SOLR dashboard UI from the TS server, i.e. http://192.168.xx.xx:8983/solr/. However, it has an error on the homepage (see screencap):

    SolrCore Initialization Failures
    {{core}}: {{error}}

    Please check your logs for more information

    {{exception.msg}}

    Either way I will get a case going also - thanks Kenzer!

  • David Smith
    edited July 23, 2019 #5

    I was able to install and configure the Indexer on a separate server (Linux) for 16.4.1.1, and it was not easy. I hope that OpenText improved the product docs for installing all these applications, because it sucked for 16.4.x. Between the Apache docs, the KB articles, and the other info I had to get from Support to fill in all the blanks, I finally got it configured. Clearly your SOLR install is hosed. Are you certain you have it configured exactly how the KB article(s) tell you? The Connector is really important to set up properly as well. If you didn't get the shards/replicas prompt, something is wrong. I would go through the KB article that tells you how to delete everything and start over from scratch and try again. It's definitely not intuitive and the docs are awful. They really need a single document that puts it all together in order and very detailed with screen shots.

    Also - make sure you use the versions of Zookeeper and SOLR that are called out in the Release Notes or the Installation doc - I don't recall which one. I installed the latest versions the first time, and that was bad. You need to really follow the crumbs that are sprinkled throughout about 17 different docs to get it right. I am exaggerating, but not much.

  • Hey David! I hear you! Just want to let you know we heard and responded - in 16.6 (April 2019) we did do a single installation guide and have already gotten great feedback on that from customers! We also heard about challenges with Solr/Zookeeper and in 16.6.1 (October 2019) we are going to be offering to install/config Solr/Zookeeper for customers. Hope you're well!

  • @soolara said:
    Hey David! I hear you! Just want to let you know we heard and responded - in 16.6 (April 2019) we did do a single installation guide and have already gotten great feedback on that from customers! We also heard about challenges with Solr/Zookeeper and in 16.6.1 (October 2019) we are going to be offering to install/config Solr/Zookeeper for customers. Hope you're well!

    This is definitely good news and steps in the right direction. I wonder if "offering to install SOLR/Zookeeper" infers Professional Services would do it and a customer would have to pay extra for that. It really isn't hard to do the install - there had just been too many fragmented docs all over the place when 16.4.x came out, and many bits of critical information was hidden away in KB articles, which is never an obvious place (for me anyway) to go look for help on a new version. I appreciate that you're taking steps to improve documentation - this is been a thorn in my side for years and years. I hope it expands to all the other legacy docs as well (TS Admin Guide for instance), which have so many outdated and flat out wrong information, it's a bit ridiculous. Now that Google is a customer, I would think there would be a nice push to solidify the whole doc set.

  • In this case we are building the Solr / Zookeeper install/config into the installation process - so not professional services, no additional cost. And yes, we are hoping to improve the whole doc set overall as we go forward.

  • @soolara said:
    In this case we are building the Solr / Zookeeper install/config into the installation process - so not professional services, no additional cost. And yes, we are hoping to improve the whole doc set overall as we go forward.

    Even better - that's really good news. Thank you for the info!

  • i am trying the steps in the KB "To completely clear out the collection and start from scratch"

    These commands both error-ed (again with that ArchiveId field i mentioned previously)

    iwndxpurgebr \DWAPP02IWCM\default\main\EStudioGlobal
    iwndxpurgebr \DWAPP02IWCM\iwadmin\main\livesite\template

    D:\Interwoven\Indexer\bin>iwndxpurgebr \DWAPP02IWCM\default\main\EStudioGlobal
    Failure. Error: -1. Detail: Error from server at http://dwsrx02iwcm:8983/solr/TeamSite: Invalid Number: 640000000000000020200000 for field ArchiveId

    D:\Interwoven\Indexer\bin>iwndxpurgebr \DWAPP02IWCM\iwadmin\main\livesite\template
    Failure. Error: -1. Detail: Error from server at http://dwsrx02iwcm:8983/solr/TeamSite: Invalid Number: 650000000000000020200000 for field ArchiveId

    So I deleted the collection via the below command line

    solr delete -c TeamSite

    now, when i run the below it says i have nothing indexed.

    D:\Interwoven\Indexer\bin>iwndxlistbr
    Indexed branches:

    QUESTION: is the branches.cfg file i attached no longer used? does everything have to be setup via a command line now? i had those EStudioGlobal & livesite branches commented out in branches.cfg and restarted the services numerous times. we only want our branches indexed, i.e. /default/main/corporate_websites

    on step 3 (Delete the contents of the nodes: Clear out folders) - these folders don't exist, we only have a solr.xml file in both node folders (see attached screencap)

    \dwsrx02iwcm\Interwoven\solr\cloud\node1\solr\???
    \dwsrx02iwcm\Interwoven\solr\cloud\node2\solr\???

    I skipped this step and did the rest - everything complete successfully.

    iwndxaddbr /default/main/corporate_websites

    D:\Interwoven\Indexer\bin>iwndxaddbr /default/main/corporate_websites
    Success.

    However, the search functionality still isn't working and its throwing the same error (An error occurred communicating with the TeamSite Index Server: TeamSite Index Server might be down : java.lang.Exception: Could not open connection to TeamSite Index Server). is there something i am missing on the TS side? like i said, all the firewall ports are opened. i can connect to the solr GUI from the the TS server, but its still the same error there also

    SolrCore Initialization Failures
    {{core}}: {{error}}

    Please check your logs for more information

  • 1) all 4 services are up (zk, solr1, solr2, indexer)
    2) i cannot access the localhost:8983 on the search server itself which is weird (they have these servers really locked down)

    i googled a few things and it appears it might be an IE/javascript issue. i am going to talk to our sysadmins...
    https://stackoverflow.com/questions/56262704/solr-solrcore-initialization-failures-core-error

    does solr/zookeper/etc need internet access to external sites? as i mentioned these servers/firewall are locked down pretty tightly. we ran into an issue with the TS install was reaching out to the spring site and wouldnt install without us changing firewall rules (WCM-24695)

    the weird thing is i can access is using the IP address from the TeamSite server.

    http://192.168.xx.****:8983
    http://192.168.xx.****:8984

  • UPDATE: making some progress, i am now able to get the search window to come up, but anytime i search on some criteria i only get a blank page back. the content_center log has null pointer exception

    2019-07-25 15:51:14,781 [default task-12] WARN com.interwoven.ui.base.impl.command.CommandHandler () - Execute com.interwoven.ui.search.ccpro.SearchCommand error.
    java.lang.NullPointerException
    at deployment.iw-cc.war//com.interwoven.ui.search.AbstractSearchCommand.execute(AbstractSearchCommand.java:438)
    at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandHandler.doExecuteCommand(CommandHandler.java:1041)
    at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandHandler.tryRunCommandDescriptor(CommandHandler.java:864)
    at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandHandler.tryRunCommandID(CommandHandler.java:738)
    at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandHandler.runCommandLoop(CommandHandler.java:330)
    at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandServlet.doGet(CommandServlet.java:164)
    at deployment.iw-cc.war//com.interwoven.ui.base.impl.command.CommandServlet.doPost(CommandServlet.java:263)
    at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
    at javax.servlet.api@1.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
    at io.undertow.servlet@2.0.15.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
    at io.undertow.servlet@2.0.15.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
    at deployment.i****
    2019-07-25 15:51:14,782 [default task-12] WARN com.interwoven.ui.base.impl.command.CommandHandler () - runCommandLoop: command iw.search.ccpro.search returned null

  • @Eric Barker said:

    Either way I will get a case going also - thanks Kenzer!

    Please open a new Support ticket here if that is possible: https://support.opentext.com so Support can work with you to review items done and review any issues with current config.

  • As to your question about internet access, I can say that 16.4 does not require external access. We are using search heavily and also have pretty much all external access blocked. Now, there were 2 issues with JBoss in 16.6 that required external access to start the UI, so there may be an error.

  • Have you seen this KB article for clearing out the entire index: https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB11841768

    Once you delete or purge branches, you will have to add them back manually with the CLT. Then you need to wait until the branch(es) are indexed fully before you can search on them.

    There is nothing trivial about moving to 16.4 and trying to get all the components of Search working. I had to open several support tickets before I had everything working properly. Don't be afraid to open tickets. And if you think you aren't getting good help or the person handling your ticket doesn't seem to understand the Indexer either - ask for your ticket to be reassigned to someone who knows it. We don't pay all this money for poor support engineers.

  • Dave, yes he has that KB and we're working on a support case currently. Thanks!

  • Dave, we are making some progress - having a weird issue now where filename searches return a blank screen/null pointer exception. text / string searches work great. I am going to post the final details once we have everything fixed.

    we did run into an issue where the indexer service wouldnt stay up. somehow the branches.db got corrupted - the fix was to delete branches.db and restart and service

    [01 Aug 2019 14:11:10,622] DEBUG com.interwoven.hunter.index.Listener (main) - Starting subscription to: Interwoven
    [01 Aug 2019 14:11:10,622] DEBUG com.interwoven.hunter.index.BranchMapping (main) - Error while calling BranchMapping.getInstance()
    java.io.EOFException
    at java.base/java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:3062)
    at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1561)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
    at com.interwoven.hunter.index.BranchMapping.loadDB(BranchMapping.java:931)
    at com.interwoven.hunter.index.BranchMapping.init(BranchMapping.java:104)
    at com.interwoven.hunter.index.BranchMapping.getInstance(BranchMapping.java:72)
    at com.interwoven.hunter.index.IndexManager.init(IndexManager.java:102)
    at com.interwoven.hunter.index.IndexManager.(IndexManager.java:85)
    at com.interwoven.hunter.index.IndexServer.start(IndexServer.java:384)
    at com.interwoven.hunter.index.IndexServer.main(IndexServer.java:239)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.rzo.yajsw.app.WrapperJVMMain.executeMain(WrapperJVMMain.java:60)
    at org.rzo.yajsw.app.WrapperJVMMain.main(WrapperJVMMain.java:43)
    [01 Aug 2019 14:11:10,638] FATAL com.interwoven.hunter.index.IndexManager (main) - Error in initialization of the Index Manager
    [01 Aug 2019 14:11:10,638] FATAL com.interwoven.hunter.index.IndexManager (main) -
    [01 Aug 2019 14:11:10,638] DEBUG com.interwoven.hunter.index.Reaper (Thread-1) - Reaper thread started
    [01 Aug 2019 14:11:10,638] INFO com.interwoven.hunter.index.Reaper (Thread-1) - Shutdown IndexServer
    [01 Aug 2019 14:11:10,638] DEBUG com.interwoven.hunter.index.Listener (Thread-1) - Ending subscription to:Interwoven

    \SERVERNAME\Interwoven\Indexer\etc\branches.db

  • Eric - I have seen something similar, and I looked back at my support cases and my issue was that I had not completely applied all the patches that were added on top of 16.4.1 and 16.4.1.1 specifically for Indexer and Search. I assume all of the patches that went on top of 16.4.x were already built into 16.6, but you never know. I am certain that the Support folks can get to the bottom of it by looking at all your logs. It's hard to say if there's any overlap in issues since 16.6 came out pretty near when I was getting some patches made for me. Plus you're on Windows and I'm on Linux and I don't know if there are any quirks between those systems. I just know that all the moving parts have to be set up in the proper order and completely before everything works together. Good luck!

  • The blank screen issue is resolved, there was an issue with one of the CC Search debuggers being turned on. We've turned it off and no longer have issues with Search.

  • Do you mean a log debug setting? How in the world does that impact the code?

  • Dave, it shouldn't, so we've opened an issue for engineering to fix it. It would also happen on your 16411 instance. Specifically it's com.interwoven.ui.search.AbstractSearchCommand

  • Very strange - and good to know. I am pretty sure I had turned that one on at some point but I don't recall that it caused any problems. I can't be certain, though.

  • Dave - agreed - really bizarre how a logger setting would only break certain parts the search UI...We had all the logging cranked up and there were no exceptions of note besides the null pointer exception shown earlier.

    Either way i'm glad we were at least able to figure it out. There are so many kooky things and issues with this 16.6 version/install/setup process.

    The OpenText Bug ID for the debug issue is WCM-25621.

    Thanks all!

TeamSite Developer Resources

  • Docker Automation

  • LiveSite Content Services (LSCS) REST API

  • Single Page Application (SPA) Modules

  • TeamSite Add-ons

If you are interested in gaining full access to the content, you can register for a My Support account here.
image
OpenText CE Products
TeamSite
APIs