TeamSite 16.6 fresh install issues / bugs / KB articles - Master List

https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB12920156

Goo call, we used to have sticky posts for new releases.

Here is what I have:

Linux: Some services do not get started with reboot because startup scripts do not pick up the login (source /etc/profile) and don't have JAVA_HOME set:

Linux:

JBOSS will not completely start without internet access. WCM-24410

cat eventsubsystem.war.failed
{"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"eventsubsystem.war\".undertow-deployment" => "java.lang.RuntimeException: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 7 in XML document from ServletContext resource [/WEB-INF/applicationContext.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 7; columnNumber: 68; cvc-elt.1.a: Cannot find the declaration of element 'beans'.
Caused by: java.lang.RuntimeException: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 7 in XML document from ServletContext resource [/WEB-INF/applicationContext.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 7; columnNumber: 68; cvc-elt.1.a: Cannot find the declaration of element 'beans'.
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 7 in XML document from ServletContext resource [/WEB-INF/applicationContext.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 7; columnNumber: 68; cvc-elt.1.a: Cannot find the declaration of element 'beans'.
Caused by: org.xml.sax.SAXParseException; lineNumber: 7; columnNumber: 68; cvc-elt.1.a: Cannot find the declaration of element 'beans'."}}

cat iw-cc.war.failed
{"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"iw-cc.war\".undertow-deployment" => "javax.servlet.ServletException: Failed to initialize
Caused by: javax.servlet.ServletException: Failed to initialize
Caused by: java.lang.NullPointerException"}}

Resolution:

The issue is due to changes made in the new version of TS as reported in WCM-24410. This will be fixed in an upcoming patch. A patched activemq-all-5.15.5-nojasypt-nospring.jar was placed in both of the following directories to resolve the issue:
ApplicationContainer/standalone/deployments/iw-cc.war/WEB-INF/lib
ApplicationContainer/standalone/deployments/eventsubsystem.war/WEB-INF/lib

Also, not certain if this is a bug or what, but I saw OD, on a base server upgrade, configure odbase.xml for webServices when it was already configured. Support has not been able to replicate it, but net result was there were 2 entries in odbase.xml for webServices and the DB configuration and OD failed to come up.

Thanks Andy - i can no longer edit the original post to add your Linux notes unfortunately - another note - we added a ticket to hide/disable that awful experience studio bar that shows up in the templates now:

OTCS Ticket 4192040 : Experience Center Integration Section on DCRs

FROM SUPPORT:
Currently, there is no way to remove that Experience Studio Integration section without code change (with unknown consequences). However, you are not the first customer to make this request, and we have a feature request submitted to either remove that section or make it selectable in a future 16.6.x update..

I have just learned from Engineering that this issue will be addressed in 16.6.01 which should be available hopefully by October, 2019.

Another issue i forgot to mention...this might be Windows only.

Content Indexer Install
1) KB10027579 states you must install JRE 1.8 and set that as the JAVA_HOME environment variable
2) HOWEVER, when you go to install content indexer afterwards, the installer errors saying "JAVA_HOME cannot point to a JRE" and it cannot proceed
3) we had to install jdk-11.0.2 and point our JAVA_HOME environment variable to jdk-11.0.2 to get the installer to proceed.
4) have both values in the JAVA_HOME ev (jre and JDK) didnt work either
5) support said we were OK leaving the JAVA_HOME pointing at the jdk-11.0.2 (despite what KB10027579 states)

Eric, that KB (KB10027579) was written for 16.4, hence the JRE1.8 requirement. With 16.6+ JEK11 is now supported and 8 is no longer required. We can go back and edit that KB so it's more clear.

Another one popped up.

Linux, I ran:

sudo ./install_archive.ipl ****.spar /default/...

sudo did not pick up JAVA_HOME and you could see the errors. That isn't hte issue, since it was trying to rebuild the UI, it wiped out my iw-cc.war completely. I ended up restoring from snapshot instead of trying to fix. While there probably was another way to fix, that was easy for me.

Need to use sudo -E or add this to the sudoers file:

Defaults env_keep += "JAVA_HOME"

I reported it to support and hopefully ENG will put a check to see that $JAVA_HOME/bin/java exists before running any commands.

@TS Support said:
Eric, that KB (KB10027579) was written for 16.4, hence the JRE1.8 requirement. With 16.6+ JEK11 is now supported and 8 is no longer required. We can go back and edit that KB so it's more clear.

please and thank you - so we can just install the JDK 11.0.2, set JAVA_HOME to it, and use that for the solr/zookeeper and Content Indexer installs?

@Andy Knipp said:
Another one popped up.

Linux, I ran:

sudo ./install_archive.ipl ****.spar /default/...

sudo did not pick up JAVA_HOME and you could see the errors. That isn't hte issue, since it was trying to rebuild the UI, it wiped out my iw-cc.war completely. I ended up restoring from snapshot instead of trying to fix. While there probably was another way to fix, that was easy for me.

Need to use sudo -E or add this to the sudoers file:

Defaults env_keep += "JAVA_HOME"

I reported it to support and hopefully ENG will put a check to see that $JAVA_HOME/bin/java exists before running any commands.

This fix is scheduled for 16.6.1

Thanks Andy - we now have 2 environments up w/ 16.6 (content indexer on the separate server) which are relatively stable/working as expected.

A few other questions/issues:

1) what exactly does the "Simple Search" do? i don't think it works...we never get any results, but when we click back aka "refine previous search" and try again, we get the expected results back.

2) Our iweventproxy.log is getting crushed w/ the below ActiveMQ errors - can you see if yours is also? if not any ideas? thanks!

"\Interwoven\TeamSite\local\logs\iwui\iweventproxy.log"

2019-08-15 13:32:55,041 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:33:26,082 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:33:57,108 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:34:28,149 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:34:59,178 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:35:30,202 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:36:01,227 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:36:32,269 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:37:03,311 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:37:34,349 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:38:05,375 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:38:36,404 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect

Eric, Simple Search is broken and being fixed in upcoming service packs/releases. It's currently searching for only results with a locale (such as en_us) even if there isn't one set, causing no results to come back. bug ID for 16.6.x is WCM-23758 and it's aimed for 16.6.0.1.

Only a couple hundred thousand lines or so.... :-)

Every 30 seconds.

Thanks TS Support (heather if that's you) - another one to add to the list.

Andy, so you are getting the same thing or similar in the iweventproxy.log? I'm not sure if i should tinker w/ opening ports or if its another bug.

Yep, same thing, checked one of the older systems (16.4.1) and it has thousands as well. Cannot hurt to log a bug but I assume it is not a big issue. Likely no help in opening a port as it's localhost

Eric and Andy, we've found a temporary workaround to remove these errors (edit eventsubsystem.war's log4j.xml under ApplicationContainer\standalone\deployments and turn the FailoverTransport logging to off from error) and have logged a request with engineering to hopefully get this resolved in future releases as well.

Bug ID is WCM-26394 and it's tentatively scheduled for 16.6.0.1.

That appears to have worked - thanks - it looks like the code is referencing property files that doesn't exist. i checked our old 7.4 version and these files weren't there either. iwclienteventproxy.properties & iweventproxy.properties

2019-08-16 13:40:36,510 [TeamSiteLog] ERROR com.interwoven.bridge100.TFileListener () - Subscription state file 'D:\Interwoven\TeamSite\servletd\logs\iweventproxy.properties' not found.
java.io.FileNotFoundException: \Interwoven\TeamSite\servletd\logs\iweventproxy.properties (The system cannot find the path specified)

2019-08-16 13:40:36,541 [TeamSiteClientLog] ERROR com.interwoven.bridge100.TFileListener () - Subscription state file 'D:\Interwoven\TeamSite\servletd\logs\iwclienteventproxy.properties' not found.
java.io.FileNotFoundException: \Interwoven\TeamSite\servletd\logs\iwclienteventproxy.properties (The system cannot find the path specified)

I'd already logged an issue for that as well. If you create the directory and a blank/dummy file those will also go away.
Bug ID for this one is WCM-26403, also targeted for 16.6.0.1 (tentatively).

@TS Support said:
Eric, Simple Search is broken and being fixed in upcoming service packs/releases. It's currently searching for only results with a locale
(such as en_us) even if there isn't one set, causing no results to come back. bug ID for 16.6.x is WCM-23758 and it's aimed for 16.6.0.1.

Is there an order of magnitude schedule for 16.6.0.1 ? We really need search to work and I cannot start upgrading unless we are pretty confident it will work.

Andy, advanced search works. It's just "simple search" which is what happens when you type something into the vpath bar at the top and hit search which is not working properly without locales set.

Got it, thanks for clearing that up.

Speaking of advanced search...how exactly do you get the custom metadata search attribute updates to show up in the advanced search window now?

I have made updates to both the below files (see attached), rebooted both servers, but am still only seeing 1 of the 2 attributes (Content ID) - do i need to delete the branches.db? or clear and recreate the TeamSite collection and index via https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB11841768

TS SERVER
\Interwoven\TeamSite\local\config\lib\content_center\customer_src\etc\conf\customer\search_custom.xml

SEARCH SERVER
\Interwoven\Indexer\etc\FieldMapping.xml

thanks!

Are there any indexed files on the branch you're searching with "Project Number" set? If not, it won't show up in the drop down.
Oh, and if you didn't run make_toolkit for the search_custom change it won't be there.

David Smith

One thing in reading all this that jumps out at me is that a bunch of these things were broken in 16.4 also and there have been many many promises over the years to make the product documentation better. I was told somewhere along the line that the 16.6 docs would be MUCH better. Clearly they are not. When is any company that owns this product going to make the product documentation current and accurate? There are so many KB articles - you can't possibly expect your customers to go find them anytime something is missing or wrong in the product docs. Now that a very large entity has purchased this product, I'm pretty sure they won't put up with this lack of effort on OpenText's part. It's unacceptable that the docs are so out-of-date.

David Smith

Eric - great new "feature" of SOLR - if you have a custom EA that isn't set in any of the files in a branch you're indexing, that EA won't show up in the drop-down list in the Search UI. Brilliant, eh? Never mind if you have custom scripts that may be running background searches for that EA and will hit Null Pointer Errors if it hits against one of the branches that doesn't use it (yet). It is also a new undocumented "feature" that bit me in 16.4. I received a manual query that allows me to test whether a specific EA is set in a branch before querying for it, which gets me around the issue. But still, you won't find it written anywhere.

David - thanks - that makes sense now. we have project_number EA set for SOME files, but not all the files. so unless the EA is actually applied to every file in the folder/sub-folder it wont show up in the advanced search pick list?

even more concerning is search service keeps crapping out (Interwoven TeamSite Index Service). this server has 16gb memory...OT do we need to increase the heap space size on the search server?

[25 Aug 2019 00:55:13,803] WARN com.interwoven.hunter.index.Listener (ActiveMQ Connection Executor: tcp://servername/192.****.xx.xx:3035@63294) - Listener received an exception from the event system (see below) -- will attempt to reconnect.
javax.jms.JMSException: Unexpected error occurred: java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1952)
at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1971)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:345)
at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:224)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Unexpected error occurred: java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:222)
... 1 more
Caused by: java.lang.OutOfMemoryError: Java heap space

David Smith

Not exactly, Eric. If the EA exists on one or more of the files, you should see it in the drop-down. Of course, it all depends on which areas you tell the Indexer you want indexed. If the EA is only in files in the WA and you don't index the WA, then you're SOL. Same with Editions - if you configure not to Index all Editions, it may or may not be there. Regardless, if it exists in ANY of the files in a branch and is actually set with a value (that's important I think), then you should have it as an option in the drop-down. Only if none of the files that are indexed have a value set for that EA will it not be there.

As far as memory - we also use 16Gb for our search server, but I don't know how much data you have that's indexed. We don't have a ton of data. Is that error from your TS server or your Indexer server? It looks like ActiveMQ is having some trouble on your TS server. Maybe you need more memory on that server, or your ArctiveMQ DB has just gotten too big. That has bit us in the past and we clean out the /activemq folder from time-to-time. But I was told last week that the Indexer depends on ActiveMQ so we should stop clearing out that folder. I don't know where it draws its memory from or how it works at all, really. ActiveMQ has been a mystery to us.

Yea I have noticed that SOLR is a memory hog. We are at 16GB as well and think that is too low. Moving to 64 bit took away all constraints about memory and it seems that TS (more than just Solr) tends to just use that memory as fast as it can be added.

Eric, we do have a KB for the recommended memory settings for Solr/Indexer:
https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB11628358

Les Bleus

Any update on the 16.6.01 release date?

Their latest email says the below...hopefully all the issues noted above for both Windows and Linux are encapsulated?

What's new in TeamSite 16.6.1?
The latest version of TeamSite, scheduled for release in November 2019, includes enhancements to intelligent experiences and usability, new cloud integrations and improvements to security and file system drivers.

TeamSite 16.6.1 highlights
• Translation automation powered by Google Translate™
• Improved content suggestions and AI-driven tagging
• New integration with OpenText™ Core Experience Insights to track events for customer journey mapping
• Import cloud audiences from OpenText™ Optimost™ for improved content targeting
• Features that simplify TeamSite onboarding, including out-of-the-box components and low-code scripts