TeamSite 16.6 fresh install issues / bugs / KB articles - Master List

bmanilow13

could we possibly get an admin to sticky a post with all the bugs/fixes/KB articles/etc? note: all of the below are on a Windows / Oracle environment.

TeamSite main server

1) "Typical (recommended)" installation type was greyed out/not available when running the installer. Was told by support this is expected and to use CUSTOM

2) there were constant null pointer errors accessing branches/folders/etc in CCPRO.
Installed mandatory patch WCM-23141. They were continuing to get null pointer errors after as well, so installed WCM-22944 as well to resolve.
Note: we had to re-run patch WCM-23141 after WCM-22944 as it undid the previous fix.

3) Reporting service was not staying up.
Had to open the spring framework URLs in our firewall (WCM-24695)

http://www.springframework.org/*
http://www.w3.org/*

4) hibernate validating all Oracle sequences causing errors/failures in commonservice.log
[default task-5] ERROR org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentImpl () - Could not fetch the SequenceInformation from the database
java.sql.SQLException: Numeric Overflow

the query that OpenText's hibernate is running is “select * from all_sequences” – so whatever sequences that the user has access to will be returned. It is doing a “getLong” on the min_value for all visible sequences. The suggestion is to restrict access to only the sequences owned by the db user.

5) wcm-service.war failed to deploy.
Resolved with KB11935187

6) UI menu item seems to be cut off (WCM-24999).
Provided workaround.

7) Perl Inline database calls (using DBI) throwing errors/failing in version 16.6

The 16.6 version removes perl modules needed for templates to connect to an Oracle DB. After some back and forth OT provided a copy of the ODBC Perl module and associated modules that were already compiled. The alternative was to download and compile it separately on the server (we couldn't get this to work at all despite much effort) NOTE: this isn't in any of the new 16.6 documentation.

Search aka Content Indexer
note: installed on separate server

1) TeamSite - How to install Solr and Zookeeper as Windows services. (pre-req) NOTE: this isn't in any of the new 16.6 documentation.
https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB10027579

2) cssdk.cfg was not set up for separate search server. un-commented section and added search server name. left port as default. NOTE: this isn't in any of the new 16.6 documentation.

search.server.host: **********************
search.server.port: 6715
search.server.maxConnectionLimit: 10

3) SOLR cloud connector configuration in the admin GUI - must save/be successful. NOTE: this isn't in any of the new 16.6 documentation but is part of the Linux install KB? https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB11103766

Configure the Connector

1. Login to TeamSite in your browser
2. Go to AdminConsole > Connectors
3. Select “SolrCloud” on the left hand navigation
4. Click on the “Edit” button to edit the configuration
5. Enter Localhost:2181 in the zookeeper URL, and click validate
6. Enter the desired shards and replicas or keep the defaults based on your business requirements
7. You should receive a message saying that your configuration is complete. Click “Close”.

3) cleared and recreated TeamSite collection and index via
https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB11841768

4) BUG Turned off debugging under ContentCenter Loggers (AdminUI>Logs) for Search (specifically for AbstractSearchCommand) which causes blank screen result issues and NPE in Content Center log. Bug ID is WCM-25621.

nipper

Goo call, we used to have sticky posts for new releases.

Here is what I have:

Linux: Some services do not get started with reboot because startup scripts do not pick up the login (source /etc/profile) and don't have JAVA_HOME set:

https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB12920156

Linux:

JBOSS will not completely start without internet access. WCM-24410

cat eventsubsystem.war.failed
{"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"eventsubsystem.war\".undertow-deployment" => "java.lang.RuntimeException: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 7 in XML document from ServletContext resource [/WEB-INF/applicationContext.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 7; columnNumber: 68; cvc-elt.1.a: Cannot find the declaration of element 'beans'.
Caused by: java.lang.RuntimeException: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 7 in XML document from ServletContext resource [/WEB-INF/applicationContext.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 7; columnNumber: 68; cvc-elt.1.a: Cannot find the declaration of element 'beans'.
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 7 in XML document from ServletContext resource [/WEB-INF/applicationContext.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 7; columnNumber: 68; cvc-elt.1.a: Cannot find the declaration of element 'beans'.
Caused by: org.xml.sax.SAXParseException; lineNumber: 7; columnNumber: 68; cvc-elt.1.a: Cannot find the declaration of element 'beans'."}}

cat iw-cc.war.failed
{"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"iw-cc.war\".undertow-deployment" => "javax.servlet.ServletException: Failed to initialize
Caused by: javax.servlet.ServletException: Failed to initialize
Caused by: java.lang.NullPointerException"}}

Resolution:

The issue is due to changes made in the new version of TS as reported in WCM-24410. This will be fixed in an upcoming patch. A patched activemq-all-5.15.5-nojasypt-nospring.jar was placed in both of the following directories to resolve the issue:
ApplicationContainer/standalone/deployments/iw-cc.war/WEB-INF/lib
ApplicationContainer/standalone/deployments/eventsubsystem.war/WEB-INF/lib

nipper

Also, not certain if this is a bug or what, but I saw OD, on a base server upgrade, configure odbase.xml for webServices when it was already configured. Support has not been able to replicate it, but net result was there were 2 entries in odbase.xml for webServices and the DB configuration and OD failed to come up.

bmanilow13

Thanks Andy - i can no longer edit the original post to add your Linux notes unfortunately - another note - we added a ticket to hide/disable that awful experience studio bar that shows up in the templates now:

OTCS Ticket 4192040 : Experience Center Integration Section on DCRs

FROM SUPPORT:
Currently, there is no way to remove that Experience Studio Integration section without code change (with unknown consequences). However, you are not the first customer to make this request, and we have a feature request submitted to either remove that section or make it selectable in a future 16.6.x update..

I have just learned from Engineering that this issue will be addressed in 16.6.01 which should be available hopefully by October, 2019.

bmanilow13

Another issue i forgot to mention...this might be Windows only.

Content Indexer Install
1) KB10027579 states you must install JRE 1.8 and set that as the JAVA_HOME environment variable
2) HOWEVER, when you go to install content indexer afterwards, the installer errors saying "JAVA_HOME cannot point to a JRE" and it cannot proceed
3) we had to install jdk-11.0.2 and point our JAVA_HOME environment variable to jdk-11.0.2 to get the installer to proceed.
4) have both values in the JAVA_HOME ev (jre and JDK) didnt work either
5) support said we were OK leaving the JAVA_HOME pointing at the jdk-11.0.2 (despite what KB10027579 states)

HeatherH

Eric, that KB (KB10027579) was written for 16.4, hence the JRE1.8 requirement. With 16.6+ JEK11 is now supported and 8 is no longer required. We can go back and edit that KB so it's more clear.

nipper

Another one popped up.

Linux, I ran:

sudo ./install_archive.ipl ****.spar /default/...

sudo did not pick up JAVA_HOME and you could see the errors. That isn't hte issue, since it was trying to rebuild the UI, it wiped out my iw-cc.war completely. I ended up restoring from snapshot instead of trying to fix. While there probably was another way to fix, that was easy for me.

Need to use sudo -E or add this to the sudoers file:

Defaults env_keep += "JAVA_HOME"

I reported it to support and hopefully ENG will put a check to see that $JAVA_HOME/bin/java exists before running any commands.

bmanilow13

@TS Support said:
Eric, that KB (KB10027579) was written for 16.4, hence the JRE1.8 requirement. With 16.6+ JEK11 is now supported and 8 is no longer required. We can go back and edit that KB so it's more clear.

please and thank you - so we can just install the JDK 11.0.2, set JAVA_HOME to it, and use that for the solr/zookeeper and Content Indexer installs?

nipper

@Andy Knipp said:
Another one popped up.

Linux, I ran:

sudo ./install_archive.ipl ****.spar /default/...

sudo did not pick up JAVA_HOME and you could see the errors. That isn't hte issue, since it was trying to rebuild the UI, it wiped out my iw-cc.war completely. I ended up restoring from snapshot instead of trying to fix. While there probably was another way to fix, that was easy for me.

Need to use sudo -E or add this to the sudoers file:

Defaults env_keep += "JAVA_HOME"

I reported it to support and hopefully ENG will put a check to see that $JAVA_HOME/bin/java exists before running any commands.

This fix is scheduled for 16.6.1

bmanilow13

Thanks Andy - we now have 2 environments up w/ 16.6 (content indexer on the separate server) which are relatively stable/working as expected.

A few other questions/issues:

1) what exactly does the "Simple Search" do? i don't think it works...we never get any results, but when we click back aka "refine previous search" and try again, we get the expected results back.

2) Our iweventproxy.log is getting crushed w/ the below ActiveMQ errors - can you see if yours is also? if not any ideas? thanks!

"\Interwoven\TeamSite\local\logs\iwui\iweventproxy.log"

2019-08-15 13:32:55,041 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:33:26,082 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:33:57,108 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:34:28,149 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:34:59,178 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:35:30,202 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:36:01,227 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:36:32,269 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:37:03,311 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:37:34,349 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:38:05,375 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect
2019-08-15 13:38:36,404 [ActiveMQ Task-1] ERROR org.apache.activemq.transport.failover.FailoverTransport () - Connect fail to: tcp://localhost:61616, error message: Connection refused: connect

HeatherH

Eric, Simple Search is broken and being fixed in upcoming service packs/releases. It's currently searching for only results with a locale (such as en_us) even if there isn't one set, causing no results to come back. bug ID for 16.6.x is WCM-23758 and it's aimed for 16.6.0.1.

nipper

Only a couple hundred thousand lines or so.... :-)

Every 30 seconds.

bmanilow13

Thanks TS Support (heather if that's you) - another one to add to the list.

Andy, so you are getting the same thing or similar in the iweventproxy.log? I'm not sure if i should tinker w/ opening ports or if its another bug.

nipper

Yep, same thing, checked one of the older systems (16.4.1) and it has thousands as well. Cannot hurt to log a bug but I assume it is not a big issue. Likely no help in opening a port as it's localhost

HeatherH

Eric and Andy, we've found a temporary workaround to remove these errors (edit eventsubsystem.war's log4j.xml under ApplicationContainer\standalone\deployments and turn the FailoverTransport logging to off from error) and have logged a request with engineering to hopefully get this resolved in future releases as well.

Bug ID is WCM-26394 and it's tentatively scheduled for 16.6.0.1.

bmanilow13

That appears to have worked - thanks - it looks like the code is referencing property files that doesn't exist. i checked our old 7.4 version and these files weren't there either. iwclienteventproxy.properties & iweventproxy.properties

2019-08-16 13:40:36,510 [TeamSiteLog] ERROR com.interwoven.bridge100.TFileListener () - Subscription state file 'D:\Interwoven\TeamSite\servletd\logs\iweventproxy.properties' not found.
java.io.FileNotFoundException: \Interwoven\TeamSite\servletd\logs\iweventproxy.properties (The system cannot find the path specified)

2019-08-16 13:40:36,541 [TeamSiteClientLog] ERROR com.interwoven.bridge100.TFileListener () - Subscription state file 'D:\Interwoven\TeamSite\servletd\logs\iwclienteventproxy.properties' not found.
java.io.FileNotFoundException: \Interwoven\TeamSite\servletd\logs\iwclienteventproxy.properties (The system cannot find the path specified)

HeatherH

I'd already logged an issue for that as well. If you create the directory and a blank/dummy file those will also go away.
Bug ID for this one is WCM-26403, also targeted for 16.6.0.1 (tentatively).

nipper

@TS Support said:
Eric, Simple Search is broken and being fixed in upcoming service packs/releases. It's currently searching for only results with a locale
(such as en_us) even if there isn't one set, causing no results to come back. bug ID for 16.6.x is WCM-23758 and it's aimed for 16.6.0.1.

Is there an order of magnitude schedule for 16.6.0.1 ? We really need search to work and I cannot start upgrading unless we are pretty confident it will work.

HeatherH

Andy, advanced search works. It's just "simple search" which is what happens when you type something into the vpath bar at the top and hit search which is not working properly without locales set.

nipper

Got it, thanks for clearing that up.

bmanilow13

Speaking of advanced search...how exactly do you get the custom metadata search attribute updates to show up in the advanced search window now?

I have made updates to both the below files (see attached), rebooted both servers, but am still only seeing 1 of the 2 attributes (Content ID) - do i need to delete the branches.db? or clear and recreate the TeamSite collection and index via https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB11841768

TS SERVER
\Interwoven\TeamSite\local\config\lib\content_center\customer_src\etc\conf\customer\search_custom.xml

SEARCH SERVER
\Interwoven\Indexer\etc\FieldMapping.xml

thanks!

HeatherH

Are there any indexed files on the branch you're searching with "Project Number" set? If not, it won't show up in the drop down.
Oh, and if you didn't run make_toolkit for the search_custom change it won't be there.

David Smith

One thing in reading all this that jumps out at me is that a bunch of these things were broken in 16.4 also and there have been many many promises over the years to make the product documentation better. I was told somewhere along the line that the 16.6 docs would be MUCH better. Clearly they are not. When is any company that owns this product going to make the product documentation current and accurate? There are so many KB articles - you can't possibly expect your customers to go find them anytime something is missing or wrong in the product docs. Now that a very large entity has purchased this product, I'm pretty sure they won't put up with this lack of effort on OpenText's part. It's unacceptable that the docs are so out-of-date.

David Smith

Eric - great new "feature" of SOLR - if you have a custom EA that isn't set in any of the files in a branch you're indexing, that EA won't show up in the drop-down list in the Search UI. Brilliant, eh? Never mind if you have custom scripts that may be running background searches for that EA and will hit Null Pointer Errors if it hits against one of the branches that doesn't use it (yet). It is also a new undocumented "feature" that bit me in 16.4. I received a manual query that allows me to test whether a specific EA is set in a branch before querying for it, which gets me around the issue. But still, you won't find it written anywhere.

bmanilow13

David - thanks - that makes sense now. we have project_number EA set for SOME files, but not all the files. so unless the EA is actually applied to every file in the folder/sub-folder it wont show up in the advanced search pick list?

even more concerning is search service keeps crapping out (Interwoven TeamSite Index Service). this server has 16gb memory...OT do we need to increase the heap space size on the search server?

[25 Aug 2019 00:55:13,803] WARN com.interwoven.hunter.index.Listener (ActiveMQ Connection Executor: tcp://servername/192.****.xx.xx:3035@63294) - Listener received an exception from the event system (see below) -- will attempt to reconnect.
javax.jms.JMSException: Unexpected error occurred: java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1952)
at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1971)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:345)
at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:224)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Unexpected error occurred: java.lang.OutOfMemoryError: Java heap space
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:222)
... 1 more
Caused by: java.lang.OutOfMemoryError: Java heap space

David Smith

Not exactly, Eric. If the EA exists on one or more of the files, you should see it in the drop-down. Of course, it all depends on which areas you tell the Indexer you want indexed. If the EA is only in files in the WA and you don't index the WA, then you're SOL. Same with Editions - if you configure not to Index all Editions, it may or may not be there. Regardless, if it exists in ANY of the files in a branch and is actually set with a value (that's important I think), then you should have it as an option in the drop-down. Only if none of the files that are indexed have a value set for that EA will it not be there.

As far as memory - we also use 16Gb for our search server, but I don't know how much data you have that's indexed. We don't have a ton of data. Is that error from your TS server or your Indexer server? It looks like ActiveMQ is having some trouble on your TS server. Maybe you need more memory on that server, or your ArctiveMQ DB has just gotten too big. That has bit us in the past and we clean out the /activemq folder from time-to-time. But I was told last week that the Indexer depends on ActiveMQ so we should stop clearing out that folder. I don't know where it draws its memory from or how it works at all, really. ActiveMQ has been a mystery to us.

nipper

Yea I have noticed that SOLR is a memory hog. We are at 16GB as well and think that is too low. Moving to 64 bit took away all constraints about memory and it seems that TS (more than just Solr) tends to just use that memory as fast as it can be added.

HeatherH

Eric, we do have a KB for the recommended memory settings for Solr/Indexer:
https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB11628358

Les Bleus

Any update on the 16.6.01 release date?

bmanilow13

Their latest email says the below...hopefully all the issues noted above for both Windows and Linux are encapsulated?

What's new in TeamSite 16.6.1?
The latest version of TeamSite, scheduled for release in November 2019, includes enhancements to intelligent experiences and usability, new cloud integrations and improvements to security and file system drivers.

TeamSite 16.6.1 highlights
• Translation automation powered by Google Translate™
• Improved content suggestions and AI-driven tagging
• New integration with OpenText™ Core Experience Insights to track events for customer journey mapping
• Import cloud audiences from OpenText™ Optimost™ for improved content targeting
• Features that simplify TeamSite onboarding, including out-of-the-box components and low-code scripts

HeatherH

16.6.0.1 (service pack for 16.6, not a full version like 16.6.1) is due out (tentatively) late October/early November. Your account executive can provide more information if needed..