nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Query on Teamsite Search

System

Hi,

We have a requirement to enable search capability within Teamsite (7.1). It has been recommended that we install TS Search on a separate box (not on the one that hosts TS) owing to performance measures and have Event subsystem enabled for the same.

Any opinion on this set up/arrangement would be really helpful for us to proceed. Especially, on the event subsystem front.

Find more posts tagged with

Comments

Rick Poulin

We have a requirement to enable search capability within Teamsite (7.1). It has been recommended that we install TS Search on a separate box (not on the one that hosts TS) owing to performance measures and have Event subsystem enabled for the same.

Any opinion on this set up/arrangement would be really helpful for us to proceed. Especially, on the event subsystem front.

Your question is kinda vague, but yes, you do want to install search on a separate server. Otherwise, if search decided to index a branch at 2pm on a friday when you're trying to publish a big release and TeamSite slows to a crawl, there won't be anything you can do about it (short of killing the search service).

Not sure what you're looking for in regards to the event subsystem. It runs within TeamSite's app container and just notifies search about changes. It's a minor configuration thing to tell it where your search instance resides, and it couldn't care less if it's localhost or endof.world.com, so there's not much to talk about.

Migrateduser

Hi Rick,

I apologize for the ambiguity. Let me try again.

You have already validated that installing search on a separate box has a clear advantage. That is certainly reassuring.

On the Event Subsystem (ES) front, we havent been using Search or Reporting till date and thus kept ES disabled. So we are unaware of the kind of performance overhead our TS box would take when we enable it. Its safe to assume that the Teamsite server has a robust configuration.

We are using TS 7.1 on windows server

Migrateduser

In addition to my query, I would also like to learn if anyone has performed an installation of TS search in a separate box and the kind of issues they have faced doing so.

Thanks
BM
TS 7.1/Windows Server

Migrateduser

We have Search on a separate server and have no special issues other than normal Interwoven head scratchers. The Indexer just stops every so often - the process keeps running, but the Indexer just decides to stop indexing. That's super annoying, so we had to create a special script to make sure the Indexer is indexing. Not sure that has anything at all to do with Search being on a separate server. Search in 7.x is ****. We found a major bug in Search (if you have more than 5000 files and create lots of Editions you will likely find it, too) that won't be fixed (so they say) until 7.3.2 comes out. That doesn't have anything to do with it being on a separate server either.

Rick Poulin

On the Event Subsystem (ES) front, we havent been using Search or Reporting till date and thus kept ES disabled. So we are unaware of the kind of performance overhead our TS box would take when we enable it. Its safe to assume that the Teamsite server has a robust configuration.

Configuring the ESS has the advantage of letting your Search Index know when something has changed in TeamSite, leading to more efficient (faster and less resource-consuming) indexing of your content. Without this, your search can only reindex at predefined intervals and, depending on the number of changes, can cause performance spikes whenever it's time to index.

In addition to my query, I would also like to learn if anyone has performed an installation of TS search in a separate box and the kind of issues they have faced doing so.

My typical approach is to install search on the same box as teamsite in dev environments, and only separate them in QA/UAT/Prod environments. That lightens the server footprint for dev (esp. in the event that you have many instances) where the horsepower isn't necessary because of such light traffic. I have never personally seen a case where my dev environments have behaved differently than prod with respect to search.

But yeah, as Smitty said, there's a lot of problems with the search product in 7.x. I think 7.3.1 is especially bad; I haven't had *too* much trouble with previous versions. Those problems don't have anything to do with whether TS and Search are co-located.

Migrateduser

Nothing to do with Search on a separate box, but since I'm ranting....

Do you realize that out of the box in 7.x, the Exact Phrase option on TeamSite's UI Search widget doesn't do an Exact Phrase search? You have to set a (undocumented, of course) flag in AutonomyContent.cfg in order for this to actually do what you expect. Oh and then re-index. It takes literally days for us to re-index. Needless to say, I've had some vary irritating interaction with IW over their POS Search product.

mart2001

And if you do decide to install the TeamSite 7.3.1 Update 1 and Update 2 for search, dig out your command prompt, learn to backup your whole TeamSite folder, delete files here and there, copy files to those files you deleted, find that they are missing files mention in the documents. And on and on...

Migrateduser

Hi Dave,

Could you throw some light on the bug in TS Search 7x that occurs if you have more than 5000 files and create lots of Editions and likely to be fixed in 7.3.2.

Migrateduser

Hi Dave,

Could you throw some light on the bug in TS Search 7x that occurs if you have more than 5000 files and create lots of Editions and likely to be fixed in 7.3.2.

In a nutshell, When Autonomy took over Interwoven, they pretty much forced Interwoven to use IDOL as the engine for Search. No one really understood both sides well enough to make it work and they jammed it together quickly. Somehow, we are the first ones to discover that if you have more that 5000 files in your branch, Search won't find a full result set. We use Extended Attributes to assign an expiration date to our content - a cron job wakes up and checks for expired content and deletes/publishes it. We discovered that the expiration process wasn't finding all the expired files. This led to a very long support case, which resulted in us finding out that Search doesn't work if you have more than 5000 files in your branch. There is a default setting (undocumented, of course - there is no Autonomy/IDOL documentation for the way IDOL is configured with TeamSite) that limits the amount of files IDOL searches to 5000 (or 4000 depending on who you ask). It also turns out that IDOL searches not only the Staging area, but every single Edition you have, which means the amount of "files/versions" it searches through is significantly higher than how many you actually have in your Staging area. So you have to set this value much higher. Problem is, when you set the value high enough, your CPU blows up to 100% during the Search, and eventually your server will crash (it took a couple of hours for us).

Granted, we have expiration processes that wakes up every 2 or 3 minutes and when that config value is set high enough, it slows down the Search significantly, and they end up piling up on top of each other and slowing it down even more, gobbling up CPU that isn't available. Something's gotta give.

The 7.3.2 changes are supposedly going to make it so IDOL only looks at Staging and not all the Editions. They have a copy of our backing store and are supposedly testing this new Search against it. We are getting vague answers as far as if it resolves all the problems we're seeing in 7.2.1. We have no intentions of upgrading anytime soon unless we are convinced it will solve all our problems. We are trying to get them to create a patch that will resolve the Search problems so we don't have to upgrade everything, but they don't think that's possible.

If you think you are experiencing these same Search problems with 7.x (and the problems exist in all versions of 7.x), I can give you the config settings you need to change to make it work. Watch your CPU, though.

Rick Poulin

Hrm. Thanks Smitty, that's good to know.

Adam Stoller

Somehow, we are the first ones to discover that if you have more that 5000 files in your branch, Search won't find a full result set.

Are you sure it's 5000 files in a branch - or 5000 files that meet the search criteria?

I believe it's the latter, in that Search / IDOL won't return more than 5000 matches - and that this problem plagues LSCS queries as well (which also uses IDOL) such that even if you set a 'start' value of something like 4500, you won't get matches beyond the 5000th one found (this is 2nd-hand knowledge, so I could be a bit off on this).

In either case - it's not good.

Migrateduser

Are you sure it's 5000 files in a branch - or 5000 files that meet the search criteria?

I'm pretty sure it's 5000 files because we absolutely don't have 5000 files that meet the search criteria. The configuration entry is called maxResults, which is also confusing as it would imply the maximum number of results. The whole IDOL/TeamSite integration is a black box. They couldn't find anyone in engineering who actually understood how it was glued together. Regardless, it's a cluster...

Every couple of weeks I set the configuration up to the higher values and let it run and it finds a few dozen files that should have expired but didn't. Then when it stops expiring files I set everything back to the defaults. As long as I don't let it run for more than an hour or two, it runs ok, but the CPU is peaked the whole time.

Migrateduser

I'm pretty sure it's 5000 files because we absolutely don't have 5000 files that meet the search criteria. The configuration entry is called maxResults, which is also confusing as it would imply the maximum number of results. The whole IDOL/TeamSite integration is a black box. They couldn't find anyone in engineering who actually understood how it was glued together. Regardless, it's a cluster...

Every couple of weeks I set the configuration up to the higher values and let it run and it finds a few dozen files that should have expired but didn't. Then when it stops expiring files I set everything back to the defaults. As long as I don't let it run for more than an hour or two, it runs ok, but the CPU is peaked the whole time.

Hi, it would be helpful if you could confirm the settting. I did some review of the search configuration files and found the following setting in search.private.properties:


# This is applicable only if iw.idol.search.useState is set to true.
# This specifies the maximum number of results that should be returned
# such that all these results may then be obtained using the state token.
iw.search.query.maxResults.size = 5000

Migrateduser

Hi, it would be helpful if you could confirm the settting. I did some review of the search configuration files and found the following setting in search.private.properties:
# This is applicable only if iw.idol.search.useState is set to true.
# This specifies the maximum number of results that should be returned
# such that all these results may then be obtained using the state token.
iw.search.query.maxResults.size = 5000

Specifically, this is the setting I was referring to. It is undocumented and doesn't exist in the OOTB version of the config file as its defaulted to 5000.

In AutonomyContent.cfg, in the [Server] section

[Server]
MaxResults=1000000

The setting you mentioned was also suggested to be increased. As well as this other one in search.private.properties:

iw.idol.search.useState=true

Samir_Friend

Nothing to do with Search on a separate box, but since I'm ranting....

Do you realize that out of the box in 7.x, the Exact Phrase option on TeamSite's UI Search widget doesn't do an Exact Phrase search? You have to set a (undocumented, of course) flag in AutonomyContent.cfg in order for this to actually do what you expect. Oh and then re-index. It takes literally days for us to re-index. Needless to say, I've had some vary irritating interaction with IW over their POS Search product.

Hi Smitty,

Can you please let us know what is the undocumented flag that you've mentioned above. It'd be helpful if you could let us know how to set it with the help of an example.

Thanks.

Migrateduser

Hi Smitty,

Can you please let us know what is the undocumented flag that you've mentioned above. It'd be helpful if you could let us know how to set it with the help of an example.

Thanks.

Here's the response from IW Support about ensuring the Exact Phrase Search will actually work the way you'd expect it to work. Beware - this will force you to re-index your branch(es), which for us might take days:

This is likely a configuration you can change for your IDOL index.

When IDOL is indexing a document, by default, IDOL will ignore the indexing of the stopword. Every language has words that do not carry much significant meaning. In grammatical terms these are normally prepositions, conjunctions, auxiliary verbs and so on (for example, words such as "the", a, and, to in English). These words can be safely ignored when processing content.
By default, stopwords are not indexed, but you can index them using the configuration parameter StopWordsIndex in the [LanguageTypes] section in IDOL config file (AutonomyContent.cfg).

For example:
[LanguageTypes]
StopwordIndex=1

sheikh1

That's a very descriptive explanation from Smitty. I been configuring and using search since 6.x and I can say 7.1 version is the most buggy and troublesome release.

In addition to other issues mentioned earlier you need to have a fix (a jar file) from support for your messaging. Since 7.1 I/A has changed their messaging model and switched to somehting like mq series with topic/queue model. I didn't dig into their source code much but somehow the ESS were failing to open the port on 3035. The fix was updated jar file and creation of a folder manually because ESS were not able to create on its own when restarted. I suggest you better check with support for compatibility of using 7.3.x version with your current teamsite setup.

Samir_Friend

Here's the response from IW Support about ensuring the Exact Phrase Search will actually work the way you'd expect it to work. Beware - this will force you to re-index your branch(es), which for us might take days:

This is likely a configuration you can change for your IDOL index.

When IDOL is indexing a document, by default, IDOL will ignore the indexing of the stopword. Every language has words that do not carry much significant meaning. In grammatical terms these are normally prepositions, conjunctions, auxiliary verbs and so on (for example, words such as "the", a, and, to in English). These words can be safely ignored when processing content.
By default, stopwords are not indexed, but you can index them using the configuration parameter StopWordsIndex in the [LanguageTypes] section in IDOL config file (AutonomyContent.cfg).

For example:
[LanguageTypes]
StopwordIndex=1

Dave, thanks for a detailed explanation.

Migrateduser

Thank you Smitty for the detailed thought process. I hope that this setting applies to all constraints that cssdk api provides. And also, when you say reindex after changing autonomy content cfg, does I mean that you have to stop and start indexing services or purge all the indexes and restart?