TS6.5 Search

I guess I'm just not finding the documentation on search and 6.5 installation all that clear - so, since we're getting ready to do a 6.5 installation this week, I figured I'd post my questions here and hope that someone can answer them - authoritatively:

http://www.quotient-inc.com

Find more posts tagged with

Comments

smenon

Let me answer your questions in order..

1. You don't have to install and configure search/indexing when you install TeamSite 6.5. You can choose to do that later. I have to check if this affects the CC-Std Home Page because of the search portlet - I don't believe it does.

2. TeamSite Search is not memory or CPU intensive. Indexing however can be - depending on the volume of content you are indexing. This is why baseline indexing can be resource intensive. Having said that - I will add that incremental indexing is pretty light on resource consumption. You might see occasional spikes - but over a period of time, the median CPU consumption is between 5% and 10% (based on internal testing). To answer your question specifically - it depends on the hardware on the Oracle box and how frequently you expect to submit and index.

3. Using a flat-file for the database is not recommended because of a known OpenJMS scalability issue. We have added support for MySQL for the event-susbystem to enable to pick a lower cost RDBMS. As long as you are storing the indices (collections) on a shared drive and you maintain the same TeamSite server, store and branch configuration, you should be able to install Search on another machine, point it to the shared collections folder and then uninstall your search from the TeamSite server. Remember to change your cssdk.cfg file to point it to the new search server.

I hope this answers your questions..

thanks

--Sunil Menon
Product Manager
Interwoven, Inc.

Adam Stoller

neither)?

You don't have to install and configure search/indexing when you install TeamSite 6.5. You can choose to do that later.

Hmm, page 20 of the TeamSite 6.5.0 Installation Guide for Unix seems to indicate this is not the case:InformationStepDefault Provided?Notes...Whether to enable Search, indexing, or both.step 9 on page 49
step 9 on page 53YesYou must enable at least one....And the information on page 49 and 53 seem to support this requirement of choosing at least one of those two options.Q. Is it reasonable to consider installing the separate serach component on a server that is already being used for an Oracle server?Is it likely to have a significant impact on the performance of the Oracle server?

To answer your question specifically - it depends on the hardware on the Oracle box and how frequently you expect to submit and index.

Presumably we would set things up to be incrementally indexed every time a submission was done (that is an option - right?) so that the load, after the initial indexing, should be relatively low and even - yes? I'm not entirely sure what the hardware setup is for the Oracle servers - I'm assuming they're reasonably well decked out though perhaps not super-mega-powerful.Q. If I opt to install everything on the TeamSite server - for now - using a flat-file for the "database" ... How difficult will it be - later on - to uninstall the search component from the TS server, install it on a separate server (perhaps the Oracle server) and/or change from using a flat-file to using an actual DB?

As long as you are storing the indices (collections) on a shared drive and you maintain the same TeamSite server, store and branch configuration, you should be able to install Search on another machine, point it to the shared collections folder and then uninstall your search from the TeamSite server. Remember to change your cssdk.cfg file to point it to the new search server.

I'm assuming here that the initial installation (all on the same server) while we work on porting our customizations over will probably be set to use a flat-file just to get it up and running but not for any real usage. If/when we move to using a different server it will most likely be to use Oracle -- in this case, I'm assuming that all existing indexes, etc. will have to be "thrown out" and all indexing will have to be re-initialized - right?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Adam Stoller

Our new TS box has 4GB RAM and dual 1.2Ghz Sparc processors.
Aside from an initial load from the initial indexing - is it reasonably likely that this hardware could reasonably handle both TS and Search on a day-to-day basis without problem?

Also - was the mysql support added in 6.5.0 or is that upcomming support in 6.5 SP1 ?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

smenon

MySQL support was added post release of TS 6.5. There is a KB (53407) that describes how you can configure the event-subsystem to use MySQL.

--Sunil Menon
Product Manager
Interwoven, Inc.

kajjubee

You can install TeamSite 6.5 without the search and/or indexing component. Search is a separate installation. It took all my system resources on a V210 2cpu/2GB RAM, solaris 9 (installed it on the TeamSite server itself), so I have stopped using search for now. Just on bringing up the search & index server I ran into this...

John742

Were you able to get Search working on this same server as Teamsite? Also what type of DB or you using? I cannot find documentation on configuiring a Database to be used with Search in the install or admin guide for 6.5. Does Search use a flatfile out of the box? It seems documentation is very limited for Search. Thanks in advance.

Adam Stoller

Actually - it is set up to work right now - and although we're still in the early stages of playing with 6.5 - the searchagent seems to be using about 1 gig of memory in SIZE but about 36 meg in RSS right now (essentially dormant at this time).

I too question the information about DB's - as all the DB related stuff seemed to be geared towards the event server and reporting system and *not* to the search system. Perhaps Sunil (or someone else) can clarify the relationship of search to external DBs - if any.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Migrateduser

Interwoven search uses Verity searching technology in the back end so as to perform efficient indexing and searching. All information related to full-text indexes and metadata-indexes are stored in the file system as collections. A collection represents “metadata” or data about a set of documents with an optimized collection architecture for searching. Document metadata include indexes optimized for searching, document locations, required fields related to documents and, optionally, custom fields related to documents. Again, this information is stored in the file system and not in any database.

Hope that answers your question.

Regards
--Anirudha

Adam Stoller

Thanks! That does answer the question.
Somewhere I thought a DB was needed, and Sunil's post about adding MySQL support didn't correct that misunderstanding but compounded it when we went through the installation and couldn't find anywhere to set up the relationship to the Oracle server we had ;-)

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Johnny

Can someone explain more about the event subsystem, it's relationship to everything else... search, reporting, data deploying etc..
There is talk about using a db for the event subsystem, but why exactly is not clear to me.

John Cuiuli

smenon

It seems like there is a lot of confusion about the 6.5 architecture for Search, ReportCenter and the Eventsubsystem. Let me try and clarify things for you..

The event-subsystem is an event publishing server publishing TeamSite events using OpenJMS. OpenJMS as part of its architecture requires a place to store the published event information until all the subscribers have consumed the events. This is just the basic event information such as "a submit event" with some unique identifier using which a subscriber can go back to TeamSite and retrieve additional information such as the list of files that were part of that submit operation. You can choose to write your own subscriber to these events if you want to build a custom event handling application. By default - OpenJMS ships with a flat-file option for persistence (also known as the default database). You can choose to store your event information for persistence in any other commercial RDBMS such as SQL Server, Oracle, DB2, MySQL etc. As part of its garbage collection operation, the OpenJMS server cleans up all persisted events when all registered subscribers have consumed the event. Unfortunately, there is a bug in the OpenJMS implementation when you use the default-database or flat-file option for event persistence, that does not clear out the events even when all subscribers have consumed the events. This results in a the size of that flat-file to continue growing until the event subsystem performance is affected and it could potentially bring the event server down. If you configure the OpenJMS server to persist the events in an RDBMS, you don't run into this bug and the events get cleared during the garbage collection step and your DB size stays small. When we shipped TS 6.5, we did not have the time to test the event subsystem against MySQL and so we only certified it against SQL Server, Oracle and DB2. Subsequently, we added support for MySQL and the configuration steps to configure the event subsystem against MySQL is documented in the KB article.

All of this is still tied to the event subsystem and has nothing to do with Search or ReportCenter.

Now if you look at the architecture for Search and ReportCenter, you will notice that they are both based on an event notification system. In the explanation above, just treat the indexing server and the reportcenter module as two OOTB subscribers to the event subsystem. This is how the event subsystem is tied to the Search and ReportCenter modules. This is also why you would still need a database to successfully deploy TeamSite 6.5 - due to the requirement of the event subsystem.

As Anirudha pointed out in his response earlier, at the core of the search and indexing solution is Verity technology which does not require any database to store the indices. Verity uses a proprietary object store called collections.

ReportCenter on the other hand needs to store the data that is used to generate reports in an RDBMS. This is the second database that comes in the picture as the storage location for reportcenter data. Note that this database is different from the database used to store the event persistence information in the event subsystem (or OpenJMS). You can obviously use the same database server, but these are two separate database schema instances with completely different schemas.

I hope this clarifies things and if not, I think that this is all covered in the TeamSite 6.5 webcast that was recorded in late October 2004.

thanks

--Sunil Menon
Product Manager
Interwoven, Inc.