nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

TS Search - indexing manager issue

Crsb

Hi,

we installed Search 3.1.1 couple months ago, it still hasn't finished initial bulk indexing. we have about 60GB of content, the TS Search is installed on its own server (redhat ES 5, 1GB of RAM).

1) would 1GB of RAM be enough to go thru bulk indexing? i personally don't think it is enough due to the massive number of files and version that will be indexed. i am thinking to request 4GB, is that reasonable?

2) when i look at the number of editions from the UI, i see 450 items from initial all the way to ed_450. i also see the same thing when i run ls in the server.
but the tsindex.log shows some edition numbers that i have never seen before like #76422? anybody knows where are those numbers coming from? are those real edition numbers?


[13 Oct 2009 12:09:20,873] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76414 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:20,885] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76415 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:20,910] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76416 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:20,926] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76417 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:20,951] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76418 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:20,969] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76419 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:20,989] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76420 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:21,009] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76421 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:21,063] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76422 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:23,968] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/ed_0284 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:26,548] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/ed_0285 in branch //CMSDEV/default/main/EN
[13 Oct 2009 12:09:26,551] INFO com.interwoven.hunter.index.BulkJobProcessor (BulkIndexer-0) - Will index edition //CMSDEV/default/main/EN/EDITION/#76425 in branch //CMSDEV/default/main/EN

Find more posts tagged with

Comments

Adam Stoller

Check the tmp/temp directory to see if there are a lot of zero-length files sitting there?

If so - this was a bug in the initial release of Search 3 - for which a patch was released. Those files were a byproduct of a bug which caused the indexing to go very slow.

Crsb

Check the tmp/temp directory to see if there are a lot of zero-length files sitting there?

If so - this was a bug in the initial release of Search 3 - for which a patch was released. Those files were a byproduct of a bug which caused the indexing to go very slow.

we have installed the only patch available when we installed Search 3.1.1. but the symptoms are very similar to what you described so i will reach to iwov support for this. however, i didnt find any files related to search under /tmp under and $search/temp.

how about the numbers "/EDITION/#76418" any idea what they represent?

nipper

how about the numbers "/EDITION/#76418" any idea what they represent?

That is a hidden edition. When you do a submit, TS automatically makes a hidden edition.

BTW, I think 1 GB of RAM is way too low for that server.

Crsb

Thanks Andy, i had some questions about them, but this article answered all my questions: https://customers.autonomy.com/support/secure/docs/FAQ/Interwoven/48741.html

it was written for TS 5.5, i wonder if most of it is valid for TS6.7.x?

it says that deleting a published edition will delete any hidden editions between the deleted published edition and the published edition before that, is that correct?

smwangi

We had the issue of very slow indexing...

make sure you have enough VM/swap at os level.

then resolve this way:

remove your indexed branches indexes

stop search, index and Eventsubsystem

disable events indexing in this file search.properties

start search and indexer

Add your branches again

smwangi

Sorry forgot then enable event indexing in search.properties file

and restart event SS, index and search.

Watch your VM/swap space all through the process

Crsb

Sorry forgot then enable event indexing in search.properties file

and restart event SS, index and search.

Watch your VM/swap space all through the process

thanks!

what would be typical available swap space to index 60GB of data which includes 2000 editions and massive amount of submit events?

we have Search installed on a VM redhat 5, 1GB of RAM, 2GB swap space and 20GB disk space for collections.

Jamik

thanks!

what would be typical available swap space to index 60GB of data which includes 2000 editions and massive amount of submit events?

we have Search installed on a VM redhat 5, 1GB of RAM, 2GB swap space and 20GB disk space for collections.

You're gonna want a much more beefier machine. We're using Autonomy IDOL as our standalone search tool for our enterprise and we've set the each machine up with 16gb of ram, 300gb disk space and non-vm servers. They are windows servers so I'm not sure about how much swap space you'll need but from what you've listed you definitely need to boost both the ram and disk space. Typically what IDOL likes to do is keep as much of the indexed fields as possible in memory so the more memory you can give it the better.

Since TS Search is a slimmed down version of IDOL you may not need as much resources as you would with a Standalone IDOL server but I will say you probably gonna want to have disk space equivalent to the size of your TS server and the more ram the merrier since I suspect they still try to put as much of the indexed content into memory as possible.

Adam Stoller

I'd check with Support, but my guess is that you'd want at least 2 and perhaps 4 GB RAM, and at least a couple of CPUs as well.

smwangi

Your specs look low. Although I did manage to index 45g with a similar number of editions. on machine with 8g of RAM 16g swap. this machine was also running Teamsite though.

If I was you I'd start with 4g of swap and only do a few branches at a time. I had a script which checked if the indexer was idle and if so added a branch for indexing.

Of course a better solution is a beefier machine.