nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

TeamSite Search 2.1.0 - Out of control!!!

FurnaceFace

TeamSite 6.7.1 with TeamSite Search 2.1.0
Solaris 9

Okay,

Got a good one here - our dev server starting throwing errors within TeamSite that it could not write to disk due to insufficient space. So I logically did a df -abk and noticed that was at 39% capacity. My iw.cfg is set to the default settings when it comes to the sizing stuff...

disklow_mbytes=50 # freeze server if disk space falls below this number (in MB)
disklow_knodes=50 # freeze server if less than this number of free inodes (in 1000s) on disk
disklowpercent=10 # freeze server if disk space or free inodes falls below this pct

I thought this was odd so I did a df -i o to see how many inodes I had left. To my surprise, I had 0 left. I was full up. Somehow on my slice, I had 7.5 million files. This server had been running fine with TeamSite 6.5 SP2 (with the appropriate search installed). I ran the TS 6.7.1 upgrade on it and our regression tests came through perfectly.

So after some hunting, I discovered that my /iw-search/temp directory had over 5 million files in it. Mostly in the temp and content directories. I stopped the index and search processes and deleted 20,000 non IW related files in some other directory. I turned the indexer and search back on and watched in dismay as the indexer was creating approximately 40 BIF files every second!

So I turned it off again and have been running a process to delete the content and bif directories out. I'm at a point now where the inodes are low enough that I can create DCRs again within TeamSite but I'm curious as to how to avoid the indexer on steroids issue. I checked all my settings and they are default - right out of the box. Any thoughts from people who may have seen this or have some suggestions?

I've attached my search.properties file for your perusing (and in anticipation of your requests)

Thank you in advance for your help,
/FF

Find more posts tagged with

Comments

Migrateduser

Please check for the following in you search.private.properties in %SearchInstall%/etc/

# For interwoven only. Do not change this unless interwoven support
# asks you to
iw.index.leave.gen.files=false

Stop both the search services.
Turn this to false if it is not.
Clear the temp files , restart again.

Please let me know if the problem still persists.

FurnaceFace

Hello HemantGaur!

Thanks for the suggestion and for your speediness... I looked up the setting you are referring to and it was already set to false. I cleared out the 5,000,000 temp files that were in there (the process took 6 hours to delete them all) and restarted the index server (with the iw.tsindex start command). Within seconds, the Index server was at it again - creating approx. 50-60 temp files a second in the /iw-search/temp/bif and /iw-search/temp/content directories.

Do you think this may be caused because I did an upgrade? To give you more insight, I've provided the search.private.properties file.

/Thanks,
FF

Migrateduser

I looked into both your configuration files and they seem to be correct.
The temp files created in the temp folder are ok, but they should be cleared when the indexer is done with the processing of those files.

During the intial bootstrapping step there will be lot of files and very high CPU consumption. I assume that you have the search server in the separate server machine and this is not hosting other services. Let indexer run for some time till you have all the branches indexed (U can still not start the search demon yet for faster indexing). Confirm the branch index status by iwndxlistbr CLT. These temp files should go away after the task is finished. If there is large number of branches in the branches.cfg you can confirm correct working by removing the entries from there and adding the branches through iwndxaddbr CLT.
--hemant

Migrateduser

One more thing, I just noticed looking at your previous posts that you have the search and teamSite on the same box. This is not the recomended configuration as it slows up both the CPU/Mem intensive TS and Search Server processes. This might be the reason for the search not getting enough resources and failing to clean the files.

debus

What was the final resolution to this problem?

I have just done a TeamSite 6.7.0 to 6.7.1 upgrade and have run into the same problem.

About 4.8 million files under [search-home]/temp were created but there are only 4 branches indexed with the largest branch having 7k files. The large number of files also consumed all inodes.

Migrateduser

Have u got the property value of " iw.index.leave.gen.files" set to false.
Please upload if there are any errors in the indexer logs.

debus

Thanks for the reply.

Yes - iw.index.leave.gen.files=false

The indexer error messages are for cannot create file errors as the file-system inodes were exhausted. Since the process was stalled due to running out of inodes I would expect the Search/temp files to remain in place.

The real question is... is it reasonable for 4.8 miilion files to be created under Search/temp for indexing 4 branches that collectively only have 20-30k files.

This seems excessive and I am wondering what the root cause may be.

Migrateduser

>>The real question is... is it reasonable for 4.8 miilion files to be created
>> under Search/temp for indexing 4 branches that collectively only have 20-30k files

NO. These files are remains of the indexer running over the time and indexer stopped/killed and these hence persist.

Stop the search services (indexer and search) and delete all the files in the Search_Home/temp/ folder. Also clear/backup the logs.

Start the indexer first and run CLT iwndxlistbr to get the indexed branches.
Index the additional branches using the CLT iwndxaddbr and than start the search service.

Let me know if there are any errors in the logs.

FurnaceFace

Hi Debus,

Sorry for my late reply - lots of RL stuff going on and I haven't had time to catch up here on DevNet.

The solution to my problem was very "windows-esque" in that I uninstalled and resinstalled Search 2.1.0 - I spent too much time trying to figure out what went wrong and not enough time wondering if I should simply do a 30 minute uninstall/reinstall.

I kept my FieldMapping.xml and branches.cfg file handy so I could simply drop them in after the 2nd installation.

Hope this helps,
/FF

citizen

Looks like the index routine jumped out of the box and started consuming human resources, too.

Could someone from Interwoven chime in on this?

Migrateduser

These temp files are required to speed up the indexing process and are cleared up automatically after they are consumed. Also look at the following config parameter in search.properties.
# Max number of files for BIF
iw.index.maxbifsize=4000

In case the indexer shuts down abnormally/killed these files can be left dangling. The bug-fix for enhancing the indexer to remove these orpahaned files at the startup based on iw.index.leave.gen.files, will be shipped with 6.7.1 SP1.

bug 71533: Large number of temp files in the search temp directory.

Migrateduser

Also please note that the mere numbers of the files to be indexed does not imply the indexable documents (and also the temp files). There are mutiple versions of the same files and also mutiple editions in a branch, which all need to be processed and indexed.

shameem

I dont know which Teamsite version you are using and not sure how it related with Search engine. The SP2 Bug fixes says

• When indexing a branch for the first time, indexing does not start immediately and goes into a loop, throwing an exception.

Check it out.

Adam Stoller

I dont know which Teamsite version you are using and not sure how it related with Search engine. The SP2 Bug fixes says

• When indexing a branch for the first time, indexing does not start immediately and goes into a loop, throwing an exception.

Check it out.

You do realize that you're responding to a 3-year-old post - don't you?