nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

iwmtbatch blocking 'Search Term' in MetaTagger UI

System

Hi all,

TeamSite 5.5.2 – SP2
MetaTagger 3.5
Solaris 8

iwmtbatch is blocking the “Search Term” function in the MetaTagger UI.

I added a new external task to our workflow that invokes the iwmtbatch CLT. This resolved an issue where the MetaTagger UI timed out after five minutes while trying to crack large files (say 5 MB or more). The batch process works great. As expected, the files are “pre-tagged” with suggested metadata, allowing them to be opened almost immediately in the MetaTagger UI.

However, there is one side-effect: Whenever iwmtbatch is invoked (from a workflow or command-line), MetaTagger UI users are unable to perform a “Search Term” in the “Search Metadata” window until the iwmtbatch completes. It’s almost as if iwmtbatch places a read-lock while it runs, preventing all other searches.

To reproduce this, you must have a “Controlled Category” field in your UI. Click on “Add/Edit”. When the “Search Metadata” window comes up, type in a searchable term and then click on “Search Term”. If iwmtbatch is running in the background, this is where the UI hangs until the process finishes.

Here’s what I’ve tried:
------------------------------------------------------------------------------------------------
Command-Line Execution:
iwmtbatch -suggest -save /<iwmnt-path>/hugepdf.pdf

Perl Executions:
my $ret = `iwmtbatch -suggest -save /<iwmnt path>/hugepdf.pdf`;
my $ret = system(“iwmtbatch -suggest -save /<iwmnt-path>/hugepdf.pdf”);
my $ret = system(“iwmtbatch -suggest -save /<iwmnt-path>/hugepdf.pdf &”);
------------------------------------------------------------------------------------------------
All of the above yield the same result.

How do I keep iwmtbatch from blocking the “Search Term” function in the MetaTagger UI?

Thanks,

Dan Pike
Senior Consultant
Trinity Consulting Group, Inc.
dpike@trinitycg-inc.com

Find more posts tagged with

Comments

Migrateduser

MT 3.x is single-threaded. If you run a big batch job against the server, it will be unable to respond to other requests.

This will be fixed in 4.0 to be released later this year.

If you have the licenses, you can use another MT server to run iwmtbatch

Nikko

orderlines.zip

27891.pdf

Migrateduser

Hi Nikko,

Thanks for replying. I also posted this issue with Interwoven Support. They gave me the exact same answer that you did. My only choice is to look into how many MT licenses we have. Perhaps we do have enough to run it on another server.

Thanks,

Dan Pike
Senior Consultant
Trinity Consulting Group, Inc.
dpike@trinitycg-inc.com

Migrateduser

Dan - Here is a bit more info:
The "iwmtbatch locking term searches" problem you are observing will show up only when iwmtbatch happens to be in the middle of processing a large file, i.e. when the MT Server is in middle of processing a large file.

On the other hand, a large iwmtbatch job of 1000 "regular sized" files will not interfere with Tagger GUI very much, even though the batch job might take 30 minutes to finish. When the Tagger GUI makes a request for term search it will go through in between the various batch requests, maybe a couple of seconds slower than usual.

Again, this will not be a problem in 4.0.

Edited by Nikko on 01/06/04 01:17 PM (server time).

Tooltip.ssd.zip

27897.pdf

27905.html

Koen

Hi there,

we did the same thing, but made it a serial task and also send an email to the job owner when the task is finished. They can they look into the job and continue setting metadata.

We made it a bit more intelligent, and checked if the metadata fields is already set, and only pre-generate metadata for empty fields.

cheers,
Koen

Migrateduser

You know, Nikko, your reply made me think (I know, it hurts sometimes). I divided up each tag by using the -suggest_by_tag option for iwmtbatch. Queued-up "Search Term" requests can now be processed between any of the suggest_by_tag calls. For a 5MB pdf, the longest a user had to wait for a "Search Term" to come back with results was 10 seconds.

The old iwmtbatch call was as follows (and since MT 3.5 is single threaded, large files prevent users from searching metadata while iwmtbatch is running):


 $ret  = `/opt/mt-home/bin/iwmtbatch -suggest -save $fileList`;

The new iwmtbatch calls below allow queued-up end-user searches to occur between each tag:


 $ret  = `/opt/mt-home/bin/iwmtbatch -suggest_by_tag "Title"            -save $filePath`;
 $ret .= `/opt/mt-home/bin/iwmtbatch -suggest_by_tag "Description"      -save $filePath`;
 $ret .= `/opt/mt-home/bin/iwmtbatch -suggest_by_tag "Keywords"         -save $filePath`;
 $ret .= `/opt/mt-home/bin/iwmtbatch -suggest_by_tag "FindOR Subjects"  -save $filePath`;
 $ret .= `/opt/mt-home/bin/iwmtbatch -suggest_by_tag "Menu Topic"       -save $filePath`;
 $ret .= `/opt/mt-home/bin/iwmtbatch -suggest_by_tag "originatorLevel1" -save $filePath`;
 $ret .= `/opt/mt-home/bin/iwmtbatch -suggest_by_tag "originatorLevel2" -save $filePath`;

This seems to work great so far. Thank you for helping me think outside the box on this issue, Nikko Smiley Wink

Dan Pike
Senior Consultant
Trinity Consulting Group, Inc.
dpike@trinitycg-inc.com

Migrateduser

Hi Dan-

Here is some more info on the behavior you are seeing. I got this info from Interwoven Engineering.

On a very large file (such as 1MB plain text), MT3.x's bottlenecks are in 2 different areas.

1. file transfer
2. actual processing of the document

while #1 is happening, document "chunked" into numerous pieces of "datagrams" to the server so that stuff like search can get between each chunks of document. So perfomance will suffer but server will still be able to repond to other requests. You can "top" on unix or have a look at taskmgr to see if your CPU usage is low while batch job is running - if it's low, it means that it's in #1 stage.

But for #2, even if u do only 1 tag at a time, you will see the server exponentially increasing its memory usage and max out the CPU. It will appear as if server is hung (no search response)... For example, when I did some measurement on my P4 laptop with 1GB of memory, 1MB of plain text took something like 4-5 mins to run over a single tag.

So even with your solution, it is dangerous to be running a batch job over large files unfortunately... You may see an OK perfomance while file transfer is happening but when #2 is happening, search will become unresponsive again.

Again, all this will not be a problem in 4.0

Migrateduser

Here is another suggestion: if you are running a multi-cpu box, you could start another instance of mtserver on a different port just for batch jobs. You can have the CLT point at the batch dedicated server. And the batch job should not bog the machine down too much as long as the machine is SMP or better. Even in a single CPU scenario, at least the IO bottleneck will be taken care of.

Nikko