Excluding files before Content Processor

j0ej0e

Hi,

We are processing about 500,000 documents, many of these are very small files (<50 words) which have no concepts/context which fails the classification. I'm thinking of excluding these files from Content Processor and the tagging process.

Is there a way to tell the Content Processor to 'skip' this file, either via the Preprocessor or any other way?

I was thinking of doing a wordcount in the postprocessor and if the document in question is less than 50 words, then strip the metadata away.
But it will be better if i can count at the preprocessor, before going thru the tagging process, and indicate to the Content Processor to 'skip' this file
from the tagging process that follows the preprocessor.

rgds,
joe

Migrateduser

Yes you have a good grasp of what is possible. Here are some details.

There are a few ways to NOT process a given file.

1. If the files are in a predictable locations - you can use the Filter RegEx to exclude a file from processing
2. If the files have identifiable name patterns - you can use the Filter RegEx to exclude a file from processing
3. You could use your idea of counting the words in a POST or Final process and blank out the meta data file [or keep just certain fields]
I would not do it in a pre-process because it may hang a down stream process, I'd do it as late as possible.

morgen