Way to find documents with similar content

Is there a way in Documentum (6.5 or later) to find documents with similar content? For example, when two documents differ in few sentences or terms, or just in non-textual part (pictures/charts). There are two tasks requiring this:

at the moment when a new content is ready for check in, quickly find similar documents in the repository and infor the user.
periodical cleanup process running through the docbase and reporting groups of documents having given degree of similarity.

I know that most of full text indexers provide such thing. There is "MoreLikeThis" function in Lucene API, and MS FAST reportedly builds something called "Document Vector" for every document. So, I hoped that, perhaps, there would be a way to get such info from FTDQL?

Find more posts tagged with

Comments

msroth

This is an interesting question and I was hoping someone smarter than me would have an answer. You could write a customization that utilized Lucene's MoreLikeThis() API but, that sounds like a lot of work, and not knowing that API, I'm not even sure it would work.

I was experimenting with the SCORE FTDQL keyword to see if I could get something to work. I started with a simple text file that reports a score of 10000. I subtracted one word from the file and it scored 9627. I removed the entire last sentence (about 10 words) and it scored 9428. I guess the point is that the SCORE is pretty sensitive and you could construct a process that scores your document and then looks for any documents with a known variance (say +/- 10%) of that score and suggest those docs to the user as potential matches.

Just a thought.

mpraditt

We're trying to discourage technical discussions in this forum. Please post all technical questions to either the Documentum Developer Community or the Documentum Support Forum. Your post is more likely to be seen and responded to in those locations.
My colleague, Jerry Silver wrote this informative post: Please Read - DO NOT POST TECHNICAL QUESTIONS TO THIS FORUM! for more information.
If you created this thread, please move it to one of the other forums. Select "Move discussion" from the Actions menu and choose the appropriate forum, as described in the Guidelines.
If you want to reply to this thread, please do not do so until it is moved.
Jerry has asked me to also respond to your post with this link as well so that others may see it. Sorry for any inconvenience.

DCTM_Guru

"There is "MoreLikeThis" function in Lucene API, and MS FAST reportedly builds something called "Document Vector" for every document. So, I hoped that, perhaps, there would be a way to get such info from FTDQL?"

- MoreLikeThis and Document vectoring aggregates documents based on topics (or groups of words). This is entirely different from what you are asking. You want search engine to find documents that have some high percentage of identical content. Search engines are not built to support this can kind of comparing. You might want to investiage how hash keys are generated. Maybe you can use hash key to compare similar documents.

msroth

This question really intrigued me so I did some reading and research. I discovered that there is an entire field of study devoted to identifying similar and near duplicate documents. Much of this research has been directed at how to identify duplicate or near duplicate documents in search engines, most notably Yahoo! and Google. Anyway, it is possible to create a "fingerprint" of a document that retains enough of its character that you can ask a database (or search engine) to return documents that score +/-10% of your document's fingerprint. I suspect that this is what Lucene is doing, but haven't looked closely at it.

I am prototyping an easy, purely Documentum-based solution to identifying similar documents based upon the research I have done. Watch my blog for future posts. If you are interested in doing some research on your own, you will want to look at the work of Michael Rabin (Fingerprinting by Random Polynomials, 1981), G. Salton (Vector Space Modeling, 1975), and Andrei Broder (Symantic Clustering on the Web, 1997). It's pretty heady stuff but fascinating. Thanks for peaking my interest on this, and hopefully this helps.

(To the forum police: Sorry to respond to this technical post (again) but it hasn't been moved by the owner.)

saladjiev

I also did such research on Internet and found some of these algorithms too. You are right, this is very interesting area, but I hoped to find an OOTB solution in Documentum before develping a custom one. Well, looks like there is no such thing. I'll definitely follow your blog on this. Meanwhile, I'm upgrading my Documentum VM to use xPlore and hoping to leverage the same MoreLikeThis function.

msroth

I posted a research report on my experimentation on this topic if anyone is interested: https://community.emc.com/docs/DOC-10945

You can also find it at my blog: http://msroth.wordpress.com

Wiesnoepersad

Check the section named “Content duplication checking and prevention” in the Content Server Admin Guide.

msroth

As discussed here and in the research paper, detecting "duplicate" content is not that great of a challenge. Detecting "near-duplicate" content (i.e., content that differs only by a few words or sentences) is a different challenge altogether. It is the latter that was the focus of the research.

jackdorsey

I use Fast Duplicate File Finder to find similar files. It works pretty well and can find similar documents even if entire paragraphs are swapped. There is a free version where the similarity mode is limited to first 10 groups so you can test if it works for you before placing an order.