xPlore freshness

We are doing some testing with xPlore and we are not 100% sure how the freshness works. The first question, does xPlore use the r_modify_date to influence ranking when freshness is on? If so how does the modify date influence the ranking returned? Does the date start weighing in more when content is hours or days newer. So for example if I have three object, the first two objects have car mentioned twice and the third has car mentioned once. The object r_modify_date are all with in minutes of each other. I am assuming if I search on car that the two objects with the two mentions show up first and the object with one mention third? Now if I update the third object (still only one mention of car) will it show up first or does it depend on how much time difference between the third object and the first two?

I guess I am assuming freshness does not mean all new objects move to the top, and that the other ranking does play into the results as well. I am trying to understand if that is true and if so how much time separation is needed to cause an object to move up?

Find more posts tagged with

Comments

XdctmX

Go through the Xplore Administration Guide, If you havent gone through this already. It has some information regarding freshness.

Configuring scoring and freshness

xPlore uses the following ranking principles to score query results:

• If a search term appears more than once in a document, the hit is ranked higher than a document in which the term occurs only once.

• If a search term appears in the metadata, the hit is ranked higher than when the term occurs in the content.

• When the search criteria are linked with OR, a document with both terms is ranked higher than a document with one term.

• If the search spans multiple instances, the results are merged based on score.

Results scores can be changed by boosting subpath metadata and by freshness of source documents. Make these changes before you index documents. To apply changes to an existing index, you must reindex your documents. Implement these scoring changes by editing indexserverconfig.xml, located in dsearch_home/config. Shut down the xPlore instances before applying your changes. Validate your

changes using the validation tool described in Modifying indexserverconfig.xml, page 36.

Boosting metadata in scores —

Scores for hits in metadata can be increased by adding a boost-value attribute to a subpath element. The default boost-value (multiplier) is 1.0. In the following example, a hit in the keywords metadata doubles the score for a result:

Boosting recent documents —

<sub-path returnable="true" boost-value="2.0" path="dmftmetadata/keywords" />

The Documentum attribute r_modify_date is used to boost scores in results. By default, a freshness boost is applied to the default collection. The multiplier is based on how recent the document is. To remove this boost, set the property enable-freshness-score to false on the parent category element. For example:

<category name=’dftxml’><properties>   </properties></category>...<property name="enable-freshness-score" value="false" />

fredyT

I have been through it many times. But it is not very clear on how the freshness booster influences the results. Our business partners will want a little more information as do we.

So for example does the freshness only happen in groupings with the same ranking. As mentioned in my earlier post if I have two object that have car mentioned twice and an third object with car mentioned once and I search on car, would I ever have a scenario where the third object although ranked lower would get boosted above the two object ranked higher (assuming first two object have same ranking and third lower ranking)? if so, what is the multiplier, if a doc is new by one day does it have x boost, if one week etc it is boosted by Y, or does the boost only work with in the same ranking?

Another example is if I have an object that has car mentioned two times but is 2 years old and I upload a new object in the system where car is mentioned once and I search on car which one is ranked higher?

Thanks

8453_bueche

A few more notes on freshness: As your example points out there is a tension between the normal tf/idf scoring (term frequency / inverse document frequency ) of lucene and freshness. In xPlore 1.0 the weighting of freshess relative to this was not straight forward and was hard to predict given a set of documents. In addition, xPlore 1.0 would weight more recent documents (within a few minutes) much more over documents that were even days old.

This has been improved in xPlore 1.1 in two ways:

1. There is now a freshness-weight that allows you to weight how much freshness influences scoring over the tf/idf (as a percentage). This is set in the indexserverconfig.xml. Think of the score as being defined by:

score = ( tf/idfScore(doc) * scoreWeight ) + (freshnessScore(doc) * freshnessWeight)

where scoreWeight = (1 - freshnessWeight)

So, one can essentially turn off tf/idf by setting the weight to 1.0, or one can completely disable freshess by setting it to zero. It is still the case that the behavior for a freshessWeight 0 < and < 1 is still an inexact science to predict, but it is much better than in 1.0.

2. The aging is linear and spread over a 6 year period. The upside of this is that it works better for most docbases, the downside is that the difference between really recent documents is less discernable.

xPlore 1.1 is released with the Documentum 6.7 stack but does support earlier versions of the Documentum Server.

hope this helps
Ed

fredyT

Thanks Ed