Configuring facets in xPlore 1.6

Hello,

(already posted in forum xPlore, but seems not be used, so reposted here...)

Using xPlore 1.4, I configured facets by editing the file "indexserverconfig.xml" and adding some new dmftmetadata tags in dmft collection.
I tried the same thing in xPlore 1.6 P09 (according the documentation, it is the same process as the version 1.4), but it didn't work. The tags I add in the file are not shown in XHAdmin (/dsearch/Data/default, index dmft).
Finally, I was able to create the facet using XHAdmin, by changing and upgrading the index dmft.
Did someone successfully configure a facet by editing the file "indexserverconfig.xml" in xPlore 1.6?
Just to know if it is a problem in my installation or another strange thing...
Thank you,
Arnaud

Find more posts tagged with

Comments

Pedro Maia

I confirm that we have xPlore 1.6 P07, and we only changed 'indexserverconfig.xml' to create facets for D2. After doing this, there are specific steps to reindex and steps to configure the facets in D2.

Arnaud_Sestier

So, it is a problem on my environment...
Reindex and D2 is OK, no problem.
Thank you, Pedro.

RazvanPorojan

@PedroMaia said:
I confirm that we have xPlore 1.6 P07, and we only changed 'indexserverconfig.xml' to create facets for D2. After doing this, there are specific steps to reindex and steps to configure the facets in D2.

Hi Pedro, nice to find you here. I am working on this topic currently. Not clear for me why reindexing is needed in order for the facets to work. According to Lucene documentation facets should also work even if the field is not set to Stored which I believe is the equivalent of setting the returning-contents to True in indexserverconfig.xml. I quote: "If (and only if) a field is indexed, then it is searchable, sortable, and facetable"

However I will assume that reindexing is required. Is there any way that we can optimize this?

Pedro Maia

Hi, @RazvanPorojan - good to meet you here!
I'm afraid I don't know how to avoid reindexing when new facet fields are added... There is a Feature Request for this:
XPLORE-16986

RazvanPorojan

@PedroMaia said:
Hi, @RazvanPorojan - good to meet you here!
I'm afraid I don't know how to avoid reindexing when new facet fields are added... There is a Feature Request for this:
XPLORE-16986

Hi Pedro. Thank you for your ansewer. I would like to have a look at the Feature Request but the link is not working. I have in mind an optimization that would allow the reindexing operation without repeating the full-text extraction. Not sure if it is possible though.

Pedro Maia

Hi @RazvanPorojan,
The link is to the OpenText Support Case - you might not have access.
Below is a summary of the case.
Sincerely,
Pedro Maia

Support SR 3939607 - Why does xplore indexes need to be rebuilt for adding new object type as facet
VERSION: 4.7

What is your question?
D2 4.7P10
Xplore 1.6

We already had facets enabled in Xplore for about 10 attributes when the system went live last year. Now we are adding a new Object type and new attributes for that type for more business units and some of those attributes will need to be added as facets.
I added the attributes into Xplore indexserverconfig.xml (updated the version number also), but initially DID NOT rebuild the index.
After creating the D2 config for search/facets, when I searched I did not get any results for facets. In the XQuery I could see that new new object type and attributes were included. I even took the xquery and ran it against Xplore and did not receive any facets for the new type. Facets for other existing attributes were however being returned correctly.
As a last resort, I then rebuilt the index and then I was able to get facets to work for the new attributes.
Since this was Dev environment, it was relatively simple. But we cannot do this for Production environment where there is about 1TB of index (single Xplore with 4 collections).

What I need is the following (please open Eng consult if necessary):
1. Why was a rebuild necessary when adding new attributes for a new object type?
2. We cannot do the same rebuild in higher environments where there is lots of data. Is there a better way. Technically there is no data/documents added yet for this new object type, so it should be aggregating from this point onwards but not sure why a rebuild is necessary.
3. is there a different way to add the metadata directly to xDB without rebuilding?
4. Can rebuilding be done via any command line instead of logging into xploreadmin web
5. What would Eng recommend for adding new/additional metadata in future for facet when index is already present for nearly 2TB of content

Closed Ticket Notification: OTCS Ticket 3939607 : Why does xplore indexes need to be rebuilt for adding new object type as facet

Your ticket 3939607 has been closed. We hope we've addressed your issue to your satisfaction. Below is some information related to your ticket resolution:Problem Statement: Why rebuilding index is required for new facets?
Expected behavior: This is by design
Impact: Cannot do this for Production environment where there is about 1TB of index (single Xplore with 4 collections).

Q1. Why was a rebuild necessary when adding new attributes for a new object type?
A1. Facet need to have xDB index to search and calculate category.
This index data structure must create in xDB database.
When you rebuild index, xPlore will create new xDB data structure with definition in indexserverconfig.xml, so that new entries in indexserverconfig.xml will be created.
Then xplore will copy data from original xDB to new xDB database which have new data structure.

Q2. We cannot do the same rebuild in higher environments where there is lots of data. Is there a better way.
A2. There is no other way to recreate new index data structure other than rebuild or recreate xDB.

Q3. is there a different way to add the metadata directly to xDB without rebuilding?
A3. No

Q4. Can rebuilding be done via any command line instead of logging into xploreadmin web
A4. Yes, you can use xPLore script to rebuild index

Q5. What would Eng recommend for adding new/additional metadata in future for facet when index is already present for nearly 2TB of content
A5. I think we can raise FR request , and try to find new solution.

The following JIRA Feature Request was entered.
XPLORE-16986 - xplore indexes should not be needed to be rebuilt for adding new object type as facet has been successfully created.

Rebuilding the index is the only way to get new facets added (until the feature is implemented)
Rebuilding index have multiple steps need to follow, we did not expose rebuild index feature to command line yet.

RazvanPorojan

Thank you very much Pedro. I will keep you posted.

RazvanPorojan

Hi @PedroMaia,

I have one more question on the topic that might be interesting for others as well:
Because I want to limit the scope of the newly configured attributes to a specific Docbase (MyDocbase), I've implemented a Docbase specific configuration in the Index Server config file by adding a new category (NewCategory) which duplicates the default one:

<category name="NewCategory">
        <properties>
                <property value="true" name="track-location"/>
                <property value="true" name="enable-freshness-score"/>
                ...

and assigned it to the desired Docbase (domain):

<domain default-document-category="NewCategory" storage-location-name="default" name="MyDocbase">
        <collection usage="Data" document-category="NewCategory" name="default"/>
        <collection usage="SystemInfo" document-category="trackdb" name="TrackingDB">
            <properties>
            ...

However, it doesn't seem to work as expected. Do I need to create a collection as well?

Thank you,

Razvan

Pedro Maia

Hi @RazvanPorojan,
I'm afraid I never made such changes to indexserverconfig.xml, so I can't help.
Thank you very much.

Sincerely,
Pedro Maia

RazvanPorojan

Update: we tested the above configuration (New Category) and it works fine.