Partial or Incremental Data Load

SebastianATL

Hi all,

We have implemented an OpenText Report based on SAP BW Data. Now, we need to decrease the amount of data loaded nightly. I know this is possible in SAP BW via partial data load mechanisms like “delta load”. Do you have any idea how to load data incrementally in OpenText? Would it be an option to only load changed data sets?

We are open for any kind of idea even if this would require the creation of additional tables and programs in SAP BW.

Thanks a lot for any help!

Best – Sebastian

jfranken

Hi Sebastian,

There is a built-in incremental update feature in data objects.  A description of how to enable and use the feature is included in the Analytics Designer User Guide.  Hopefully it will meet your requirements.

SebastianATL

Hi Jeff, 
thank you very much for your answer! I read the Analytics Designer Guide. But as far as I understood, it's only possible to add new lines: 

Data rows can only be added to the result set returned by a data set, provided that the data set definition does not change between updates. If you change the definition of a data set, for example, by adding or deleting a column, you must generate a new data object store without using the incremental updates option.

Or has there been any recent change to also allow to modify lines? A mentioned any help is much appreciated! 

BR - Sebastian 

jfranken

Hi Sebastian,

I found an enhancement request to include updated rows in the incremental update:

https://knowledge.opentext.com/knowledge/cs.dll/kcs/kbarticle/view/KB7185543

The request is still open.  I don't know the plan for getting it added to the product.  I suggest talking to your sales representative.  The sales team takes the lead on enhancement requests and they might also be aware of other solutions.

 

SebastianATL

Hi Jeff, 
thanks for this hint. I reached out to Steffen, our sales representative. 

BR - Sebastian

jfranken

Hi Sebastian,

I've been trying to think of a workaround, but I haven't come up with a good solution.  It might be possible to do something like:

• timestamp the rows in the database table with the most recent update time
• save the time the last data store cache was executed
  
• create a new data set that selects rows where timestamp > cache time
• filter out rows in the original data set where ID is in new data set
  
• union the two data sets

Obviously this is just a general outline.  I have not tried to implement or test it.  Hopefully Steffen can give you more information on the enhancement request (BIRT-3582). 

SebastianATL

Hi Jeff,
thanks for this idea. I had something similar in mind. But instead of using two tables, I thought about using a counter booking for existing entries and append changed records. But at first glance, your idea seems easier to implement! Thank you very much. 

I really appreciated that you came back once more. 

Best - Sebastian