DA Best Practices - Daisy Chain Simple jobs or build a Job Chain?

Hi all,

I am working on a piece of code I want to run in the DA framework. My code uses nodeCrawler to walk a folder hierarchy, and for what I'm attempting to do, it works pretty nicely. Now my dilemna is I have something which on one hand lends itself nicely to a job chain, but on the other, doesn't lend itself to the job chain's method of breaking up the job which for the most part assumes a min and a max DataID and the range gets chunked.

Since I'm using a node crawler, my process more lends itself to specifying a fixed number of items processed at a time, and more importantly, the exact number is not always known when you kick off the crawler - you keep running it until it says "complete".

With that last thing, I thought that the best method maybe to use a simple job and have the simple job schedule a new version of itself if the node crawler isn't yet complete. For a large crawl, this could introduce a lot of daisy chained simple jobs.

I'm currently working in a 10.0 environment but want this to wok in a 10.5 and 16 environmnt. Am I heading towards any pitfals?

-Hugh

Find more posts tagged with

Comments

Hi Hugh,

Not done much with DA myself. The premise behind DA is that you have a lot of jobs that don’t do much. When I read that you have a node crawler, then it sounds not like a job for DA, because it takes too long. Why not do that as a background process. Why not split it up in a job that does its work for the current node and then adds the same job for its children.

Hans

From: eLink Entry: Content Server Development Forum [mailto:development@elinkkc.opentext.com]
Sent: Donnerstag, 16. Juni 2016 20:35
To: eLink Recipient
Subject: DA Best Practices - Daisy Chain Simple jobs or build a Job Chain?

DA Best Practices - Daisy Chain Simple jobs or build a Job Chain?

Posted by Ferguson, Hugh On 06/16/2016 02:30 PM

Hi all,

I am working on a piece of code I want to run in the DA framework. My code uses nodeCrawler to walk a folder hierarchy, and for what I'm attempting to do, it works pretty nicely. Now my dilemna is I have something which on one hand lends itself nicely to a job chain, but on the other, doesn't lend itself to the job chain's method of breaking up the job which for the most part assumes a min and a max DataID and the range gets chunked.

Since I'm using a node crawler, my process more lends itself to specifying a fixed number of items processed at a time, and more importantly, the exact number is not always known when you kick off the crawler - you keep running it until it says "complete".

With that last thing, I thought that the best method maybe to use a simple job and have the simple job schedule a new version of itself if the node crawler isn't yet complete. For a large crawl, this could introduce a lot of daisy chained simple jobs.

I'm currently working in a 10.0 environment but want this to wok in a 10.5 and 16 environmnt. Am I heading towards any pitfals?

-Hugh

[To post a comment, use the normal reply function]
Forum:	Content Server Development Forum
Content Server:	Knowledge Center

The reason for using the node crawler is two-fold. a) node crawler can do a bottom up enumeration, so that I get the most dependent items first, which is what I need, and b) it can be paginated, that is, it crawls for a page worth of nodes, then saves itself to cache (well OK, you have to invoke the SaveToCache command).

I'm now realizing that given that this operation is heavily dependent on the order of the file hierarchy, it may not be advantageous to use DA's as the real power of DA's is splitting up the job in chunks when the order in which the chunks complete is irrelevant. In my sitation, it's heavily order dependent. I originally considered the DA route as a way of keeping it off the front-end threads.

-Hugh

David Templeton

Hugh

The reason for using the node crawler is two-fold. a) node crawler can do a bottom up enumeration, so that I get the most dependent items first, which is what I need, and b) it can be paginated, that is, it crawls for a page worth of nodes, then saves itself to cache (well OK, you have to invoke the SaveToCache command).

Going forward the node crawler is probably the worst mechanism to use for any tree operation. For many reasons:

<![if !supportLists]>1. <![endif]>When it was originally created the depth and number of nodes it had to hit we significantly smaller than today, 10000 crawled nodes was considered a lot

<![if !supportLists]>2. <![endif]>The nodecrawler is single threaded, well that screams performance issues right there

<![if !supportLists]>3. <![endif]>Transactions are an issue as well

<![if !supportLists]>4. <![endif]>The nodecrawler is NOT really recoverable/re-startable on its own, if something happens to the instance that is running the thread you are left in some known state unless you build a bunch infrastructure.

I'm now realizing that given that this operation is heavily dependent on the order of the file hierarchy, it may not be advantageous to use DA's as the real power of DA's is splitting up the job in chunks when the order in which the chunks complete is irrelevant. In my sitation, it's heavily order dependent. I originally considered the DA route as a way of keeping it off the front-end threads.

First, DA can do the bottom up enumeration, in fact the physical delete task, aka purge. It rolls out the DA tasks down the tree then on the rollup in reduce completes the actual purge operation. While the tasks are executed independently it stops at branch points until of the children are complete. This is the real power of the DA infrastructure. For items like copy, to have the DA tasks reach a level then spawn sub tasks for the next level allows the branches to start to copy independently of the other. Another example, I have a module I use to populate data to boot strap a new system and it crawls the file system, it does the same thing starts at the top and crawls its way down the file directory. In the process is spawns other tasks…

For such a hierarchy need the job chain is your friend and with the update in 10.5 in 2015-12 much improved support.

David

Hi David,

I'm now rebuilding with job chains and am part way there. I had a close look at th purge agent as you suggested. If I read it correctly, basically, if you have a master task and a bunch of sub-tasks, so long as you do the operation you need rolled back in the Reduce method and not in Map, any sub-task flagging an error can make the entire operation roll back (good). I just want to double check that this is as designed. Also, I'd heard other developers complain that a transaction you start in the Map task that you choose to roll back will get committed regardless. Has that behaviour been observed?

Thanks

-Hugh