nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

More DA Job Chain questions (and comments)

HFCDev

Hi,

I've been doing more work on my JobChain that will walk a folder structure. I have a couple of questions about the way things are supposed to work as well as some observations.

I've written my Split function and it does its job of chunking the data. The first odd thing I noticed was that the Map() function gets called in the first run of the agent request handler right after split for the first chunk from my split (i.e. the first item in the Set list). All the other chunks get called on the next run of the agent request handler (perhaps when I'm doing this with the service, these sub jobs will break themselves out differently). Am I correct in assuming that it's called a job chain because the first chunk in the chain becomes the master job to which all the others are now subordinate?

The next thing I noticed was that .fTaskData seems to have a different meaning depending on which function you're in (Chris Meyer commented on that in some DA code he wrote that I looked at). In Split(), .fTaskData is whatever you originally put in; in Map(), .fTaskData is either the same as the former, or is the chunk data you put in...ok no real problem here. However, in Reduce() or Finalize(), it's whatever you put into the retVal of your Map() function. This can make it tricky to write your Fingerprint so that it stays the same as what is in the taskRec from the calling request handler. Comment to OTDEV: It would be nice to be able to access the taskRec object that has the task ID in it. CUrrently there is no pointer back to this. There is .fWorker, but it doesn't give either a fingerprint or a task ID. I am thinking about the developer that at some point is going to need to track the process of the job chain so it can be monitored, and so that operators know when it's done. On that note, I was able to pass the original fingerprint all the way through to Finalize. Penultimate question: Can I assume in a job chain that Finalize() only gets called for the master job (in this case, the first chunk that had the other chunks as children)? Final question: Is there any way of consolidating results from the child Map() calls so that in Finalize, if I had a running tally in each job, I can add them up? Currently, even though Reduce() has access to the master count as well as any child count, I'm not seeing this consolidation anywhere that Finalize has access to. At this point, it's a nice to have.

If you've read this far, congratulations, you're as big a geek as I am

-Hugh

Find more posts tagged with

Comments

David Templeton

[dt] that is the intention, but we have observed an issue with this and needs to be addressed.

[dt] please see other response about GetFingerprint, it is only in Map that calls getFingerprint

There is .fWorker, but it doesn't give either a fingerprint or a task ID. I am thinking about the developer that at some point is going to need to track the process of the job chain so it can be monitored, and so that operators know when it's done.

[dt] how do you mean monitored? Please expand on this. The fWorker is an internal reference

On that note, I was able to pass the original fingerprint all the way through to Finalize. Penultimate question: Can I assume in a job chain that Finalize() only gets called for the master job (in this case, the first chunk that had the other chunks as children)?

[dt] it only gets called for the top of the tree

Final question: Is there any way of consolidating results from the child Map() calls so that in Finalize, if I had a running tally in each job, I can add them up? Currently, even though Reduce() has access to the master count as well as any child count, I'm not seeing this consolidation anywhere that Finalize has access to. At this point, it's a nice to have.

[dt] we agree return information should be allowed to be set for finalize to pick up

David

John W. Simon, Jr.

Hey Hugh,

Your comment about split and map are right on. I had issues with this on my first DA as my code was processing the first chunk twice. This was due to a problem with the way my split worked (I was returning the original taskdata as well as the split taskdata in the set) but I eventually got it figured out.

In my testing I learned that the finalize() script is run for the parent but no results from the child tasks are passed in. It is because the _SubclassExecute in the Reduce TaskType deletes completed tasks so it makes sense that no children are found to pass into finalize() on the parent task. I spoke with Chris Meyer and together we came up with a way to pass results back to the parent so the process can be reported on (bascially using the cache). I have an example module I can share, if you are interested just email me.

Cheers...

HFCDev

Quoted David Templeton on 03/04/2015 10:04 AM:

[dt] how do you mean monitored? Please expand on this. The fWorker is an internal reference

What I mean is that if I kick off a DA job that will take several hours or days to run (i.e. something you need to do on 100s of millions of object, users are going to want to see their progress. When you schedule your task, you get back a task ID and an initial footprint. You may want to write this information to a table that tracks the number of objects processed, number of objects skipped, any associated errors that occurred, and you may need the Finalize to write back to this table to tell the users that the job is complete. I think with what I have, I can do this but it does mean I have to put my initial fingerprint (the one from when Split() occurs as an argument to the sub-tasks). This initial fingerprint is the only piece of data I can access that links back to what the user will have received when the task was first scheduled.

-Hugh

Chris Meyer

Hi Hugh,

What I mean is that if I kick off a DA job that will take several hours or days to run (i.e. something you need to do on 100s of millions of object, users are going to want to see their progress. When you schedule your task, you get back a task ID and an initial footprint. You may want to write this information to a table that tracks the number of objects processed, number of objects skipped, any associated errors that occurred, and you may need the Finalize to write back to this table to tell the users that the job is complete. I think with what I have, I can do this but it does mean I have to put my initial fingerprint (the one from when Split() occurs as an argument to the sub-tasks). This initial fingerprint is the only piece of data I can access that links back to what the user will have received when the task was first scheduled.

The API doesn't provide a way to track a DA in progress; although columns and facets sort of does it during a rebuild (i.e., it shows "Building..." on the Specific tab until the rebuild is complete). Unfortunately, the approach isn't reusable since the flag is set on the column or facet node directly (i.e., DAPINODE.pStatus) external to the DA (in $LLIAPI). You could do something similar, but this would only work if your DA were dependent on a DTree node. Even then, you don't want to start modifying values on core data types.

You could use the fingerprint to track your DA, but this means having to store it somewhere to persist a reference to it. To get around this (and a few other problems) I subclassed Job Chain and added a "lock key" parameter when starting a DA. Paired with the name of the DA (i.e., .OSNAME) it forms a unique key that I use to track a DA in progress (by writing it with CAPI.IniPut, and deleting it when the DA finishes). However, the lock key is also passed through the Map, Reduce, and Finalize calls and could be used as a key to store statistics about the DA somewhere. What's nice is that the lock key is deterministic (i.e., a userid, dataid, or anthing else) so you can always see if a DA for a given key is in progress (and likewise fetch your statistics).

Anyway, I hope that gives you some ideas. Let me know if you'd like to see it in action sometime: I find it simplifies things by a lot.

HFCDev

Hi Chris,

I saw your approach with Lock() (since I have the module in which you did it . I've come up with a similar but slightly different approach that matches my needs to be updating the status along the way. Since I know what my fingerprint will be ahead of time, I can create an entry in my status table and then I can pass its row ID in with the initial task data (and with the taskData that gets pushed as a sub-job chunk). By definition it will be unique (identity column). The only thing left now to figure out is a slick way of making sure that only one thread can update a single row at a time...see my other post on the subject.

-Hugh

David Templeton

First things first

DO NOT USE CAPI.Ini* FUNCTIONS FOR STATUS CHANGES!!!

You will kill the performance of the entire eco-system!!! Those functions should only be used, as stated over and over and over again for configuration settings that change very infrequently. They are cached in threads to reduce the db interaction and when changes are made will invalidate these caches.

If you think about tracking the fingerprint, well, this will break as well in a number of ways, some line of investigation include:

<![if !supportLists]>- <![endif]>Hashing the fingerprint to allow uniqueness, therefore straight up query will not help

<![if !supportLists]>- <![endif]>You have to check multiple tables and conceivably cause lock escalation that will hamper the system

As for tracking of a particular DA task, we have this on the backlog to no longer change the ID but change it state along the process. Then provide an API to allow calling apps to get status. If this is what you are wanting and wish to work with a supported interface please contact OT support an refer to issue LPAD-28728

Regards

David

Chris Meyer

Hi David,

DO NOT USE CAPI.Ini* FUNCTIONS FOR STATUS CHANGES!!!
You will kill the performance of the entire eco-system!!! Those functions should only be used, as stated over and over and over again for configuration settings that change very infrequently. They are cached in threads to reduce the db interaction and when changes are made will invalidate these caches.

Thanks. I wasn't aware that CAPI.Ini* functions were that inefficient with mutations. Fortunately, I'm using it infrequently since a single IniPut and IniDelete call is made over the life of the DA run... which is used on rare occasion anyway. I may consider looking at other places to store this.

I'm sure you're aware that $LLIAPI.ThreadCache uses CAPI.IniPut to store a tick value in KINI. Does this imply we shouldn't use the thread cache to persist values that change from time to time? If not, what's the alternative?

Thanks.

David Templeton

Chris

The thread cache is designed to allow caching of complex configuration structures and the thread cache version number is the state of configuration, it uses the tick value to indicate change only. It doesn’t change often and therefore doesn’t break the principle. It also means that such data is reloaded across the cluster automatically on change instead of every request.

For the da tasks taking a ‘long time’ do not design structures that expect this behaviour. If you need a similar non-cached table then there LLSystemData available and accessible via LLIAPI::LLSystemData

This utility offers the same functionality provided by the following

CAPI functions:

CAPI.IniDelete()

CAPI.IniGet()

CAPI.IniList()

CAPI.IniPut()

Instead of interacting with the KIni table, this utility interacts

with the LLSystemData table which has similar columns and index setup.

With the implementation of the system settings caching mechanism, it

is a requirement to move all non-system settings data out of the KIni

table. Therefore, this utility provides a smoothly transaction by allowing

developers to store any data that are not system settings data.

The following functions are implemented for this utility with similar

functionality and interface (same return value and input parameters)

as their CAPI counterpart:

Delete()

Get()

List()

Put()

Please refer to the documentation written in the functions for more

information.

David