Need Help with Job Chain style Distributed Agents

Hi,

I've written a simple Job Chain distributed agent that node walks a tree. Right now all I'm doing is echoing out the nodes that I find because I want to make sure this all works before I get it doing anything useful. I'm running on CS10.0U13 against SQL Server 2008 (case Sensitive).

I've succeeded in getting my split function to create a set of taskDatas with a range of ID's within the range of valid Data IDs for children of my root object. I can see it running through each of the sub tasks. Where I'm running into problems is that after running, the master job never completes because the sub-tasks never get an entry in WorkerQueueCompleted. In fact, the only place I ever see that an entry gets written to this table is in $DistributedAgents.TaskType.Reduce._SubclassExecute() but there it only writes to this table after it checks that none of the child tasks is missing from this table. How does the child task get into this table if I can never get past this point?

I should point out that I'm doing this in builder after having run the distributedagent.RunDistAgent and distributedAgent.RunAgent request handlers.

-Hugh

Find more posts tagged with

Comments

HFCDev

I think I have my answer. My theory is that the parent job runs first, so when it runs its Reduce() method, there are still child jobs. When the runAgents req handler runs again, the jobs are done and the parent can finish. I need to do a few more tests but this is looking like it.

-Hugh

John W. Simon, Jr.

You got it exactly. You should run the func=distributedAgent.RunDistAgent in between every stage of your func=distributedAgent.RunAgent.

Also one thing I found very helpful in testing is to pass the runIntervalSec parameter into RunAgent to limit how long the worker executes (func=distributedAgent.RunAgent&runIntervalSec=5). This sets it to run for 5 seconds, the default is an hour...made me think my agent was broken the first time I tested it.

Cheers...

Tim_Coleman

Quoted Hugh Ferguson on 03/03/2015 12:41 PM:

-Hugh

Yep, the child tasks' Map() functions will be called first, then Reduce(). Once all of the child tasks have been reduced, then the parent task can be reduced and finalized. I hope it works out for you!

Tim

HFCDev

Thanks for the responses.

One follow-up question. I've observed that when Reduce is called, that the taskData that was passed in to Map is no longer passed in to Reduce or Finalize, but rather taskData is the retVal of the call to Map. This has implications for the GetFingerprint() when called from either Reduce or Finalize if the fingerprint depends on anything passed in to taskData (i.e. for JobChains, the data range). Is this changed in 10.5? I'm in 10.0 U13.

-Hugh

David Templeton

Hugh

GetFingerprint is not called when the Reduce and Finalize are created. It should be noted that as an applicaiton developer you should not have to know the inner workings of DA since it would be subject to change and the last thing we want is have developers touching the internal code.

David

David Templeton

Quoted Hugh Ferguson on 03/04/2015 02:09 PM: