MapReduce in job chain executes a lot when sub-tasks added via Map()

Hi,

I noticed an oddity in the job chain that is probably intentional, but I thought I'd bring it up as this "feature" makes it more difficult to run in builder (i.e. single threaded). If your job chain Map() invokes _MapSubtask() 1 or more times, you will notice in the logs that it attempts to repeatedly run the Reduce function for the calling task, and fails out saying "INFO : This task cannot be completed because subtasks are still open. Retry on the next worker cycle."

If this occurs while you're in Builder, it may take a while for your thread to complete (if at all). It almost appears that the Reduce method is repeatedly executed in a loop with no sleep until it gets the desired results (i.e. all subtasks completed). If your subtasks involve a large folder tree, this could take quite some time.

I can see the power in this however, because it means if you did a bunch of stuff in the subtasks, you have an opportunity to roll back everything.

Question to OT: is the behaviour I noticed in CS 10.0 U 2015-06 intentional? Does it change in later versions?

-Hugh

Find more posts tagged with

Comments

David Templeton

Hugh

This reduce execution thrashing has been changed in 10.5 2015-12 with the rework of the framework. Now, the reduce task will be put to ‘sleep’ and the children are checked in set wise operation and if the children tasks are complete the task is ‘woken’ and executed.

David

HFCDev

Sweet! I suspected that would have been among the improvements done in 10.5 and beyond.

-Hugh

HFCDev

BTW I don't suppose you or anyone has a workaround for 10.0 development so that a task which implements a bunch of sub-tasks within the Map() function (as opposed to doing a split) can be run in a manageable time? I was thinking of doing a patch for my dev sandbox only that would do a temporary sleep in that thread of execution so that I could still use builder.

-Hugh