nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Limit active jobs.

dazzlad

Hi

I am hoping this is simple enough.
I am developing a workflow that will perform some pretty bandwidth/cpu intensive operations.
I don't have any visibility on the load that the TS server is supposed to cope with yet, so I'd like to add an iw.cfg variable along the lines of:

maximum_concurrent_workflows=50

So when I leave the site, I can get the client to play with this value if load starts to overwhelm the server.

My question is in two parts:

1. Have I missed something in the manual? Can I already configure this somehow?

2. If I have to implement this (please no), is the process below sane.

- Externaltask: parse iwgetwfobj to see if too many tasks are running.
- If too many tasks are running pop my job ID on to some kind of queue.
- Transition to a user task with a timeout.
- When timeout occurs, perform checks again. pop a queued task of they queue if possible.

Thanks.
Darren.

Darren.

Find more posts tagged with

Comments

reddyl

I am not sure if there exists any such configuration to limit the number of workflow in the system.

The workflow subsystem only assigns the JobID and taskid's at the instantiation time so you need to queue up the workflows and then intantiate when the number tasks falls below a threshold.

Migrateduser

No, you haven't missed anything in the manual -- you'll have to implement this yourself.

Instead of a usertask transitioning to a timeout (because this basically uses up a workflow), I'd set up a daemon to manage the queue.

Instead of taking this tactic, what about monitoring the CPU / memory usage on the box? There are tools out there with which TeamSite could interact and restrict a job from starting if CPU usage is above a certain amount.

Dave

Environments:
(1,2) TS6.5, TS6.5SP1 on W2K3
(3) Vignette V7 Portal on Solaris 9

dazzlad

Hi Dave,

Really like the idea of monitoring the CPU usage, however may be a bit tricky as I have to distribute these TeamSite customizations as a *cross platform* self installing package. The usphot of this is that I am trying to keep it so that there are no installs other than TS/OD/CSSDK are required.

I am hoping that someone out there has developed a nice thread safe FIFO queue in 100% perl.

Thanks.

Darren.

Migrateduser

Well, good luck to you -- it certainly seems like a fair amount of work and I doubt if anyone here would be so generous, even Jeff who is normally very accomodating (except for his repeated allegations that I'm on crack (there's a time for everything and that's called "college")).

Dave

Environments:
(1,2) TS6.5, TS6.5SP1 on W2K3
(3) Vignette V7 Portal on Solaris 9

reddyl

The deamon should use cache mechanism to store the output of iwgetwfobj since its CPU intense operation.

I am not sure if this approch is better or not
- TeamSite trigger script that will increment the number of jobs value in database or file each time a wrokflow gets invoken.

dazzlad

In reply to:

- TeamSite trigger script that will increment the number of jobs value in database or file each time a wrokflow gets invoken.

Which event is fired when a workflow is invoked?

Darren.

dazzlad

In reply to:

The deamon should use cache mechanism to store the output of iwgetwfobj since its CPU intense operation.

Each new job needs to know how many active jobs there are at the time of invocation. Surely there is no scope for cached information here?

Thanks.

Darren.

Adam Stoller

You can use TeamSite::WFsystem's GetActiveWorkflows() and/or GetActiveTasks() methods to easily get a count of active jobs or tasks (depending on which you're using as a basis of determining a threshhold. If you do it in the WFT code - you can prevent the new job from kicking off. If you do it within an externaltask then you need to remember to keep in mind that each job / task-for-computing-this is going to be including itself and any other "pending" jobs in its count.

Assuming you go with the externaltask mechanism, there's no central queue available OOTB but you could create one in a number of different ways to provide FIFO type behavior. The externaltask ("et_A" - used solely for determining if the next task can be run) could simply exit without transitioning after logging it's desire to proceed in the central queue; another externaltask ("et_B") transitioned to after the intensive task is completed) could check the queue to find the next task available - if there is one, it simply sends a callback to that task to transition it and removes the entry from the queue (perhaps in the other order)

An off-the-top-of-my-head way of doing this would be to create a directory (IWHOME/tmp/taskqueue/) and when the threshhold has been reached, have et_A touch a file with it's own taskid as the filename in that directory (e.g.: IWHOME/tmp/taskqueue/3832), et_B could then retrieve the list of files in that directory and sort them (by name will probably work otherwise use something like the '-M' test) and take the oldest file, rename it ("_3832") and then create a task object for that taskid (3832) and do a callback on the task (or just call the iwcallback CLT directly), and finally remove the file (_3832) - if anything goes wrong in the transitioning, you can leave the renamed file in place and pluck the next oldest task from the queue and work on that, etc.

The idea needs some refinement and details to implement - but I think it's a pretty simplistic and reasonably reliable mechanism to do what you're describing.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

reddyl

"WorkflowActivate" events is fired each time a new job is instantiated, check iwevents.log file for details.

dazzlad

Thanks for that.

I have had some time to play with this today.

My initial external task will now check how many jobs are running and if there are too many, transition itself to a user task (this way a user can force start something if need be).
It also writes a zero byte file named as it's own job ID to a queue directory.

The final part is to have completing jobs read this directory and re-start suspended tasks until the job limit is reached again.

My next question is....
I am reading my queue directory like this

Windows: dir /D /B
Solaris: ls -tc1

Is there a nicer way to date order directory listings with Perl? Everything has to be cross platform and I really don't like having OS conditionals in my code.

If anyone is interested in the end result, I can post here.

Thanks!

Darren.

Edited by dazzlad on 01/05/06 05:48 AM (server time).

jbonifaci

I'm not sure if there is an easier way, but you could probably do something like this:

my $directory = '';

opendir(DIRECTORY, $directory);
my @sortedFiles = sort {-M $a <=> -M $b} grep(-f, map("$directory/$_", readdir(DIRECTORY)));
closedir(DIRECTORY);

print join("\n", @sortedFiles);

This will of course return the full path to the files. If you just wanted the file name, you could add:

my @sortedFiles = map($_ =~ m|^$directory/(.*)$|, sort {-M $a <=> -M $b} grep(-f, map("$directory/$_", readdir(DIRECTORY))));

~Jeff

Adam Stoller

My initial external task will now check how many jobs are running and if there are too many, transition itself to a user task (this way a user can force start something if need be).
It also writes a zero byte file named as it's own job ID to a queue directory.

I believe you want to use a task ID for the files - not a job ID - as per my earlier suggestion (a job does not have a transition, a task does)

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

dazzlad

In reply to:

I believe you want to use a task ID for the files - not a job ID - as per my earlier suggestion (a job does not have a transition, a task does)

Yep, sorry, 'twas a typo.

Darren.