Scheduling Workflow New Jobs

Greetings-
what's the optimal way to invoke a new workflow job on a scheduled basis (i.e. end of every week, month, etc) without any user interaction (or maybe just one initial setup user interaction)- an idea I'd like a sanity check on is to
set up a cron job, but what to invoke? is there an .ipl to invoke with my .wft as a parameter?

possibly alternative/related question: has anybody ever tried to use "recurring_days" variable in "work_order.wft" to accept an endofweek or endofmonth value (i.e. use of Perl localtime)? if so, how was this accomplished? Perl code within <template_script> section of wft?

Find more posts tagged with

Comments

Migrateduser

I've used the recurring_days variable in work_order.wft, but that was around two and a half years ago when I originally wrote work_order.wft. I have never used it since. As I look at my original code, I remember what a pain it is to do this sort of thing. All that's happening in the work_order.wft to invoke recurrence is to use a dummytask to wait the specified amount of time before kicking off another workflow (itself) using the instantiator. I have not used the <wftask> element in any of my workflows (in fact, I don't see any documentation for this element anywhere, so I'm not really sure if it's a Workflow Builder only thing or not), but I would think all you'd have to do is create a workflow that keeps track of whether it is a recurring workflow and manage itself, but instead of my paintstaking methods (which was all that was available in TS 4.2.1) use the dummytask to control when it recurs and the <wftask> element to control the instantiation. You will have to use one of my tricks to pass the original recurring interval to the nested workflow, otherwise you'll have to enter the recurring interval every time you instantiate a new job, which is not what you want to do. There may be more to it as well, as this was a very difficult thing for me to pull off back then. But it is doable.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

gsumers22texas

thanks for input- sounds very involved-

does a cron job that invokes the iwjobc and iwinvokejob CLTs sound feasible? any experience with these? can/should the workflowfile variable of iwjobc just simply be the wft file name? The wft would be set up like any other of the defaults

gsumers22texas

how does (or does it?) the TeamSite::WFworkflow module play within your suggested use of the dummy and wftask tasks? I can't find hardly any mention of how wftasks are used either? is this module what you refer to as the "instantiator"? or could that also be the CLT iwjobc?

Migrateduser

It is certainly feasible to use a cron job to schedule recurring workflows. However, if you require a user interface (i.e. you havea TAG_info section in your wft), you will need to programmatically use the nasty objects that are provided, but undocumented. See the thread in this Forum titled "array of files used by workflow-object in perl" for how to use this method. If you don't need any UI at the beginning of the workflow (i.e. all your data can be hardcoded in a job spec file), you can simply call iwjobc and iwinvokejob and that's probably the easiest way to go. Rob Huffstedtler would be an excellent resource for you as he has done probably all of this at one time or another.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

gsumers22texas

very good to hear some confirmation / alternatives of these ideas, and thanks for sharing your experience and reference-
know of any way to alert Rob Huffstedtler via devnet?

I think the approach we're currently considering is as follows:

one wft (modified off of one of the defaults) to still allow for "ad-hoc" instantiation of workflow jobs via the UI --> New Job panel, basically whenever a Site Administrator detects faulty content

a second wft (basically a copy of the first) to allow for "scheduled" instantiation of workflow jobs via a cron job-
created using the following methodology:
1. finalize development of wft (test in TeamSite workflow UI)
2. set iw_debug_mode and iw_output_file variables in wft
3. invoke wft and gen the job specification file
4. iwjobc prints out Job ID of job specification file
5. iwinvokejob instantiates the job once again (and again and again?) in the application (with same Job ID) in the Workflow UI
6. Set up cron job to invoke iwinvokejob on set schedule desired (end of week or end of month still to be figured out in cron)

Does this seem feasible and stable design? or are there any limitiations concerning Job IDs assigned that make this unreliable? Can JobID repeat within TeamSite indefinitely so that iwinvokejob won't grow "stale"? do any server or OS level actions (reboot, OS upgrades, etc) reset or effect Job IDs?

Migrateduser

WFworkflow.pm provides you with hooks into the workflow so you can get information about your workflow easily while in an externaltask. It does not, as far as I know, help with nesting workflows. The dummytask would be used at the end of your workflow to essentially put the job to sleep for the allotted amount of time until the recursive process needs to kick itself off again. The drawback to this is that the original job is alive for all that time, but if that isn't a concern, then it's no big deal. Once the dummytask reaches its time limit, it kicks off the <wftask> (which in work_order.wft is a cgitask that calls the instantiator directly and uses some weirdness to let the instantiator know which wft to instantiate) which starts the workflow over again and in the meantime the original job completes. I am going to open a case with support about the <wftask> element and see where the documentation is. The instantiator is the magic script that translates your wft into a job spec file and invokes it. The CLT iwjobc works on a job spec file only, not a wft file.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

Scroll down the list of posts in this Forum and soon you'll find one written by user "robhuffstedtler". Click on his name and you should be able to send him a private message with one of the links that appears on his profile page. Or he'll eventually respond to one of these posts most likely.

Rather then run iwjobc once and iwinvokejob many times, I would run them both each time. I don't think it would allow you to invoke the same JobID over and over again, and even if it does allow you to do that, it just doesn't seem like the right thing to do. Just run them both each time from your cron script - that will keep you out of trouble.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

gsumers22texas

Smitty-
to run both iwjobc and iwinvokejob each time in my cron, isn't there another preliminiary CLT to run? this clt (???) would be the one that creates the job specification file that iwjobc will read for to present job id- do you know what is this CLT that creates (or recreates) the job specification file and the new job id? the only way I know how to do this is to run it via the UI (New Job)- nothing in the CLT or WF manuals jumps out at me for this?

Migrateduser

What I typically do to make a job spec file (and there are plenty of other ways to do it) is to run my wft once using the iw_output_file key in my TAG_info section (this is documented in the Workflow Developer's Guide in the Debugging Workflow Files section). When you use this key, after you enter all the values into the initial GUI that pops up when you start a job through the TeamSite UI, it will save the resulting job spec file to the file you specified in your iw_output_file key. Then just use that job spec file as input to the iwjobc CLT in your cron. You can adjust any values in the job spec file by simply editing it if you need to. I suppose you could always parse it programmatically as well if you wanted to.

You can always just skip the wft file and create a job spec file yourself as well. All a job spec file is is the XML part of your wft. So if you don't need any user input, just create a job spec file yourself. There's nothing magic about it. Try iw_output_file and I think you'll be fine. If you need the syntax and where to put it in your wft, email me.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

gsumers22texas

this is exactly the way I know to create the job spec file (through the New Job panel) and have done so- but the more long-term question remains: is it feasible to keep using the same job specification file (and thus same Job ID) over and over again through a cron job that is running both iwjobc and iwinvokejob? in an earlier reply, you thought this wasn't a great way to do this (and I can see a scenario as to why: what if the job spec file is deleted or corrupted?)- so what else can we put in our cron job (before the iwjobc) to replace the manual use of the New Job panel to keep recreating / replacing the job specification file (and thus job id)? you've touched upon this question in your reply ("there are plenty of other ways to do it") but are any of those ways potentially scriptable and what is the name of this script?
would a certain trace in TeamSite reveal the script that's doing this "Create File" action?
would the solution be a standalone perl script that uses the WFworkflow.pm module be the key to auto-generating the job spec file? thus the steps in the cron job would be:
1. perl script invoking WFworkflow.pm module --> creating new job spec file and job id
2. iwjobc to get job id
3. iwinvokejob to instantiate

thanks again for your great help

Migrateduser

There's no harm in running the same job spec file through iwjobc/iwinvokejob over and over again. iwjobc will create a new JobID every time you run it, so there's no harm in that. If you need to customize your job spec every so often and don't want to re-execute your wft file a bunch of times, you can always just write a script that produces a job spec file. If you have dynamic requirements that will cause your job spec to require variable inputs, you can do this:

o keep a config file available and change the config file when necessary
o Maintain a job spec file template, that has placeholders that are easily parsed and replace the placeholders with values from the config file using a parsing script (easily created)

That's one way. I haven't needed to do this, but I easily could if the need arose. Parsing is so easy in perl and figuring out a way to implement placeholders in a text file is pretty simple. Using key/value pairs in your config files would be a good implementation.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

gsumers22texas

I think your reply just freed the logjam in my understanding-
I didn't realize that iwjobc actually created a new JobID, I thought it just read the same original one that was created with the creation of the job specification file- so the same job specification file can have renewed job ids (thus I now see value in your suggestion to include iwjobc in the cron!)

we haven't yet identified anything that will be need to be changed dynamically within the job spec, but if some evolve, we're going to use an almost identical wft with a <template_script> section and TAG_INFO values to use in the New Job panel for more "ad hoc" job creations by Editors- the 1st wft will be instantiated once via New Job, then revmoved from available_templates.cfg, and then instantiated through cron after that for eternity

I like you're config file / job spec file template idea and I think will pursue this as a way to add some "backup" to this setup

signed,
a fan of work_order.wft

Adam Stoller

FYI: I don't remember whether it happened in 5.0.x or 5.5.x - but certainly in TeamSite 5.5.2 you no longer need to run iwjobc *and* iwinvokejob separately - you can now use the '-i' flag with iwjobc to invoke the job directly after "compliation".

As far dealing with moderate-changing workflows from the command line... I long ago created a simple Perl script that took as input a "job specification template" and a series of key=value pairs, performed the key=value substitutions and wrote out a temporary job specification file and ran it with iwjobc/iwinvokejob (before the '-i' flag was available).

As has already been described - start with a wft, and use that to generate the initial job specification file - then everywhere you want values plugged in - inset simple string patterns that you can later search for and replace (e.g., ****_REVIEWERNAME_****) - and then, plug those key=value substitution pairs on the command line when you want to instantiate the workflow - either manually or from cron (or from a custom menu, etc.)

As for leaving a workflow hanging around with a dummytask until it some point re-instantiates itself -- my advice: Don't do it. Leaving workflows lying around tends to bog down the server. It doesn't seem like much when you're doing the initial development (one or two workflows hanging around), but in production use this tends to get to be something like the order of 500 to 1000 workflows hanging around and that ends up being a lot of data to store/parse through. If you don't need to do it, don't. Cron / 'at' is a far better solution.

--fish
(Interwoven, Curriculum Development)