nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Workflows stalling on External Tasks

MrShake

Hey Guys, I'm seeing a new behavior that I hadn't seen before.

What we've got (and its not much yet) is workflows using the 7.2 workflow modeler, that are stalling at external tasks.

All External Tasks are perl based. I have not found a pattern, I have seen stalls happen with as few as 4 or 5 files attached, to as many as 100 files attached to a workflow. And, it doesn't always stall on the same task.
9 out of 10 times, it does stall on the first external task though.
And then, to cloud it even more, it SOMETIMES (not all the time) stalls on the next, or one of the next external tasks if I use iwcallback on the command line to force the workflow forward.

I'm seeing nothing in the logs, and my perl scripts generate logs as well, and it looks to me like the perl task is never even starting. Is there anywhere else but iw-home/local/logs to look for an error thrown trying to run an external perl task?

It may not help much, but the task it usually fails on is a perl script that checks file type, if its an XML, it grabs the related DCR and auto attaches it. But its also stalled on email scripts, space detectors, and other tasks.

The final complicating factor is that the autoattach DCR script is the same we have used for many years under the old WFT models.

Thoughts on where to start looking? Nothing in the logs in iw-home/local/logs and nothing in my perl logging.

Find more posts tagged with

Comments

ISCBorisB

workflows [...] are stalling at external tasks

What is your tasks' lock flag?

If it is set to true, you may stall upon attaching already locked Files. If that happens,
WF Engine will try to obtain locks again and again (after some timeout), possibly indefinitely.

There are OTB WF mechanisms to deal with this situation intelligently but before we get there,
check if it is indeed the issue.

dataitem.png

xbindings.png

MrShake

Lock is set to false.

I added a timeout link that emails me after a 10 minute wait and then sends back to the previous task. So at least I know every time this stall happens.

nipper

Put a log in as one of the *very* first lines of the perl. Pretty much before anything happens.

I have seen this happen with thousands of files attached, but not as small as 10.

Any special characters in filenames or anything like that ?

Is it repeatable on the same files ?

MrShake

I've got logging on the first line of perl possible, nothing.

No special characters.

It is repeatable if you kill the workflow and start a new one, it will hang at the same place.

MrShake

Another note from today:

A user pushed a workflow with ~40 files on it. It stalled at the first external task. I had setup a timeout link, and it sent it to a notification task to email me. THAT task stalled as well.

nipper

OK, are these users OS users ? Can they log on the server ? Perl needs to be run as them (typically)

2017-03-14 09_19_31-OS BIRT - VMware Workstation.png

SortingOnAggregation.rptdesign

Adam Stoller

Also check the amount of memory on the server and the number of inodes configured for the server. Although those kinds of problems usually result in some kind of diagnostic logging in one of the various log files - it's worth checking to make sure. (Though, as per Nipper - verify that you are using OS-users only and not Non-OS-users.)

MrShake

These are all OS users.

Though, if it were a perl problem, NO workflows with external tasks would work... and 90% do, just the ones with a lot of files are not working.

ISCBorisB

These are all OS users.

Though, if it were a perl problem, NO workflows with external tasks would work... and 90% do, just the ones with a lot of files are not working.

It is hard to make any non-trivial suggestion with so limited data.

That said, it *sounds* like some non caught run-time exception where code dies before it can execute the transition.
To check that you may want to encapsulate your "main" routine into eval command and log out $@ variable.
On its own this won't make your code work but at least you will see the problem if there is one.

(Personally, I would recommend to do it always, transitioning to some "Error Processing Task" to cover all
"Gee, I did not even think about that..." situations)

MrShake

It is hard to make any non-trivial suggestion with so limited data.

That said, it *sounds* like some non caught run-time exception where code dies before it can execute the transition.
To check that you may want to encapsulate your "main" routine into eval command and log out $@ variable.
On its own this won't make your code work but at least you will see the problem if there is one.

(Personally, I would recommend to do it always, transitioning to some "Error Processing Task" to cover all
"Gee, I did not even think about that..." situations)

I don't think we are getting that far... let me break down the workflow to the point of it stalling:

(Using Workflow Modeler 7.2.0.0)
Start
Default Link
User Task (Create Content)
Default Link (Send for Review)
External Task (Auto Attach Files)

The workflow is stalling BETWEEN the User Task (Create Content) and the External Task (Auto Attach Files)

The External task (CMD.EXE /c $IW_HOME//bin/DCRautoAttach.ipl) does not appear to be being called, as this .ipl file contains a print to log in the first few lines of the script, and it does not log.

nipper

Try to narrow down when it works or not, by user, number of files, etc

Also, try it without the cmd.exe, just iwperl perlfile.ipl

Finally trim the perl script down to almost nothing, just return the task ID done.

Adam Stoller

[...]

The External task (CMD.EXE /c $IW_HOME//bin/DCRautoAttach.ipl) does not appear to be being called, as this .ipl file contains a print to log in the first few lines of the script, and it does not log.

As Nipper pointed out - you should not be using "CMD.EXE /c" for this - instead you should be using "$IW_HOME/iw-perl/bin/iwperl.exe $IW_HOME/??/bin/DCRautoAttach.ipl"

The "??" are because I don't have access to a server right now to see if this is an OOTB script - and you have a double-slash there. If this is an OOTB script, remove one of the slashes; if this is a custom script - it should not be located in IW_HOME/bin - it should be some place like IW_HOME/custom/bin or IW_HOME/local/bin etc.

MrShake

Thanks for sticking with me guys, I had a vacation last week, so I'm back at it this morning.

This is not an OOTB script, its a custom. It is, however, in the regular bin directory, as thats where admins before me have always done it.

I'll test removing the CMD part of the call.

The problem is, I've been unable to recreate the issue myself, but I'm going to keep trying.

My question for you guys, is there any other logging I can be checking when a user informs me their workflow is stalled?

ansari1279

The External task (CMD.EXE /c $IW_HOME//bin/DCRautoAttach.ipl) does not appear to be being called, as this .ipl file contains a print to log in the first few lines of the script, and it does not log.

Hi,
Were you able to solve this issue??
I seem to be in same trouble.
I have created a custom WF from PLC and added an external task which calls a perl script.
$IW_HOME/iw-perl/bin/iwperl $IW_HOME/custom/wf/myExtCall.ipl
The script does not log anything and seems that it doesn't get executed.
(Even I placed a syntax error in perl script)
The WF pass the task and transit to the next task with success. The WF does not halt neither shows any error.
false- lock attribute, no permission issue.

Thanks in advance.

nipper

If you have a syntax error then it will never run

One more thing to try,

when the job is stuck, run the script form a command line, with the appropriate Job ID< Task and workarea (plus any other parameters you pass)

Adam Stoller

Thanks for sticking with me guys, I had a vacation last week, so I'm back at it this morning.

This is not an OOTB script, its a custom. It is, however, in the regular bin directory, as thats where admins before me have always done it.

I'll test removing the CMD part of the call.

The problem is, I've been unable to recreate the issue myself, but I'm going to keep trying.

My question for you guys, is there any other logging I can be checking when a user informs me their workflow is stalled?

You can try to look through the various logs in IWHOME/local/logs/iwui and ApplicationContainer/server/default/log/server.log to see if anything is getting noted there - but you're generally best off if you add your own logging functionality to the code so that you have control over it.

Adam Stoller

Hi,
Were you able to solve this issue??
I seem to be in same trouble.
I have created a custom WF from PLC and added an external task which calls a perl script.
$IW_HOME/iw-perl/bin/iwperl $IW_HOME/custom/wf/myExtCall.ipl
The script does not log anything and seems that it doesn't get executed.
(Even I placed a syntax error in perl script)
The WF pass the task and transit to the next task with success. The WF does not halt neither shows any error.
false- lock attribute, no permission issue.

Thanks in advance.

Try running iwperl -w -c IWHOME/custom/wf/myExtCall.ipl from the command line on the server and see what shows up.

MrShake

You can try to look through the various logs in IWHOME/local/logs/iwui and ApplicationContainer/server/default/log/server.log to see if anything is getting noted there - but you're generally best off if you add your own logging functionality to the code so that you have control over it.

I've got logging in the scripts code, but I do not believe the script is even being called.

I have hopes that eliminating CMD will help