nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Unable to allocate shared memory for request

dazzlad

Hello,

Yesterday afternoon external tasks stopped running in our Workflows.

Our iwjoberrors.log file is full of messages like:

[PHP]Could not run /path/to/my/task.ipl for task 331051; retrying in 1 minute[/PHP]

Also, iwtrace.log is filling up with:

[PHP]Unable to allocate shared memory for request[/PHP]

We are on TS 6.5 SP2 on Solaris.
I've temporarily disabled the event subsytem as we suspected this may have something to do with the issue.

Running the external tasks from the command line as the relevant user completes successfully and progresses to the next task.

Has anyone seen this before, or have suggestions as how to diagnose the issue?

Many Thanks.

Find more posts tagged with

Comments

dazzlad

Just noticed that every time the error occurs in iwtrace.log there is a matching TaskInvokeExternal event in the events log - the timestamp is always identical.

nipper

You may need to reboot this puppy, I see errors similar to this in DOS, but not Solaris. Are there any processes with an abnormal amount of memory running or as zombies ?

dazzlad

Just found the solution:

For future reference see...

https://support.interwoven.com/kb/kb_show_article2.asp?ArticleID=54739

Thanks!

dazzlad

Oh, and for those who aren't sure to delete the workflow store means to delete the iw-store/workflow directory.

TS will recreate it upon startup.

ISCBorisB

Oh, and for those who aren't sure to delete the workflow store means to delete the iw-store/workflow directory.

TS will recreate it upon startup.

For quite some time now Andy advocated a simple remedy for the root cause of your problem.

Make sure that you time-out every and all of your WFs External Tasks sending a mail to Application Admins.
You may for example transition to the user task with "Try again" / "Terminate" Options afterwards, stuff like that.
This will ensure that stalled (dead/not properly transitioned) External Tasks will not *hang* your WFs forever

dazzlad

Would you mind posting where 'Andy' has detailed the root cause of the problem as exhaustive searches on the error messages didn't reveal any answers on devnet....

Are you saying that workflow store corruption is caused by having too many external tasks that are hanging pending completion?

The support site article details a solution but no reasons as to why the problem occurs.

Thanks.

Untitled.png

nipper

Are you saying that workflow store corruption is caused by having too many external tasks that are hanging pending completion?

It is likely that you have external tasks that failed to complete for one reason or another. I harp on this often, people must put timeouts in their external tasks and if they fail to complete in a reasonable amount of time, transition to an admin task. I use 1 hour for timeouts.

I have seen cases where there are 10s of thousands of hung external tasks. That eventually will cause the system to fail.

dazzlad

Thanks for the info, I'll be sure to look at adding timeouts the tasks.

However what I am really looking for is something to link our error messages directly to the support site solution I posted earlier.

At the time of the time when our workflow external tasks stopped working we had around 100 jobs in the workflow store in various states.

We regularly purge jobs that have been inactive for long periods of time using a scheduled script, so we never have thousands of jobs running.

I do find it a little frustrating that IWOV post a solution but give no indication as to why this condition occurs.

Thanks.

nipper

Thanks for the info, I'll be sure to look at adding timeouts the tasks.

However what I am really looking for is something to link our error messages directly to the support site solution I posted earlier.

At the time of the time when our workflow external tasks stopped working we had around 100 jobs in the workflow store in various states.

We regularly purge jobs that have been inactive for long periods of time using a scheduled script, so we never have thousands of jobs running.

I do find it a little frustrating that IWOV post a solution but give no indication as to why this condition occurs.

Thanks.

It certainly *smells* like too many hung jobs. That can manifest itself in many ways. That being said, I do not understand why, if you are removing jobs, it is an issue, unless a job that is just killed doesn't clean up properly.

ISCBorisB

Would you mind posting where 'Andy' has detailed the root cause of the problem

Try search like the one illustrated below.

It is not necessarily a *corruption* that causes your issue. It is enough that external task does not transition properly.
That can be caused by any number of reasons like script design errors (termination bypassing transition call, cavalier use
of "die" command and the like), compilation errors, run-time errors, dead locks, you name it. In any such case WF resources
are consumed and when you kill WF they may not always get released completely or even correctly

dazzlad

Thanks for taking the time to provide me with all of this information.
It is very much appreciated.

As you'll undersrtand in order to spend some time reviewing our external task codes I'll need to put something in front of my boss in order to justify the time invested.

As with a lot of implementations I guess the workflows have evolved over time with many staff/contractors contributing to the codebase so it is hard for me to vouch for their quality.

ISCBorisB

Thanks for taking the time to provide me with all of this information.
It is very much appreciated.

As you'll undersrtand in order to spend some time reviewing our external task codes I'll need to put something in front of my boss in order to justify the time invested.

As with a lot of implementations I guess the workflows have evolved over time with many staff/contractors contributing to the codebase so it is hard for me to vouch for their quality.

Oh, I agree completely. I certainly did not advocate serious WF Code review, this is your decision to make.

Adding timeout clause to WF External tasks however is useful and often not that hard to do. This way even if you have coding
issues they at least do not accumulate to become one big WF System problem. Correctly implemented timeout notification will
also show you what scripts really are trouble makers.

dazzlad

Guess you'd suggest doing something with the TaskTimedOut event to monitor this?

nipper

Guess you'd suggest doing something with the TaskTimedOut event to monitor this?

No, you don't need to. In the WFT external task add a timeout transition (set it to an hour or so) and send that to an email task (to email you of the failure) and then an admin task. From that you should be able to go to each task that has a timeout (if you fix the issue) as well as a cancel task to cleanup, unlock, etc.

ISCBorisB

Guess you'd suggest doing something with the TaskTimedOut event to monitor this?

No I won't. Although possible it's an overkill.

Normally I would do just timeout transition to the mail task that informs Administrative group that so-and-so External Task
has timed out. What to do after that depends on your on-line support structure. You can simply transition to the *Clean/Terminate*
(shoot first and ask questions later) or transition to some group task (Support Group?) with options like "Try Again" or
"Terminate WF", this sort of things.