Workflow timeout during execution of server side script

Henrik Persson

Hello!

 

I have a problem when running a server side script that takes longer than 60 seconds in version 9.1.0.

Metastorm engine lets the function run to an end, in my case roughly 6 minutes, but because Metastorm times out (probably the MicrosoftWorkflowTimeout=60 in the registry) then the folder is rolled back to it's previous state.

 

I tried to make the timeouts longer but that did not work as it should because then my Timed action in the next stage is

messed up, this probably because of the long transaction caused running my server side script, which seems to block some internal stuff in Metastorm engine.

And personally I don't think that setting the timeout time to long is a good solution on my problem.

 

What I need is a way to make my lengthy server side script be executed asynchronous. I have tried to use delegates but

I runned into problem how to use the WS connection object and Mstm object. I don't know how to instantiate

those objects when the function is started using a delegate.

 

Does anyone have an idea how to solve the problem with executing a lengthy server side script within Metastorm?

 

I could write me my own external WS which could do all of the time consuming parts async but that will take some time

and I have a working solution in Metastorm server side C# for low volume calls, i.e. when all the WS/DB calls are executed

before timeout.

 

Best regards,

Henrik

 

Brian Mellert

Glad I'm not the only one to experience this!  I just haven't had time to chase it down further with the Help Desk due to competing priorities.  It used to be changing the execution timeout setting in the web.config took care of this in V7, but it doesn't seem to do so in V9.  I have a couple of system stage actions which take several minutes and need more than the short time out.

 

If you hear anything, please update the post.  Also, if I get time to get back to that and I hear anything I'll update as well.

Ethan Beisher

As a workaround, could you have the action raise a flag? This flag would create a new folder whose job is simply to execute the long running code. Have the folder auto-archive eventually.

Henrik Persson

I don't think that (the flag solution) will work because there will be the timeout in that case to because I think the MicrosoftWorkflowTimeOut is global.

I have tried to put the long time executing server side script on all different location OnActionAfter, OnStageStarted etc withgetting the same timeout problem.

 

To solve my emediate problem I did a loop in the processmap using conditional actions and did 500 calls every itteration until all calls were done and then let the folder continue. Not a very good solution faulthandling wise but it works for now.

 

Regards,

Henrik

Brian Mellert

A flag won't work for us either as its processing several records after getting feedback from users.  The "master folder" is trying to process the record.  The action could take 5-10 minutes under certain circumstances, but its being terminated by the engine sooner.

Tony Wheeler

Hi Henrick,

 

The problem you are running into with delegates is that once you start a new thread, you lose the current BPM Engine context. The Mstm object, ProcessContext object and "YourProcess" object all rely on that current BPM Engine context to figure out what you are allowed to do. Among other things, this context tells the engine who you are and what the current Folder ID is. If you try to use any of these objects in the new thread, you're likely getting null exceptions on the object.

 

You still however, have access to the compiled MBO types in the new thread, so you can explicitly pass in any business object that you may need in the new thread as an argument and have nice access to Intellisense. I haven't figured out how to keep access to the MSTM object though, so as long as you resist the temptation to use the Mstm object, you can execute your long running scripts asynchronously with read only access to all your ProcessContext and "YourProcess" objects.

 

The decision to use async processing shouldn’t be taken lightly... you can really hose your engine if you spawn too many threads or don't handle exceptions properly, but when you want to keep your UI responsive or you want to run expensive scripts, it’s a very powerful tool at your disposal. Async processing should not be used where anything on the current UI is reliant on the results on the Async work. In this case, you would want the thread blocked to ensure everything happens in time and in order.

 

The basic idea for async work is as follows:

 

using System;
using System.Data;
using System.Collections.Generic;
using Metastorm.Ide.Extensibility;
using Metastorm.Runtime.Core;
using Metastorm.Runtime.Types;
using System.ComponentModel;
using System.Threading;

namespace Metastorm.Runtime.Models.AsyncWorkerProj
{

  public class WorkDelegate
    {
        public void CreateNewThread()
        {
            // Spawn a new thread to do concurrent/parallel work.
            // This will not wait for a response, so the UI will be unblocked.
            // You will have no access to the current BPMEngine's thread so you must 
            // pass in desired business objects to the new thread as arguments

            ProcessContext pc = new ProcessContext();
            AsyncWorkerProc pd = new AsyncWorkerProc();

            Thread t = new Thread(() => DoAsyncWork(pc, pd));
            t.Start();      
        }

        private void DoAsyncWork(ProcessContext pc, AsyncWorkerProc pd) {

            // You do not have access to the current BPMEngine Session here (No Mstm!!)

            try         
            {
                // Do thread work here
                Thread.Sleep(30000);
                System.IO.File.WriteAllText(@"C:\Log.txt", pd.AsyncWorkerProcData.MyCustomVariable);            
            }

            catch(Exception ex)     
            {
                // Handle thread exceptions - DO NOT THROW
                System.IO.File.WriteAllText(@"C:\Log.txt", ex.ToString());
            }

            finally 
            {
                // Do thread cleanup
            }   
        }   
    }

    public class WorkThread
    {
        public static void DoWork()
            {
                WorkDelegate d = new WorkDelegate();
                d.CreateNewThread();
            }   
    }
}

Now in your solution you can do the async work anytime by calling WorkThread.DoWork().

 

Async models usually include the concept of a callback method that you can pass in to alert the UI thread that the async work has been completed. Because of the way the engine handles the current BPM context, the UI thread is already gone (the user is not even guaranteed to be logged in by the time the long-running script finishes), but we can alert the PROCESS that the async work is finished via a flag. We can’t raise this flag with Mstm object so instead you can either shell to the eRaiseFlag.exe utility or Process.Start() in the finally{} block to alert the PROCESS that the work is done.

 

See the attached solution.

 

Henrik Persson

Thanks for the extensive answer! I have not had the time to try out the example you attached but I get the general idea though.
Regards,
Henrik

Petio_Ivanov

I have found a similar issues with timeouts. Specifically Transaction Timeouts.

 

I had an Administration form which processed a batch of data in a looping block of code.Everything was going fine and after about 5 minutes I was getting weird errors and the following exception occurred.

"The transaction associated with the current connection has completed but has not been disposed."

 

Basically this was caused by a transaction timeout which is wrapped around the action on the server side script. Any attempts to access the database after 5 minutes resulted in an exception.

 

I found hints to the resolution on Page 17 of the 9.1 SR3 Release Notes:

http://communitycentral.metastorm.com/metastorm/attachments/metastorm/BPM9@tkb/57/1/Metastorm%20BPM%209.1%20SR3%20Release%20Notes.pdf

 

The registry setting

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Metastorm\e-

work\Engine\Database\QUERYTIMEOUT

Specified the 300 second time out for the transaction.

 

After increasing this to 1 hour, I realized that the problem still occurred after 10 Minutes this time.

 

And this is because of the Machine setting that the release notes talk about :

.NET Framework type System.Transactions.Transaction has a default

maximum timeout ( maxTimeout) value of 10 minutes and this will override

any user settings.

 

This setting does not actually override the user settings but "caps" them. Or in other words the system will use the lesser of the two values.

 

So the solution is to change the setting like so:

Add the following section to the bottom of machine.config file, just prior to the final closing tag.

 

 <system.transactions>
  
 </system.transactions>

 

The machine.config file in this case is located in C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config

 

Now with both settings at 1 hour my transactions were no longer timing out and aborting.