nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Workflows during backups

nipper

OK, so I had a strange one yesterday.

Backup occured 10:30-10:42 pm last night.

Workflow in process, steps are:

Approval -> DataDeploy (not DAS) from WA to Staging DB -> submit to Staging -> DD Staging area to Prod DB

DD log (staging) happened, files were not marked as modified so it looks like the submit happened,
but the WF hung in the DD to prod. No process was running so it died or did not run somehow.

I am perplexed as to how the submittask worked while the freeze was on.

Anyone see issues before with active WFs during backups ?

GOnna try to reproduce on my dev server.

Andy

Find more posts tagged with

Comments

Adam Stoller

It sounds like the server become froze after the submit but before the second DD.

This is where having error handlers built into your workflow come in handy - it allows you to send a callback for the active task (presumably the second DD task) and have it pop out to a user/group/cgi task from which the user can choose to transition back to the task that got hung.

If all of this is happening within a single externaltask script it becomes a bit trickier, as you need to write the code to allow re-entry and deal with the difference in status of the files and such.

The other possibility - which also requires the code be able to handle re-entry, is to just run the script from the command line with the proper arguments.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

nipper

> It sounds like the server become froze after the submit but before the second DD.

I forgot to mention that the logs for the 1st DD was at 10:38, so that happened while it
was frozen, and it APPEARS the submittask did as well (which I do not understand).

The prod deployment is the same as Stage except moving from Staging Area to Prod Table
while the other goes from WA to Stage Table

I will see if I can find places that perl would have dumped.

nipper

Adding a little more strangeness (if someone has the opportunity to reproduce I would appreciate it,
do not need to use DD, anything that can waste time)

I ran a WF that took the file, submitted, ran external task which ran DD, dumped output & ran Task->callback

I then froze the repository when the perl script started.

DD went and completed. The log file was made. At the end of my script I change perms on logfile (to verify
it ran) and call task->Callback

Perms were changed on the log, but according to the task list, that Job is stuck on the external task (which has
completed). No change after the lock was removed.

So, my question is: Does $task->CallBack work when the repository is frozen ? If not, all the error chacking in the
world will not help.

BTW, TS 552 SP4 Sol 2.8

Andy

Adam Stoller

Are you checking the return status from the CallBack?

($success, $immediatetask) = $task->CallBack(0, "message");

The perldoc doesn't specify what $success should be - but my guess would be that you could use code like:

if (!$success){
   # something went wrong
}
else {
   # we are okay
}

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

nipper

>Are you checking the return status from the CallBack?
>
>($success, $immediatetask) = $task->CallBack(0, "message");
>
>The perldoc doesn't specify what $success should be - but my guess would be that you could use code like:
>if (!$success){ # something went wrong}else { # we are okay}

No, but what then ? This is my last line in my wf & I cannot do jack to send an error back since it is sent
with CallBack

One thing I did notice is that when the submit task ran with the lock on the BS I received the Resolve Conflict
screen.

That was unexpected, but does explain some oddities I have seen in the system.

Andy

Adam Stoller

No, but what then ? This is my last line in my wf & I cannot do jack to send an error back since it is sent with CallBack
Send email and loop (with a decent sleep() value) through the script again, or perhaps part of it, if you can determine at what point it failed...

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

nipper

OK, but I really would like to hear IF task->callback is supposed to work when the repository is frozen,
I thought it would but obviously it does not.

Certainly makes WF much more complicated

Andy

Adam Stoller

Since the workflow store is within the backing store - I don't see why you would think that callback would work when the store is frozen?

It would be good if it logs the error somewhere (if it doesn't already?) though.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Migrateduser

It seems like the callback action (and probably tons of others) should just block and sit in the cache until the baconstore is thawed, or there should be some config for what do do in this case somewhere.

nipper

>It seems like the callback action (and probably tons of others) should just block and sit in the cache until the baconstore is thawed

Yea, cause this basically makes callback unusable as it is.

Unless I am the only person in devnet that just calls task->callback without checking return codes
and looping until it can complete.

I do know that IW consulting DOES NOT do that in thier IPL scripts.

Does IW tell you to do that in certification class ?

Andy

Migrateduser

I will never be certified, but I doubt backing up is part of certification. In general I am not sure that Interwoven PSO knows best practices in content management or even Interwoven implementations.

I do check the return code of callback and use enterprise exception reporting to raise it, but it would still take manual intervention. I am not sure every client writing their own loops around this is a good idea.

Something interesting here - why isn't the workflow in its own store? Does that mean that each store has it's own workflow substore, and all jobs created on branches in that store use that substore? Does that mean you can't have a job that has one task in one store and one task in another? Does something validate this when you create the job?

nipper

>Something interesting here - why isn't the workflow in its own store? Does that mean that each store has it's own workflow substore, and
> all jobs created on branches in that store use that substore? Does that mean you can't have a job that has one task in one store and one
>task in another? Does something validate this when you create the job?

Agreed

I understand the need to lock the repository from updates for backup. If TeamSite lost code or if a restore was invalid that would be
catestrophic. However, these are workflow events. Currently there are 2 choices (since I doubt ANY implementation of WF has
the abilty to run during a freeze): either those WF die a horrible death or you may lose the reporting of an approval or two if
the somehting is restored from backup. With no WF reporting, would you know ?

I find, every couple weeks, jobs that have hung for no apparent reason. I suspect this has caused a number of them. In my testing I
did notice if the submittask runs when the system is locked, you get the resolve conflict step. That is perfectly acceptable, there is
no way that step should go through the lock.

BTW, no clue on your other questions

Andy

nipper

>Since the workflow store is within the backing store - I don't see why you would think that callback would work when the store is frozen?
>
>It would be good if it logs the error somewhere (if it doesn't already?) though.

Anyway I received info from support on this:

task->CallBack SHOULD WAIT while the repository is locked and complete after iwfreeze has finished.

They confirmed it does not.

They have filed a bug (related to 552) # 52865 for this issue.

No clue if it works in 6.X, but I doubt it. Engineering did not appear to know about the problem until we filed
the bug.

(I really expected IW to say it works as specified & implement fish's suggestion)

Andy