nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Workflow status during iwfreeze

Mashuk

Hi,

Can someone tell me what happens to a workflow when an iwfreeze takes place. We are thinking about implementing an incremental backup strategy throughout the course of the day. Therefore during those times, we will need to issue an iwfreeze to freeze the content store. What would happen to a workflow that was executing an external perl task when an iwfreeze is issued. Would it finish the perl task and then stop, or would it stop mid session and fall over, or would you have to re-execute the perl task on the command line or would it get corrupted?

Is any elegant way to tell a workflow that an iwfreeze has happened, and the workflow should wait until the backing store is unfrozen before continuing?

We are on TeamSite 6.5, Solaris

Thanks in advance
Mashuk

Find more posts tagged with

Comments

Adam Stoller

Good question!

I believe the only way to do this is to have additional checking peformed within your externaltask script if/when a failure occurs - you can parse the output of iwstat to determine if the store is frozen, and if so sleep for a bit and re-check (continue until store is no longer frozen) and finally go back to the function that failed in the first place because the store was frozen.

From a high-level this is not too difficult.
From a low-level this will take a considerable about of design effort to ensure you're handling all the situations correctly, especially issues with re-entry into a subroutine that failed out because of the freezing.

I remember starting to think about this several years ago, but I don't believe we ever got the go-ahead to invest the time in developing a robust solution then.

Mashuk

So if I didn't do anything in the workflows to programatically handle the iwfreeze, would the workflows just hang, even if the backing store was returned to its unfrozen state.

I have another question on backing stores.

Is it possible to run the iwfsck command in parallel. As an example say I have 2 websites A and B and each of those websites have their own backing stores. As part of a recovery process would it be possible to run the iwfsck process on each backing store simultaneously, or does this have to run sequentially.

Thanks
Mashuk

Adam Stoller

So if I didn't do anything in the workflows to programatically handle the iwfreeze, would the workflows just hang, even if the backing store was returned to its unfrozen state.

I'm not sure if the workflow would hang, I think it depends on the state of the workflow at the time the freeze occurs - if it's in an externaltask (or perhaps a cgitask, maybe even a dummytask) the behavior is probably entirely dependent upon the developer of the customization.

In the case of an externaltask - if you have multiple callbacks (at least one for failure) and your code doesn't do anything to determine the root-cause of the failure (i.e. doesn't check for frozen state) - then it is most likely that the workflow will be set to transition to the failure state as soon as it is able ... which brings up another question [IWOV?]:

What happens with task transitions that occur just prior - or during [?] the freezing of a backing store? Do the transitions take place? Do the transitions get queued? or do the transitions get lost?

If it's the last of those - then you might be SOL here.

I have another question on backing stores.

Is it possible to run the iwfsck command in parallel. As an example say I have 2 websites A and B and each of those websites have their own backing stores. As part of a recovery process would it be possible to run the iwfsck process on each backing store simultaneously, or does this have to run sequentially.

I don't know the answer - but I suggest you ask such a distinct question in its own thread rather than confusing the subject matter for this thread.

Mashuk

So if I don't handle iwfreeze conditions, then the workflow will become unpredictable. This will help in understanding and then outlining our backup strategy.

Thanks for your help.

Mashuk

jbonifaci

What happens with task transitions that occur just prior - or during [?] the freezing of a backing store? Do the transitions take place? Do the transitions get queued? or do the transitions get lost?

It's the last, your external task will try to transition the workflow, but because the store is frozen, the transition will error and be forgotten about. Unless you do something like Adam is suggesting and somehow handle the transitions taking place after the store is unfrozen, your workflows will be "stuck" in the external task (assuming you don't have a timeout value).

That being said, you'll likely want to try to design your workflows so that the tasks don't activate during your backup hours.

~Jeff

kaalraa

As per our concept, what we have applied is somewhat different instead of playing around with if and buts clauses of iwfreeze.

What we have done... is simply add one more task after failure notification of a task. The task is named as Reactivate_Last_Failed. After each task which has a probability of failure we just pass the task name to a pre-defined variable called $returnToTask. The Reactive Task is a CGI task which just reads on the variable value and resets that particular task in case of need after seeing the reasons of failure in logs.

In my opinion this is an easy way out... instead of playing with iwfreeze. Rest its you and your application code Smiley Happy

jbonifaci

After each task which has a probability of failure ...

The best option is to design your workflow to eliminate that probability of failure. There should definitely be error handling in workflows as well, but ideally your workflow never gets into your error handling routines.

Adam Stoller

jbonifaci - you quoted me but attributed the text to Mashuk - however, thanks for answering the question (which is what I thought the answer was, but couldn't remember and didn't have a system to verify it on)

kaalraa - your solution is a non-solution based on Jeff's information (which I'm assuming to be accurate) - because your assuming that the externaltas failed because of the iwfreeze but also managed to succeed in transitioning to your error handling task; which wouldn't happen if the store is frozen.

I agree that having error handling tasks is important in the design of workflows (did a presentation at GearUp last year that included that) and I still believe you need to code your scripts to be as robust as possible within whatever constraints you have (sometimes you don't have sufficient time to test them in all circumstances - such is life)

However it seems that the "solution" for this scenario is probably a cron-job or perhaps administrative custom menu item script which looks through all active tasks and attempts to determine which ones are potentially in a hung state because they were unable to complete their business and/or unable to transition because of service interruption (frozen store, power outage, etc.) - and then provide the option to transition the job to where you want [in this case - most likely to your error-handling task(s) from where you can then cause the workflow to transition back to the task that was hung]

kaalraa

Thanks Adam, for all your suggestions and advise. Also it gives me great push to learn more about the administrative areas of TeamSite as till now i am more leaning towards Development.

Thanks again.

jbonifaci

jbonifaci - you quoted me but attributed the text to Mashuk - however, thanks for answering the question (which is what I thought the answer was, but couldn't remember and didn't have a system to verify it on)

Hmm, I was able to go back and edit my post still and change it to you. Wonder what the edit time limitations are in the new software.

As far as the info, not a big deal, my company has a freeze every night for incremental backups and I have had several late nights where I'm trying to debug what is wrong with a workflow until I look at the clock, Smiley Wink

Mashuk

Is the strategy of having small incremental backups during the course of the day a recommended one? Do any of you, have such a strategy? If so what are the nasties to look out for in the workflow other than what was already mentioned?

Thanks
Mashuk

mstradling

It depends on the usage patterns of the TeamSite server, but in general I'd not recommend running backups during the day.

In most situations a TeamSite install will only service a geographically local user group and that user group will work office hours. OK, some might start early and others work really late, but how much use do you predict between 2am and 5am?

I can see a number of issues with regular backups through the day - how will your users know if the system is working or not? Who's going to answer the phone when the users who's spent an hour writing content in to a DCR clicks save and looses it all because the server has frozen between opening the DCR and saving it?

I'd generally predict more users and more workflows to be effected by a freeze during the day than over-night.

Having run a TeamSite system that the business idealy wanted up and in use 24/7, a nightly backup between 3am and 6am was by far the least disruptive. There was the least to write about, and the least users awake to read it at these times.

jbonifaci

You definitely don't want to be doing backups of the backing store in the middle of the day. As the previous poster pointed out, having the store down during business hours is unacceptable. Backups of the backing store should be done during non business hours, as the previous poster mentioned. Although, I would be more inclined to have the backup window around 12-4am.

If you feel you absolutely need to take backups in the middle of the day, you could do incremental backups of the mounted backing store (usually your Y: drive). This would not require an iwfreeze, but could still impact performance. You would still want to take nightly incremental backups of the frozen backing store, but this would allow you to recover files from the current day if anything went amiss. I feel this is not needed and nightly backups are all you need, but if your client insists on backups throughout the day, this would be an option. Although, I'll just stress again that I wouldn't recommend it.

~Jeff

Mashuk

The intention is to carry out incremental backups during the day and do a full backup at night, so that if there was a scenario where the Teamsite server went down or failed, it would be easier to revert to the disaster recovery server. Im no back up expert but our expectation is that we could possibly do incremental backups on various branches which would mean teamsite would only be down for a few minutes at a time.

I can see Mark's scenario of where a user has been editing a dcr for an hour and then when they come to save, they can't. But surely when the backing store is unfrozen, they can save the dcr, if it is still open???

Thanks
Mashuk

jbonifaci

They could, that's true, but IMO it is unacceptable to freeze your store during business hours. This is an extremely bad design.

As far as doing incremental during the day and full at night, this seems like overkill to me. Incremental at night and full over the weekend is all you need and will have the least impact on the user.

~Jeff