Active Jobs limitation

Dear All,

I am just curious. Today we had a major performance problem with our TeamSite server. We were not able to pin point the root cause as yet. During our investigation we found out that there are around 35,000 active jobs stuck in the system. Once we started killing the jobs the performance started to become normal.

My questions are:

-- Is there a limitation on number of active jobs?
-- Does higher number of active jobs create performance degradation?

We are running TS 6.5 SP2 on Win 2003.

Thanks in advance for your response.

Find more posts tagged with

Comments

nipper

Obviously the answer to your second question is yes.

There is no physical limit on the # of jobs, but jobs do consume resources since they are stored in the backingstore. I have seen implementations with a few thousand hung jobs, NEVER near 35K

Your larger issue is why you had so many jobs.

From my experience, it is due to 1 of 3 reasons (or a combination of the 3), files being resubmitted may times, external tasks stuck, or review tasks never taken.

The 1st 2 are the most common. All of my WFs have 2 features to help this. 1st, a file is not allowed in a WF if it is already in one. It is a bad practice to allow it. You will also find problems early. Second, any external task (deployment, email, anything) has a timeout. If the timeout is reached, an admin task is run, which means I get notified and actually fix the issue instead of letting it sit there. My timeouts are a couple minutes for emails and an hour for generateHTML and deploy. But you can use a day for them and you will still avoid the problems. Of course you have to actually research to find the reason a task failed and then figure out how to fix it.

HTH

Andy

phd

Dear Andy,

Completely agree with what you are saying, but the thing that gets me thinking is that these jobs didn't get created in a day. I mean, we had some jobs stuck for 900 days. Why would the performance degradation suddenly happen today.

Also we have still not done a complete clean of these stuck jobs. Ther are still 29,000 stuck and the performance is back to normal already or I can say even better than before.

Regards
Pawan

nipper

Dear Andy,

Completely agree with what you are saying, but the thing that gets me thinking is that these jobs didn't get created in a day. I mean, we had some jobs stuck for 900 days. Why would the performance degradation suddenly happen today.

Also we have still not done a complete clean of these stuck jobs. Ther are still 29,000 stuck and the performance is back to normal already or I can say even better than before.

Regards
Pawan

Then some of those jobs were doing something or consuming system resources. Did you remove old or new ones ? It is possible that you freed up enough resources. to make the system workable again. In Unix you can run out of i-nodes, which is hard to determine unless you know what you are looking for. Maybe the same thing happened. Regardless, that is a problem, since you have had jobs hung for 3 years, you need to fix the issue but also prevent that from happening again.

ISCBorisB

From my experience, it is due to 1 of 3 reasons (or a combination of the 3), files being resubmitted may times, external tasks stuck, or review tasks never taken.

I would like to add two more items to this short list:

4) Ignored "Submission Conflict" in case of OOTB Submission WF and multiple Workareas. Multiple Submission WFs end up
waiting for conflict resolution that never comes.

5) Cavalier use of lock="t" task attribute instead of a dedicated locktask. That one can be especially nasty with the
start task where WF does not show up and the user may try to initiate it over and over again. Btw, timeout in this
case does not help either (since task never starts, timeout event is not registered)

nipper

I would like to add two more items to this short list:

4) Ignored "Submission Conflict" in case of OOTB Submission WF and multiple Workareas. Multiple Submission WFs end up
waiting for conflict resolution that never comes.

5) Cavalier use of lock="t" task attribute instead of a dedicated locktask. That one can be especially nasty with the
start task where WF does not show up and the user may try to initiate it over and over again. Btw, timeout in this
case does not help either (since task never starts, timeout event is not registered)

True, but either of those should get caught in the "don't allow a file in multiple jobs at once " check at the beginning of the WF.

ISCBorisB

True, but either of those should get caught in the "don't allow a file in multiple jobs at once " check at the beginning of the WF.

I was just trying to add to the list of possible WF accumulation reasons if no countermeasures are taken - the one you've started

Tbag

There's also no guarantee that a linear increase in the number of workflows corresponds to a linear decrease in performance. It could be more of a step function in terms of impacting performance.