External workflows hanging in Teamsite.

We have an issue with our workflow process and are hoping someone else has experience with it. We have an open support ticket number is 1214954 currently 12 days old on our production system.

The Issue:
We are able to create a dcr but when submitting to staging the external workflow hangs. This was working; data was being sent to the database and files were being being deployed to the proper locations. This process then stopped working, we have verified that the files involved have not been altered and there have been no changes to the environement.

Our Process:
The direct submit external workflow itself calls the perl deployment script, which calls the Opendeploy deployment script which calls the datadeploy deployment script. This process again worked for two weeks without issue and then the the external workflow hung in TeamSite there are no errors generated and nothing appears (job number) is displayed in the workflow tab located in the professional view of TeamSite.

It is interesting becuase the file submited as part of the workflow does end up in staging, but the deployment piece does not appear to run.
We cant tell exactly where it hangs... But the submit to staging does occur and the deployments do not run at all (hence there are no OD logs for the deployments...)

We have another workflow, the standard submit to staging from a workarea, there are no issues with this workflow. Our users are able to submit content and all other scheduled daily deployments are running sucessfully.

In the process of trying to manually cancel all hung workflow tasks, TeamSite itself hung and went down and had to be restarted.

While working with support we removed the workflow directory from Backing Store and restarted TeamSite but it had no effect resolving this issue.

Our Platform:
Solaris 8
TeamSite version 6.1 SP 1
Open & Data Deploy 6.0

There is nothing unusual running in the background, no big batch jobs or deployments. We have a small by most company's standards TeamSite implementation. We would really appreciate any help that can help to remedy this issue.

thks .c & .e

We have provided thus far to Interwoven support copies of the below files listed:
debug view of workflow
direct_submit_deploy.wft
available_templates.cfg
od_deploy.ipl

iwtrace log
iwjoberrors log
iwevents log

<iwhome>/local/logs/iwui * (please get all logs in this directory (full logs).
<iw-home>/local/logs/* (please send us the latest logs out of this directory.
<iw-home>/iw-webd/logs/* please get all logs in this directory (full logs).
<iw-home>/servletd/logs/*

Find more posts tagged with

Comments

nipper

Most likely it is in the Perl script. Either it is dying in the middle, or returning a bad value.

Debugging this is a pain.

Open a tmp file, start putting logging statements in the perl and see if you can figure out where it is dying.

$10 says somebody made a change recently.

Post the pl. When was it modified last ?

HTH

Andy

Migrateduser

Thanks Andy - I'll try that. We only have two developers myself and our sys admin. But just in case do you want that in cash?
thks c

iwovGraduate

If I understand correctly, your externaltask with the command
/iw-home/TeamSite/iw-perl/bin/iwperl /iw-home/TeamSite/local/bin/od_deploy.ipl
hangs. You can easily debug this by running it from the command line:

/iw-home/TeamSite/iw-perl/bin/iwperl /iw-home/TeamSite/local/bin/od_deploy.ipl <job_id> <task_id> <workarea_vpath>

Do that and post the output along with your od_deploy.ipl file.

Migrateduser

Apparently this was a system issue, some type of memory leak - not a problem with workflow.
I've posted the fix here just in case it happens to someone else.

1. Stop Teamsite (/etc/init.d/iw.server stop)
2. Remove /iw-home/TeamSite/tmp/iwprocessd.arena file
3. Remove /iw-home/iw-store/workflow directory (be very careful, only this directory)
4. Start Teamsite (will probably have to reboot to get teamsite back up...)

We were told this could happen again and if so follow the above steps again.

- c

Bowker

We had a simiilar issue (more than once) that was caused by each of the following:
1) The receiver machine ran out of disk space (duh)
2) The number of temporary files for filelist deploys stored in the temp folder got to many many many thousands of files.
3) server machine ran out of disk space for the log files

Just a few other possible things that could cause issues without any changes to the code.

Dan Bowker
Northern Trust
Web Publishing Technology

Johnny

I just ran into this very problem.

It's a brand new install..

no diskspace problems...
no inode problems...

any other clues as to what could have happened?..

Could it be as simple as a file corruption? (arena)?

John Cuiuli

kajjubee

I am having the same problem. TeamSite 6.5 solaris 9. If I run the task from the command line, the task works, transitions to the next task and the next external task is hung again. This used to work and suddenly is behaving like this wierdly. Antonson, doesn't the fix u provided ( from Interwoven, 4 steps) affect the current jobs?

Thanks
K

Edited by kajjubee on 03/28/05 02:44 PM (server time).

kajjubee

Restarting of TeamSite fixed this issue. But the annoying part is that I had to resubmit the hung jobs. This has happened to me twice so far... I think this has happened to me since the 6.5 upgrade with search.

harrij

We are experiencing same problem with 6.1SP1 on Solaris 2.8. Our case id is 1217275.

In our case the removal of iwprocessd.arena and empyting the workflow directory did not help. Only thing that has helped so far is disabling the iwprocessd daemon by setting use_process_daemon=no in iw.cfg [iwserver] section but that has other negative effects on the performance.

The problem is in the iwprocessd which seems to die for no apparent reason soon after iwserver is started.

The problem started after we moved the server to a different location which required shutting down the server. After reboot the problems started.