Number of files limit?

I have a file list deployment defined that is called from a workarea. It fails when there are a large number of files deployed. The latest failure was for a job with about 1100 files to deploy. I've had similar failures with other workflows.

I get multiple (100's) of entries as follows in the iwjoberrors.log:
"[Tue Jun 16 10:38:01 2009] Could not run \Interwoven\TeamSite/iw-perl/bin/iwperl \Interwoven\TeamSite\local\config/wft/LowesNG/wft_opendeploy.ipl for task 1512232; retrying in 1 minute"

Other tasks and jobs complete without error using the same workflows and ipl's but with less files.

Are there any actions that I can take to keep large file list deployments from failing?

Thanks.

Find more posts tagged with

Comments

nipper

How is this running ? As a cgitask or an external task ?

If it runs as a cgi, then you are limited to the 5 minute timeout for apache/IIS

If it is an external task then there may be a timeout associated with it.

SethHickel

If it is an external task then there may be a timeout associated with it.

We are using an external task. Any guidance on where such a timeout setting may be? Is it written in the ipl itself?

ISCBorisB

We are using an external task. Any guidance on where such a timeout setting may be? Is it written in the ipl itself?

Take a look at [post=132526]this[/post] post. If that's your case you can set external_task_add_filelist=false in iw.cfg.
Note that this change is global, all WFs may be affected

SethHickel

By default WF Engine passes attached Files as External Task Command Line Arguments. There are therefore two limitations, both Platform/OS
dependent: "Too many Arguments" and "Maximum Command length exceeded".

Note that 'external_task_add_filelist' setting is global. If you change it, all existing workflows may be affected.
You'd have to make sure that TeamSite::WFtask::GetFiles method is used to obtain Attached Files List wherever needed.

I will try adding external_task_add_filelist=false to my iw.cfg after confirming that all my workflows calling an external task make use of the TeamSite::WFtask::GetFiles method.

Thank you.

SethHickel

Still working on getting dev setup to test the change to iw.cfg - BUT - we are also failing on large files - I have a file today that is 2G that the business needs to push. I know that a 40mb file has taken 17 minutes in the past. Any words of wisdom on what to look at for timeout occuring because of file size too?

nipper

depends on the OS. Many OS filesystems (still) have a 2 GB limit.

Adam Stoller

If you're using the more recent versions of OD, you can do incremental byte-differential deployments which, while they won't help too much on the first deployment, should make subsequent modification deployments faster.

Also, you can look into using compressed deployments

Also - consider using non-transactional deployments for such large files - as you need to allow for 3 times the size of the files in a transactional deployment so a 2Gb file requires 6Gb free space on the receiver.

sniv

Hey,

Seth..First thing do please mentioned the OD version and platform with thread..it's more important part....

Anyways.

Check with iw-perl/bin/ directory.

You may see a 0 KB file named iwperl apart from iwperl.exe.

Sometimes, a dummy 0 KB file named iwperl is created in folder iw-perl/bin/. This causes the external task to hang.

Do remove this file and run deployment else use iwperl.exe in command instead of only iwperl.

Regards!
Vinayak

SethHickel

need to allow for 3 times the size of the files in a transactional deployment

Going to review the documentation - but - what is all this space used for? Is it temporary space used during comparison or what? And would it be safe to say that if we did not use transactional deployments for large files that the throughput and/or time it takes to deploy could be improved?

With respect to deployments with a large number of files: would you also suggest avoiding transactional deployments?

We tried a deployment last night off-peak hours of a 44mb flash file. We got an average transfer rate of 554 KB/s. Using the same deployments during peak hours we averaged from 78 kb/s to 497 kb/s. This was the range for the entire peak business hours.

Our next test will involve multiple files (around 1000) to see what type of transfer rate we get at peak and off peak hours.

Also, you can look into using compressed deployments

This sounds interesting. I've not considered this. So - we would accomplish this by externally compressing the file(s) and then deploying the compressed file to the correct area on the receiver file system and then call some utility to uncompress? Or - is there some method that uses TeamSite/OD methods?

please mentione the OD version and platform

TeamSite 6.7.1 (Not SP1)
Open Deploy 6.1.1

ISCBorisB

What is all this space used for?

This space is used to maintain roll-back information for aborted transaction. While transaction is in progress there are
three maintained "states" of each deployed File: existing, "old" (copy of existing) and changing "new".

would you also suggest avoiding transactional deployments

Non-transactional Deployments are faster and require less resources. What you sacrifice is referential integrity.
Some Files may go, some not etc... If that's Ok - sure, use non Transactionals.

is there some method...

Compression setting belongs to transfer Rules, you just indicate that you want it, the rest is automatic on
both sides. You gain in bandwidth and pay for it by CPU cycles. If I were you - I would not even try it for single
2GB File though

SethHickel

If I were you - I would not even try it for single
2GB File though

I'm not sure I understand your comment. Is it the fact that it is only one file or is it the size (too small or too big?)

Thank you for your responses - very helpful.

ISCBorisB

I'm not sure I understand your comment. Is it the fact that it is only one file or is it the size (too small or too big?)

Thank you for your responses - very helpful.

Do not use compression for the single 2GB File. It is way too big.

You will have 100% CPU Usage on both sides for quite some time and I doubt if you will gain much on transfer.
Standard winzip is a good benchmarking tool. Can you zip your 2GB at all? How much time does it take? How much
you gain in compressed size?

Do use non-transactional Deployment to transfer one huge file! After all, if it fails it fails. In the "single File Transactional"
scenario you have all the disadvantages of Transactional Deploy without any benefit