Submit process in large branch

Hi,
we have just migrated our TS architecture to the version 5.5.2 and we are exeriancing a problem during submitting of files in the Staging Area using workflow (we can see the submitFSE operation using iwstat command).
In the submitting task we use the iwsubmit.exe -r -w command and sometimes it takes more than 800 seconds to complete the process.
We also tried to launch the command in a dos shell but the problem is still the same.
During this operation all the other requests are queued and the system seem to "freeze".
When another user try to start another submit sometimes he gets an error reading metadata for a datadeploy process, but retrying the operation everything works fine (seem that the process can't access the file system-metadata information stored in the backing store).
During this freezing periods we also tried to browse the Y drive on the server but it wasn't possible until all the queued process are completed.
In our templating/xx/data folder we store more than 5000 dcr's (in ts 4.2.1 wasn't possible to split dcr in subfolders) and our customer told us that he cant change the way to store the content.
I saw in the knowledge base an article that talks about a limitation in number of files stored in a folder (max 512) but in our customization we have other folders (images) that keeps more than 5000 files but it doesn't face the same problem during submit processes.
Could someone help me ??

Find more posts tagged with

Comments

Adam Stoller

Having more than 1000 files in any directory (within Team or outside TeamSite) is likely to cause performance problems as directory lookups need to parse through all those files.

You say the problem was during a submit - how many of the mass number of files actually needed to be submitted? If more than 100, I'd recommend using the Submit Direct option versus using the Submit [Workflow] option.

Now that you're in 5.5.2 I'd recommend planning (and implementing) a process to subdivide your data directories to more manageable sizes - this will help in both the short- and long-run.

I'd recommend considering doing the same for your images directory too - as even if you don't see problems there now you're generally "asking for trouble" because file system operations (again, not just in TeamSite) will tend to degrade with directories containing such a large number of files.

--fish
(Interwoven Senior Technical Consultant)

boninba

Fish,
first of all thank you very much for your very quick answer.
The problem is that we didn't face this issue in ts 4.2.1 and we didn't find in the documentation any entry concerning the limitation of number of files managed by TS.
We are developing a work around to bypass this issue but it won't be a good solution, technically speaking (submit operation of the filelist created at the end of the workflow processes).

msrich

W2K TS5.5.2 SP2B

We're seeing iwsubmit take progressively longer in a batch sumibt from a directory containing just 300 files. Would you take a glance at the log attached and give me your opinion? Thanks.

Adam Stoller

I don't have time to look over logs right now - if you are having significant performance issues I suggest contacting either Interwoven Support or Interwoven Consulting for assistance (unless someone else out here can look through your logs and manages to see something of relevance)

--fish
(Interwoven Senior Technical Consultant)