nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

IWAT CreateFSE Problems With Importing Large Files

Bjorn

I have an iwat trigger set up for CreateFSE. I'm using it on a certain branch to run a Java class that sets certain EAs when a file is imported. I know the iwat is triggering fine because, well, it works. But only with smaller files.

It seems like the problem is that the local file manager applet creates a dummy placeholder file or something as the file is being uploaded by the import functionality, which it renames to be the "real" file once the import is done. I can get around the naming issue because it seems like the naming is done consistently, like if I am importing:

/TeamSite-1/main/branch/SupportingFiles/WORKAREA/content/documents/ibatis.pdf
the file is first created as
/TeamSite-1/main/branch/SupportingFiles/WORKAREA/content/zz_lfm_documents_ibatis.pdf

You can check this viewing the file path sent by the trigger when the CreateFSE event happens as a result of an import.

With a smaller file, this is no big deal because it uploads quickly enough where this initial file creation and renaming happens so fast it doesn't impact anything.

But with a larger file, the trigger runs the class before the file is done importing (because the initial file ""zz_lfm_..." is created), so the "real" file doesn't exist yet even as the class that was triggered by the CreateFSE event is finished running.

Anyone have any experience with the situation with which I am dealing?

Find more posts tagged with

Comments

Adam Stoller

I have an iwat trigger set up for CreateFSE. I'm using it on a certain branch to run a Java class that sets certain EAs when a file is imported. I know the iwat is triggering fine because, well, it works. But only with smaller files.

It seems like the problem is that the local file manager applet creates a dummy placeholder file or something as the file is being uploaded by the import functionality, which it renames to be the "real" file once the import is done. I can get around the naming issue because it seems like the naming is done consistently, like if I am importing:

/TeamSite-1/main/branch/SupportingFiles/WORKAREA/content/documents/ibatis.pdf
the file is first created as
/TeamSite-1/main/branch/SupportingFiles/WORKAREA/content/zz_lfm_documents_ibatis.pdf

You can check this viewing the file path sent by the trigger when the CreateFSE event happens as a result of an import.

With a smaller file, this is no big deal because it uploads quickly enough where this initial file creation and renaming happens so fast it doesn't impact anything.

But with a larger file, the trigger runs the class before the file is done importing (because the initial file ""zz_lfm_..." is created), so the "real" file doesn't exist yet even as the class that was triggered by the CreateFSE event is finished running.

Anyone have any experience with the situation with which I am dealing?

I haven't dealt with this specific problem - but you seem to have identified the situation without coming up with the (IMO) natural fix: have your script loop until the zz_* file is gone / the expected file exists.

Or did you try this and it didn't work?

Bjorn

I haven't dealt with this specific problem - but you seem to have identified the situation without coming up with the (IMO) natural fix: have your script loop until the zz_* file is gone / the expected file exists.

Or did you try this and it didn't work?

Thanks for the response. That was my first approach and it didn't work for a couple reasons. Namely because it's not just the zz_* business that gets in my way, but because the "temp" file is apparently created at the root of the workarea instead of the actual dir where it will be imported and the directory names are prepended to the filename like "zz_lfm_dir1_dir2_actualfilename". So it makes it tricky to create something that is "extensible" enough because I don't know the "real" filename other than what I am parsing out from the area relative path arg I've gotten from the trigger. I may know when the zz_lfm_ **** is gone, but it's trickier than at first glance to find out the real path to the file.

Like what if a file was imported that had both underscores and the same names as a directory structure? I could test for the existence of the file and it might be a false positive, depending on the dir structure. I could provide an example of this happening, but I've already cut bait and don't want to waste either of our time.

What I did was ended up adding a trigger based on the 'Lock' event, which the LFM does at the end of its import action and then added logic based on the default comment which the file import always adds. A semi-kludge, but c'est la vie. Good enough sometimes has to be the best.

Thanks again for your response.

Adam Stoller

What I did was ended up adding a trigger based on the 'Lock' event, which the LFM does at the end of its import action and then added logic based on the default comment which the file import always adds. A semi-kludge, but c'est la vie. Good enough sometimes has to be the best.

Thanks again for your response.

Another possibility would be something like this (untested code):

...
my $count = 0;
while (! -f $filename && -M $filename > 0.1) {
   # log delay?
   sleep 1;
   $count++;
   if ($count > $MAX_TRIES) { # prevent endless loop
       # log abort
       die "$0: Exceeded $MAX_TRIES";
   }
}
...

The idea is to loop until you see the expected file in place AND that it's modification time is less than 0.1 days (2.4 hours) old (feel free to change that to something even less). You could also verify that the file has size > 0 though I suppose someone might want to import a zero-length file for some reason?

It's still a bit kludgy, but probably a bit more efficient than checking for the comment...

TroyP

I have an iwat trigger set up for CreateFSE. I'm using it on a certain branch to run a Java class that sets certain EAs when a file is imported. I know the iwat is triggering fine because, well, it works. But only with smaller files.

I wanted to do something similar using iwatcreate. When my CreateFSE events were not being logged I contacted support and they said,

Solution: ModifyFSE logs an event whenever a file is modified for the first time after a submit. There is no event for each time a file is modified. The reason for this is that you'll end up with too many events for modifications through the y drive, since each file save generates multiple writes.

Is there a chance that you are submitting these files which may take longer for larger files?

Adam Stoller

I wanted to do something similar using iwatcreate. When my CreateFSE events were not being logged I contacted support and they said,

Is there a chance that you are submitting these files which may take longer for larger files?

The thread pretty clearly states that the process is running on Import (not Submit) via the CreateFSE (not ModifyFSE) event. So, I don't think the above is particularly on-target for this thread (useful information though it may be).

Bjorn

Another possibility would be something like this (untested code):
...
my $count = 0;
while (! -f $filename && -M $filename > 0.1) {
   # log delay?
   sleep 1;
   $count++;
   if ($count > $MAX_TRIES) { # prevent endless loop
       # log abort
       die "$0: Exceeded $MAX_TRIES";
   }
}
...
The idea is to loop until you see the expected file in place AND that it's modification time is less than 0.1 days (2.4 hours) old (feel free to change that to something even less). You could also verify that the file has size > 0 though I suppose someone might want to import a zero-length file for some reason?

It's still a bit kludgy, but probably a bit more efficient than checking for the comment...

The lock approach was indeed inefficient. After going back to the drawing table, I took a hybrid approach based on what I was doing and some of your suggestions. Here's an overview

1) Wait until zz_lfm_* temp file DOES NOT exist. This needs to be done so that when I convert the silly zz_lfm_* path into a real path in step 2, I know for sure that the file is done uploading/importing, and I can trust logic checking for its existence.

2) Parse zz_lfm_* temp file to obtain path of real file (trickier than you might think). This involves removing the "zz_lfm_" from the path and then converting the prepended directories on the original filename (dir1_dir2_*) to be actual directories in a path (dir1/dir2). So if I imported a file "e_f_g" into the "d" directory in the path "a/b/c/d", the real path I get from the trigger is "zz_lfm_a_b_c_d_e_f_g" and I convert to the real path of a/b/c/d/e_f_g.

3) Finally do the stuff I need to do to the actual imported file.

Adam Stoller

The lock approach was indeed inefficient. After going back to the drawing table, I took a hybrid approach based on what I was doing and some of your suggestions. Here's an overview

1) Wait until zz_lfm_* temp file DOES NOT exist. This needs to be done so that when I convert the silly zz_lfm_* path into a real path in step 2, I know for sure that the file is done uploading/importing, and I can trust logic checking for its existence.

2) Parse zz_lfm_* temp file to obtain path of real file (trickier than you might think). This involves removing the "zz_lfm_" from the path and then converting the prepended directories on the original filename (dir1_dir2_*) to be actual directories in a path (dir1/dir2). So if I imported a file "e_f_g" into the "d" directory in the path "a/b/c/d", the real path I get from the trigger is "zz_lfm_a_b_c_d_e_f_g" and I convert to the real path of a/b/c/d/e_f_g.

3) Finally do the stuff I need to do to the actual imported file.

I'm a bit confused by the above - what value gets passed in ARGV to the script from the event trigger?

\n";}close($fh);...If you do this, and you still can't figure it out - please post the output from the above.

Bjorn

None of the above. And technically nothing gets passed into ARGV because I'm using Java. Smiley Happy

This is what gets passed as args if I import the file "test_file.pdf" to the path "\TeamSite-1\main\branchname1\subbranch\WORKAREA\content\documents\craptastic"

- Timestamp like: [Wed May 06 09:47:02 2009]
- User like: DOMAIN\id
- Role like: master
- Event like: CreateFSE
- VPath to WA like: \TeamSite-1\main\branchname1\subbranch\WORKAREA\content
- path to file in workarea like: \zz_lfm_documents_craptastic_test_file.pdf

The key is the last one, the "area relative path". If you look at it, you'll notice it's NOT simply weird because of the zz_lfm_ -- it's weird because "documents_craptastic_" is not actually part of the filename, but rather the directory path in which the file is being imported. Obviously this doesn't happen if you just create a new file in a workarea -- only if you're importing a new file. I don't know why the Local File Manager applet interacts with TeamSite like that, and it's definitely not how I as an armchair programmer would have done it.

This sort of prepending of the directory path to the filename is what creates the problem because it makes it tough to determine the real filename, especially if the filename has underscores in it.

Fortunately, the solution I posted yesterday is working well.

Adam Stoller

None of the above. And technically nothing gets passed into ARGV because I'm using Java.

This is what gets passed as args if I import the file "test_file.pdf" to the path "\TeamSite-1\main\branchname1\subbranch\WORKAREA\content\documents\craptastic"

- Timestamp like: [Wed May 06 09:47:02 2009]
- User like: DOMAIN\id
- Role like: master
- Event like: CreateFSE
- VPath to WA like: \TeamSite-1\main\branchname1\subbranch\WORKAREA\content
- path to file in workarea like: \zz_lfm_documents_craptastic_test_file.pdf

The key is the last one, the "area relative path". If you look at it, you'll notice it's NOT simply weird because of the zz_lfm_ -- it's weird because "documents_craptastic_" is not actually part of the filename, but rather the directory path in which the file is being imported. Obviously this doesn't happen if you just create a new file in a workarea -- only if you're importing a new file. I don't know why the Local File Manager applet interacts with TeamSite like that, and it's definitely not how I as an armchair programmer would have done it.

This sort of prepending of the directory path to the filename is what creates the problem because it makes it tough to determine the real filename, especially if the filename has underscores in it.

Fortunately, the solution I posted yesterday is working well.

Hmm - interesting.

Based on that - yeah, I guess you have to go through the contortions you were doing. You should probably file a detailed use-case as a feature request with Support (it won't help you now but it might help in the future)

Just for giggles - create a small perl trigger script that just dumps out the ARGV contents - enable it for CreateFSE and see if it returns the same thing (I'm curious about the temporary filename being provided instead of the target filename).

Bjorn

Just for giggles - create a small perl trigger script that just dumps out the ARGV contents - enable it for CreateFSE and see if it returns the same thing (I'm curious about the temporary filename being provided instead of the target filename).

What I lack in expediency, I make up for in slowness. But here're the results of your test:

[Wed Sep 23 14:49:52 2009]
domain\user
master
CreateFSE
\TeamSite1\main\clients\beta\WORKAREA\one
\zz_lfm_junk_trunk_ibatis.pdf

This is using the LFM applet (file - import from CCPro) to import the file 'ibatis.pdf' to the vpath \TeamSite1\main\clients\beta\WORKAREA\one\junk\trunk

Notice its manipulation of the path and filename.

Migrateduser

Sorry for resurrecting a 2 year old thread, but we have been dealing with this same exact issue regarding the CreateFSE event trigger and the Import function. The solution of creating a trigger on the Lock event checking for the string "import" in the Lock Comment is brilliant - works like a champ.

I wanted to thank Bjorn for the solution - I was just about to raise holy **** with support about their CreateFSE event with Import. I still may. I think it's absurd that their own tool to put files into the repository doesn't work properly with their own tool that captures new file creation. Anyway...crisis averted - thanks Bjorn.

The thread can go back to sleep now...

boriskerzner

Thanks Bjorn for posting your solution, which I just implemented! This thread was a big help, and it felt great to know that I wasn't the only one who was receiving the file in a non-final intermediate form. The one thing I *did* notice, however, is that even after the intermediate file was gone, it was not always the case that the deduced final destination file was immediately present. I added a "sleep" of 3 seconds (arbitrary number) after the intermediate file was gone, but before checking for the destination file, and that seemed to take care of the problem. Maybe someone in the future will encounter the same caveat, and know that he/she is not alone.