nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Number of files in workflow

Annad

I have a workflow that runs in on the 'new_job' signal. The users are allowed to select a directory which might have any number of files.
When the users select a directory or a large numer of files (over 300), the workflow is have access to these files on the first task but the task has no files in the second task.

I tested for 300 files and it worked for at most 300 but then (ex: 301 files) the second task did not have any files in it.

Does anyone know the reason for it, or is it the limit on the number of files for a workflow?

Thanks.
Anna

Find more posts tagged with

Comments

Adam Stoller

it might help if we could see the workflow template or job specification derived from the workflow template.

However, on the face of it, this might be something you need to go through support on if it's that repeatable.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Annad

Hi,
please see the attached file for the workflow template.

Thanks.

Adam Stoller

When you say there are no files in the second task I assume you're talking about "Set EA" - yes? or "Unlock DCR Only" ?

In either case - could you post the externaltask script associated with the task which you are saying seems to be hit-or-miss with regard to seeing the files that were associated with the workflow in the first task?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Annad

Yes,
it is for the 'Set EA' task, and please see the attached file.

Adam Stoller

I've seen the wft - what I'd like to see is $iwhome/local/custom/scripts/BatchUpload/HRDC_Batch_Set_EA.ipl

Since you're saying it is during the operation of this script that you seem to be missing files where were known to be attached to the job in the preceeding task.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Annad

Oh,
I am sorry. I meant to upload the right file but didn't.
Here it is.

Thanks.
Anna.

Adam Stoller

I'd suggest adding the following code around line 110 of that script to help verify whether the problem is with the number of files associated with the task at the beginning of the script: @files = $task->GetFiles();
#=# ADDED CODE
if (open(OUT, ">>$iwhome/tmp/dbg.log")){
print "[".localtime()."]INIT - File Count: ". @files . "\n";
close(OUT);
}
#=# END ADDED CODE

And while we're at it - add the following code around line 78 of the WFT: @files = __VALUE__('iw_file');
#=# ADDED CODE
if (open(OUT, ">>$iwhome/tmp/dbg.log")){
print "[".localtime()."]WFT - File Count: ". @files . "\n";
close(OUT);
}
#=# END ADDED CODE Between these two code additions and the output that gets generated into $iwhome/tmp/dbg.log hopefully we'll be able to verify what you believe you are seeing.

If the numbers come back the same each time - yet there appear to be missing files elsewhere (either within the externaltask script or elsewhere) - we'll have an idea of where to begin the next set of tests like this ... yes?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Annad

Thanks for your suggestion. I tried that and the number of files are not the same. The number of files in the 'set EA' task is 0. However I tried printing the number of files in the first task (Convert To DCR) and no files are missing in that task.

Any idea what's going on??

Thanks.

Adam Stoller

If for nothing else than my sanity - could you past the contents of the debug log?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Annad

well,
since I was already writing the to a log file in both the wft file (if you had noticed that) and the scripts, I wrote the number of files using that.
You can see the number of files printed at the beginning of the file (# OF FILES: <number of files>)

Thanks

see the attached 3 files for the wft, task Convert To DCR , and task set EA.

wft log file: HRDC_Batch_Upload-anna.fernando.log
Convert To DCR : HRDC_Batch_Convert_toDCR-anna.fernando.log
set EA : HRDC_Batch_set_EA-anna.fernando.log

Thanks a lot.

Adam Stoller

I don't see anything jumping out at me - and don't have time to really dig into this - I'd suggest you take this information to Support and open a case with them. The onus will be, unfortunately, on you to get them to replicate the problem in their environment.

The **one** thing I can think of that **might** help you is in the iw.cfg find the section that starts with:

[workflow]

and add the following line:
external_task_add_filelist=false

Then run iwreset and try re-running your test case(s).

The reason why this **might** help is that you might be running into a problem where, even though you're not accessing the file list from @ARGV (you're using $task->GetFiles() which is the correct way to do it) - it's possible that the command line is getting overloaded and messing something up.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Annad

Thanks for your time.
I tried your last suggestion but it didn't help. Anyway, I will submit this to the support site after I take a close look at the scripts and any other factors.

Thanks again.
Anna