nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

core dumping Perl

nipper

So I do not know how to debug this.

I have a script that use's Smitty's recursive file list to get a list of files. Put's that into an arrary. Then it checks each file for some attributes (see if it is modified, who, when, etc) and writes that into a file.

I have verified that this is not data generated, if I run on the lower lever directories it works. But when I run it on templatedata it seg faults and core dumps.

So I assume this puppy is just too big. Any limits I should care about ?

TIA

Andy

Find more posts tagged with

Comments

Adam Stoller

Care to post the script and put it up for critique?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

nipper

put it up for ridicule ?

sure.

Reading your perl tips, one thing I am thinking I need to do is instead of passing the complete file list array I should pass the reference. On my list of todos.

Migrateduser

Ugh. I am cringing waiting for all the experts to critique my code. On the other hand, since Andy is posting it, it's really Andy's code now. Any bad coding form must be stuff he modified from my original. There, that was easy...

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

nipper

I see, not only am I a plagiarist, I am an inaccurate one. :-)

thanks buddy

Migrateduser

It's just that I put in some obvious bad coding form in the version I sent to you just to see if you'd be sharp enough to catch them. Evidently I overestimated your skill. My bad.

Smiley Happy

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

nipper

No you actually under estimated my laziness.

I think I found it, it came accross a js file which failed in the DCR read code.

I forgot the if (! $rootnode) test.

I suspect that the writes were buffered (and lost) since it did not seem to get that far.

Andy

Adam Stoller

First, this is how I would write get_recursive_files()

#=====================================================================
# Find all files within directory 'dir'
#=====================================================================
sub get_recursive_files {
    my ($dir, $refList) = @_;
    my @subdirs = ();

    opendir(BASE, $dir) || die "[opendir] '$dir' ($!)";
    my @basefiles = readdir(BASE);
    closedir(BASE);

    foreach (@basefiles) {
        next if /^\.+$/;
        if (-d "$dir/$_"){
            push(@subdirs, "$dir/$_");
        }
        else {
            push(@$refList, "$dir/$_");
        }
    }
    get_recursive_files($_, $refList) foreach(@subdirs);
}

However, in your case, you only want the modified files, so you could probably get faster results using something like this:

#=====================================================================
# Use iwlistmod to get a list of all modified files and
# sub-directories starting from '$dir'
# Extract just the 'files' from the above list
#=====================================================================
sub get_modified_files{
    my ($dir, $refList) = @_;
    chomp(my @modlist = qx($iwhome/bin/iwlistmod "$dir"));
    foreach (@modlist){
        push(@$refList, "$dir/$_") if(/^f\+\s+(.*)$/);
    }
}

If you want to sift the NEW from the MODIFIED, you could try something like the following, however I have not done any benchmarking to figure out whether it is faster to do it this way, or to use the get_modified_files() routine and do the iwattrib commands that you have in your code already:

#=====================================================================
# Use iwcmp to get a list of all modified files and
# sub-directories starting from the workarea '$wa'
# Extract just the 'files' from the above list and separate
# into two categories: "new" and "changed"
#=====================================================================
sub get_modified_files{
    my ($wa, $refNew, $refChanged) = @_;
    (my $stg = $wa) =~ s|/WORKAREA/[^/]+|/STAGING|;
    chomp(my @modlist = qx($iwhome/bin/iwcmp "$wa" "$stg"));
    foreach (@modlist){
        if (/^(\S+)\s+(\S+)\s+(\S.*)$/){
            if ($1 eq "f+"){
                if (($2 eq "f+") # conflict
                    || 
                    ($2 eq "f")) # modified in WA
                { 
                    push(@refChanged, "$wa/$3");
                }
                elsif ($2 eq "-"){
                    push(@refNew, "$wa/$3");
                }
            }
        }
    }
}

There are several other changes that I would probably make to the script as written (declaring variables with 'my', using TeamSite::Config::iwgethome() instead of hard-coding the path to IWHOME in your code, etc.), but I think the big-bang-for-the-buck would be switching to using one of the above routines. If I find time, I might try rewriting the whole thing and re-posting it as an attachment, but I am not sure if/when I will find time to do that ;-)

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

nipper

Thanks for the critique fish, I will make those changes.

BTW, I had another perl task that went through DCRs looking for a missing (newly required) field and added it. Originally fixing on about 8000 records and doing a submit direct. It took over 3 hours on my dev server. I wanted to submit anything I change IF it was not updated by someone else as well, so I could not just submit everything.

So I made an adjustment, making the required changes and if the file was not previously modified, putting it into a file and did a submit -f at the end. went down to about 30 minutes door to door (though iwserver probably was doing a lot in the background after).

I knew that would have an impact, did not think that much.

Andy

Migrateduser

Furthermore, it looks like the script builds a list of files and them iterates over the files in the list processing each one independently. Then why build a potentially hugh, in-memory list in the first place? Why not simply process each file as you crawl through the directories?

Consider using File::Find which is very fast and does most of the work for you.

Brinko Kobrin
Interwoven Staff Engineer

nipper

> Consider using File::Find which is very fast and does most of the work for you.

Sounds like I might do some benchmarking next week. I had used @list = `ls -R` & then parsed it. Smitty & fish have different ways & Brink suggests another. Since I use this type of code often (about 15K DCRs & we do not use DAS so metadata search won't work), getting it correct is important. Of course ANYTHING is faster than my first hack: find templatedate -type f -exec script {} \;

It worked but ran about 4 times longer than having the script parse the directory structure. It did let me prove the concept, had it been 5 minutes instead of 2 minutes, I would not worry, but 2 hourse instead of 30 minutes is a big impact.

Andy

Migrateduser

It looks like the per-file processing could be tuned-up a bit as well. For example, if the "modified" attribute is not "TRUE" you don't need to run iwattrib to get "moddate" and "lastuser". These values will be ignored.

If you run some benchmarks, you may want to separate the directory scanning time from the file processing time. That will reduce the time that it takes to give you run your benchmarks. Try making a version of your script that just counts the files, but doesn't do any additional processing.

Brinko Kobrin
Interwoven Staff Engineer