Discussions
Categories
Groups
Community Home
Categories
INTERNAL ENABLEMENT
POPULAR
PUBLIC CLOUD
PRIVATE CLOUD
Quick Links
MY LINKS
HELPFUL TIPS
Back to website
Home
Web CMS (TeamSite)
DCT efficiency
System
I had always heard that fork and dbopen were two expensive operations. I am working with a DCT that has 5 inline statements, 4 of which fork Perl scripts that open database connections, mostly to populate drop-down options. Other than writing the entire DCT as one monster inline is there any way to improve the efficiency? I tried writing all the options in FormAPI generated by a single inline, but that breaks in IE5.5, and anyway if they're in a replicant then you have to repopulate them every time they add an instance.
It takes about 15 seconds for this DCT to open (with no other users on the system) and I'm probably not done yet; I'm afraid it will be ridiculous once I start opening large DCRs.
Is there any form of connection pooling available for Perl?
Find more posts tagged with
Comments
Adam Stoller
You might be able to make use of Apache's mod_fastcgi (it appears to come with TS6 - but I haven't seen any documentation regarding using it for customer/field-developed code).
There's also a question regarding how efficiently the Perl scripts were written and/or whether you'd be better off with some other kind of scripting language (like /bin/ksh or something) which might be less expensive to start up (though also somewhat less powerful as a language)
--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com
Migrateduser
This is something about Perl that kills me - I want to put my routines in modules so I have less code to test and copy around, but every module I reference in a script has to be parsed entirely each time the scripts referencing it execute. I don't think this is true for compiled languages (there may be a little reflection but this is minimal compared to parsing every line for syntax) and I think it's a pretty significant design flaw in the language. Other than the modules (say, to encapsulate DB password and username lookup) the scripts are pretty trivial - I can put timing stats in the script but I am confident that it's really the excess parsing, forking and connecting I want to avoid.
This project is going live with 5.5.2 and it's not my platform so I am hesitant to make changes to the core product, especially if they are not documented, tested, supported, etc...
Adam Stoller
Not sure about time required to parse used modules, but you can make use of Perl's Benchmark utility to test efficiency of routines.
I wrote a very simplistic piece of code:
#!/usr/iw-home/iw-perl/bin/iwperl -w
use Benchmark;
$file = "/Users/ghoti/.tcshrc";
timethis(1000000, "\$i = foo();");
sub foo{
my $i = 0;
open(IN, "<$file") || die " '$file' ($!)";
while(){
$i++ if($_ =~ /alias/);
}
close(IN);
return $i;
}
and ran it 3 times:
timethis 100000: 18 wallclock secs (13.18 usr + 3.51 sys = 16.69 CPU) @ 5991.61/s (n=100000)
timethis 100000: 18 wallclock secs (13.02 usr + 3.75 sys = 16.77 CPU) @ 5963.03/s (n=100000)
timethis 100000: 18 wallclock secs (13.12 usr + 3.73 sys = 16.85 CPU) @ 5934.72/s (n=100000)
I then modified the above to look like:
#!/usr/iw-home/iw-perl/bin/iwperl -w
use Benchmark;
use lib ".";
use foo qw(foo);
$file = "/Users/ghoti/.tcshrc";
timethis(1000000, "\$i = foo();");
and create a
foo.pm
as such:
package foo;
use vars qw(
@ISA
@EXPORT_OK
$VERSION);
use Exporter;
$VERSION = 1.0;
@ISA
= qw(Exporter);
@EXPORT
= qw();
@EXPORT_OK
= qw(foo);
sub foo{
my $i = 0;
open(IN, "<$main::file") || die " '$file' ($!)";
while(){
$main::i++ if($_ =~ /alias/);
}
close(IN);
return $i;
}
1;
and ran it 3 times:
timethis 100000: 18 wallclock secs (13.17 usr + 3.27 sys = 16.44 CPU) @ 6082.73/s (n=100000)
timethis 100000: 18 wallclock secs (12.73 usr + 3.65 sys = 16.38 CPU) @ 6105.01/s (n=100000)
timethis 100000: 17 wallclock secs (13.23 usr + 3.24 sys = 16.47 CPU) @ 6071.65/s (n=100000)
The numbers are pretty close, across 100,000 iterations of the code - though there does seem to be a slight edge towards the all-inclusive code over the external module code (the one 17 second run may have been a fluke?)
--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com
Migrateduser
These numbers only work if your module only has the subroutines you use - if you have a module with lots of subs that are not used by your script it still has to parse them. I guess I could use one module per subroutine, but I don't see how that works for OO Perl for classes that don't have inheritence (the one module would have all the code, or include other modules to the same result).
I guess I'm stuck with the performance.
AKB
Yeah this does bring back the discussions that we had couple of weeks back about java and perl...
Adam Stoller
I just ran the same test - the first time with
foo.pm
containing only the subroutine
foo()
:
timethis 100000: 17 wallclock secs (13.46 usr + 3.50 sys = 16.96 CPU) @ 5896.23/s (n=100000)
timethis 100000: 18 wallclock secs (13.48 usr + 3.46 sys = 16.94 CPU) @ 5903.19/s (n=100000)
timethis 100000: 18 wallclock secs (13.20 usr + 3.40 sys = 16.60 CPU) @ 6024.10/s (n=100000)
The second time with
foo.pm
modified to contain 8 copies of the subroutine (none exported by default):
timethis 100000: 17 wallclock secs (13.15 usr + 3.61 sys = 16.76 CPU) @ 5966.59/s (n=100000)
timethis 100000: 17 wallclock secs (13.35 usr + 3.55 sys = 16.90 CPU) @ 5917.16/s (n=100000)
timethis 100000: 18 wallclock secs (13.39 usr + 3.74 sys = 17.13 CPU) @ 5837.71/s (n=100000)
and lastly with the 8 subroutines all exported by default:
timethis 100000: 18 wallclock secs (13.09 usr + 3.71 sys = 16.80 CPU) @ 5952.38/s (n=100000)
timethis 100000: 18 wallclock secs (12.89 usr + 3.59 sys = 16.48 CPU) @ 6067.96/s (n=100000)
timethis 100000: 18 wallclock secs (13.22 usr + 3.56 sys = 16.78 CPU) @ 5959.48/s (n=100000)
I don't see any noticable / consistant differences here. Granted, yes, I'm using a simplified test case, but it's still running 100,000 times so even a small difference on an individual run should show up fairly clearly.
FYI - with the last case, I also ran a single run (100,000 iterations) through /usr/bin/time - to get an idea of the overhead of invoking Perl:
timethis 100000: 17 wallclock secs (12.57 usr + 3.90 sys = 16.47 CPU) @ 6071.65/s (n=100000)
17.50 real 12.82 user 3.91 sys
I think you might be making a mountain out of a mole-hill here...
--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com
Migrateduser
I'm just saying that it's noticeable and it gets worse with the more complicated files, but maybe it isn't the execs. If it gets real bad I will check it with time/timethis maybe with some loops.
Are you using timethis from ms or some other? I mean, is it loading the modules repeatedly or just executing the subs repeatedly?
Adam Stoller
timethis
(and
timethese
) is a function provided from the
Benchmark
module.
Good point though - this is probably not re-loading the module N - times within N iterations of the call to the function foo(). Okay this time I've got a bunch of different files:
foo1.ipl
- all inclusive, no module
foo2.ipl
/
foo2.pm
- module contains only one subroutine
foo3.ipl
/
foo3.pm
- module contains 8 subroutines, none exported by default
foo4.ipl
/
foo4.pm
- module contains 8 subroutines, all exported by default
bar.ipl
#!/usr/iw-home/iw-perl/bin/perl -w
use Benchmark;
timethese(500, {
'foo1' => "system('foo1.ipl');",
'foo2' => "system('foo2.ipl');",
'foo3' => "system('foo3.ipl');",
'foo4' => "system('foo4.ipl');",
});
Results:
Benchmark: timing 500 iterations of foo1, foo2, foo3, foo4...
foo1: 6 wallclock secs ( 0.04 usr 0.55 sys + 1.67 cusr 3.28 csys = 5.54 CPU) @ 847.46/s (n=500)
foo2: 24 wallclock secs ( 0.05 usr 0.52 sys + 15.81 cusr 6.39 csys = 22.77 CPU) @ 877.19/s (n=500)
foo3: 28 wallclock secs ( 0.02 usr 0.68 sys + 17.29 cusr 6.63 csys = 24.62 CPU) @ 714.29/s (n=500)
foo4: 27 wallclock secs ( 0.03 usr 0.83 sys + 16.84 cusr 6.52 csys = 24.22 CPU) @ 581.40/s (n=500)
---
Benchmark: timing 500 iterations of foo1, foo2, foo3, foo4...
foo1: 6 wallclock secs ( 0.06 usr 0.52 sys + 1.44 cusr 3.57 csys = 5.59 CPU) @ 862.07/s (n=500)
foo2: 27 wallclock secs ( 0.03 usr 0.78 sys + 15.60 cusr 6.50 csys = 22.91 CPU) @ 617.28/s (n=500)
foo3: 27 wallclock secs ( 0.02 usr 0.61 sys + 16.57 cusr 6.67 csys = 23.87 CPU) @ 793.65/s (n=500)
foo4: 27 wallclock secs ( 0.04 usr 0.74 sys + 16.70 cusr 6.82 csys = 24.30 CPU) @ 641.03/s (n=500)
---
Benchmark: timing 500 iterations of foo1, foo2, foo3, foo4...
foo1: 7 wallclock secs ( 0.04 usr 0.52 sys + 1.36 cusr 3.69 csys = 5.61 CPU) @ 892.86/s (n=500)
foo2: 26 wallclock secs ( 0.04 usr 0.77 sys + 16.03 cusr 6.50 csys = 23.34 CPU) @ 617.28/s (n=500)
foo3: 25 wallclock secs ( 0.04 usr 0.68 sys + 16.78 cusr 6.91 csys = 24.41 CPU) @ 694.44/s (n=500)
foo4: 27 wallclock secs ( 0.03 usr 0.68 sys + 17.31 cusr 6.93 csys = 24.95 CPU) @ 704.23/s (n=500)
This clearly shows that the use of a module
does
impact performance (
sorry for doubting you about this before, I hadn't realized I wasn't testing what we were talking about
) However the impact doesn't seem too closely related to the number of subroutines defined within the perl module (though again, I'm using a simplistic case, so mileage will certainly vary)
--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com
Migrateduser
I played around with a script that just had includes and it is certainly my code (as opposed to strict, File::Basename, Getopts:
td, etc.) that makes it take longer. I wonder if it's my coding style or if my module is just too big or references too many other modules...
gzevin
is mod_perl compiled with apache in our installations?
usually if you force DBI to run under mod_perl, it will stay like an re-entrant function in memory and all subsequent calls to a DB should be lighning fast?
has anyone tried this? I did it - but not in Teamsite..
Greg Zevin, Ph.D. Comp. Sc.
Independent Interwoven Consultant/Architect
Sydney, AU
Adam Stoller
TS6 (on Solaris) appears to have mod_fastcgi, but not mod_perl. I've never had a chance to play with either of these, but I've felt for a while that it would make sense for the iwwebd server to have that functionality enabled so that it could be used to make execution of perl scripts more efficient.
--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com
