nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

performance problem with iw_solution_email.ipl

yuvpurush

Hi,
Im using TS671, Win 2003 EE SP1.

Im using the default iw_solution_email to generate email notifications. while using in the workflow, the iw_solution_mail.ipl take nearly 25% of the cpu usage for about 3 secs. The Server is a 2 CPU box of 2.9 Ghz with 4 GB RAM. The email_map.cfg contains about 1800 entries.

Has anyone encountered a performance problem with iw_solution_email earlier. Could the size of the email_map.cfg be a cause for the problem?

Thanks
Yuv

Find more posts tagged with

Comments

nipper

I have not seen this issue, but I think you are correct on the issue being the email_map. If you peruse the PM, iwov_emailmap.pm you will see that it iterates through the list doing a compare.



@userlist   = $rootNode->get_node_list('user'); 
 foreach $userlist (@userlist) {
  $name = $userlist->get_attrib('name');
  $email = $userlist->get_attrib('email');
  $fullname = $userlist->get_attrib('fullname');

  # Look for a case-insensitive match of the 'name' from the
  # config file to either the full username or the username
  # without the NT domain.
  if (((lc $name) eq (lc $full_username))
                    || ((lc $name) eq (lc $bare_username))) {

That works great for a small set (couple hundred or less) of email addresses. But it is a brute force method.

I do not know if one could convert this to a hash and make it faster, but a regex would certainly be faster. If you are comfortable in Perl you can play with it.

Maybe someone else may look at it as well.

Andy

Adam Stoller

I have not seen this issue, but I think you are correct on the issue being the email_map. If you peruse the PM, iwov_emailmap.pm you will see that it iterates through the list doing a compare.
@userlist   = $rootNode->get_node_list('user'); 
 foreach $userlist (@userlist) {
  $name = $userlist->get_attrib('name');
  $email = $userlist->get_attrib('email');
  $fullname = $userlist->get_attrib('fullname');

  # Look for a case-insensitive match of the 'name' from the
  # config file to either the full username or the username
  # without the NT domain.
  if (((lc $name) eq (lc $full_username))
                    || ((lc $name) eq (lc $bare_username))) {
That works great for a small set (couple hundred or less) of email addresses. But it is a brute force method.

I do not know if one could convert this to a hash and make it faster, but a regex would certainly be faster. If you are comfortable in Perl you can play with it.

Maybe someone else may look at it as well.

Andy

Another place you lose time is with the reading in of the email_map.cfg file:

open (CFG, $email_map_file) || die "Cannot open $email_map_file: $!\n";
$xml = join ("", <CFG>);
close CFG;

I'd suggest changing it to:

open(CFG, '<', $email_map_file) || die "Cannot open $email_map_file: $!\n";
$xml = do { local $/; <CFG> };
close CFG;

- I just threw together a quick benchmark script for these two routines, and put roughly 500 entries into my email_map.cfg and got back the following results for a run of 1000 tries:

Benchmark: timing 1000 iterations of iwov, slurp...
    iwov: 3 wallclock secs ( 2.67 usr + 0.13 sys = 2.80 CPU) @  357.53/s (n=1000)
   slurp: 0 wallclock secs ( 0.16 usr + 0.11 sys = 0.27 CPU) @ 3773.58/s (n=1000)
   (warning: too few iterations for a reliable count)

- So I bumped the # of entries up to about 1500 and the numbers changed:

Benchmark: timing 1000 iterations of iwov, slurp...
    iwov: 8 wallclock secs ( 7.97 usr + 0.23 sys = 8.20 CPU) @  121.91/s (n=1000)
   slurp: 1 wallclock secs ( 0.45 usr + 0.13 sys = 0.58 CPU) @ 1730.10/s (n=1000)

So I think you could get some significant time savings by just changing that one piece of code.

nipper

Another place you lose time is with the reading in of the email_map.cfg file:
open (CFG, $email_map_file) || die "Cannot open $email_map_file: $!\n";
$xml = join ("", <CFG>);
close CFG;
I'd suggest changing it to:
open(CFG, '<', $email_map_file) || die "Cannot open $email_map_file: $!\n";
$xml = do { local $/; <CFG> };
close CFG;
- I just threw together a quick benchmark script for these two routines, and put roughly 500 entries into my email_map.cfg and got back the following results for a run of 1000 tries:
Benchmark: timing 1000 iterations of iwov, slurp...
    iwov: 3 wallclock secs ( 2.67 usr + 0.13 sys = 2.80 CPU) @  357.53/s (n=1000)
   slurp: 0 wallclock secs ( 0.16 usr + 0.11 sys = 0.27 CPU) @ 3773.58/s (n=1000)
   (warning: too few iterations for a reliable count)
- So I bumped the # of entries up to about 1500 and the numbers changed:
Benchmark: timing 1000 iterations of iwov, slurp...
    iwov: 8 wallclock secs ( 7.97 usr + 0.23 sys = 8.20 CPU) @  121.91/s (n=1000)
   slurp: 1 wallclock secs ( 0.45 usr + 0.13 sys = 0.58 CPU) @ 1730.10/s (n=1000)
So I think you could get some significant time savings by just changing that one piece of code.

Do you really expect a significant improvement from that ? If I read your data correctly, it took 8 milliseconds to open/read the file (for each of 1000 iterations). That as opposed to 1 millisecond. While that is not trivial (if you are running a bunch of emails), I highly doubt that causes his initial issue of 3 seconds of a high CPU usage.

nipper

Also, for the original question, this routine searches each node of the XML until it finds a match. By moving some of the more frequent users to the top of the email_map file, you *should* see improvement in the time the foreach loop runs. Of course, it is possible that the XMLParser is where the bulk of the time is being spent and then it will not help much.

You may want to try Adam's benchmark code and see how long it takes to read the file and parse the XML.

Andy

yuvpurush

Hi all,
Thanks for the suggestions.

I tried changing the code to below as suggested by adam. But no improvement. Still the same 25% CPU also the parsing of the header part takes about 9-15 secs.


open(CFG, '<', $email_map_file) || die "Cannot open $email_map_file: $!\n";
$xml = do { local $/;  };
close CFG;

I then checked with around only 10 entries in the email_map.cfg. I can find significant increase in the response time. The header part parsing was reduced from around 10 secs to only 1 sec and also the cpu usage dropped to around 6%. Now identified the root cause.

Thanks a lot.

nipper

You may want to play with regex and see if you can regex the data you need out of the email_map file. If I have a chanfce I will play with it today. That will certainly drop your load and response time.

Andy

nipper

Here is a SAMPLE regex that you can play with to replace the XML parsing if you so desire.

The first regex works fine, but the second, using the domain\user match is failing. I will look at that but I assume someone may see the mistake before I do. Obviously the regex is tied to one type of quotes (double quotes) and is not looking for whitespace, which would need to be done.


#!c:\interwoven\iw-home\iw-perl\bin\iwperl

my $xml = qq{<emailConf>
<user name="aknipp" fullname="Andy Knipp" email="aknipp\@myco.com"/>
<user name="user1" fullname="User One" email="user1\@myco.com"/>
<user name="user2" fullname="User Two" email="user2\@myco.com"/>
<user name="user3" fullname="User Three" email="user3\@myco.com"/>
<user name="user4" fullname="User Four" email="user4\@myco.com"/>
<user name="domain\\user5" fullname="User Five" email="user5\@myco.com"/>
<user name="domain\\user6" fullname="User Six" email="user6\@myco.com"/>
<user name="domain\\user7" fullname="User Seven" email="user7\@myco.com"/>
</emailConf>
};

my $find = "aknipp";
$xml =~ m|name=\"$find\" fullname=\"(.*?)\" email=\"(.*?)\"/>|i;
$fullname=$1;
$email=$2;
print "Find - $find Fullname - $fullname email - $email\n";

$find = "domain\\user7";
$xml =~ m|name=\"$find\" fullname=\"(.*?)\" email=\"(.*?)\"/>|i;
$fullname=$1;
$email=$2;
print "Find - $find Fullname - $fullname email - $email\n";

exit(0);

HTH

Andy

nipper

Here is a regex that is whitespace tolerant as well as either type of quotes. Still have not gotten the domain to work:



$xml =~ m|name[ \t]*=[ \t]*[\"\']$find[\"\'][ \t]+fullname[ \t]*=[ \t]*[\"\'](.*?)[\"\'][ \t]+email[ \t]*=[ \t]*[\"\'](.*?)[\"\'][ \t]*/>|i;

nipper

OK, so here is one with the domains working. If you feel adventurous, try to replace the iwov_emailmap.pm code with something like this.



#!c:\interwoven\iw-home\iw-perl\bin\iwperl

my $xml = qq{<emailConf>
<user name ="aknipp" fullname ='Andy Knipp'     email = "aknipp\@myco.com" />
<user name="user1" fullname="User One" email="user1\@myco.com"/>
<user name="user2" fullname="User Two" email="user2\@myco.com"/>
<user name="user3" fullname="User Three" email="user3\@myco.com"/>
<user name="user4" fullname="User Four" email="user4\@myco.com"/>
<user name="domain\\user5" fullname="User Five" email="user5\@myco.com"/>
<user name="domain\\user6" fullname="User Six" email="user6\@myco.com"/>
<user name="domain\\user7" fullname="User Seven" email="user7\@myco.com"/>
</emailConf>
};
my @users = ("aknipp","user1","user4","domain\\user6");
foreach $find (@users) {
if ($find =~ m|(.*)\\(.*)|){
 $domain = $1;
 $user   = $2;
 $xml =~ m|name[ \t]*=[ \t]*[\"\']$domain\\$user[\"\'][ \t]+fullname[ \t]*=[ \t]*[\"\'](.*?)[\"\'][ \t]+email[ \t]*=[ \t]*[\"\'](.*?)[\"\'][ \t]*/>|i;
 $fullname = $1;
 $email    = $2;
 } else {
 $xml =~ m|name[ \t]*=[ \t]*[\"\']$find[\"\'][ \t]+fullname[ \t]*=[ \t]*[\"\'](.*?)[\"\'][ \t]+email[ \t]*=[ \t]*[\"\'](.*?)[\"\'][ \t]*/>|i;
 $fullname = $1;
 $email    = $2;
 }
print "Find - $find Fullname - $fullname email - $email\n";
}

exit(0);

ISCBorisB

OK, so here is one with the domains working. If you feel adventurous, try to replace the iwov_emailmap.pm code with something like this.

A limitation you probably have to spell out is that you assumed a fixed order of user Tag attributes

nipper

A limitation you probably have to spell out is that you assumed a fixed order of user Tag attributes

Good point. I assume you *could* do a regex that is order independent, but mine is ugly enough, I cannot imagine how bad that one would look.

ISCBorisB

Good point. I assume you *could* do a regex that is order independent, but mine is ugly enough, I cannot imagine how bad that one would look.

You can certainly do it order-neutral, without trying I would say it'll be 10 times easier to do by 2 regexes in a row.
Short of showing off there is no need for that IMO, not without trying performance impact first anyway (and perhaps even then)

yuvpurush

Andy,
Thanks a lot. I will try that out and post the results.

Thanks
Yuv

untitled.bmp

Adam Stoller

OK, so here is one with the domains working. If you feel adventurous, try to replace the iwov_emailmap.pm code with something like this.

or this:[html]use strict;
use warnings;

my($fullname, $email) = (q(),q());

my $xml = qq{

fullname="Andy Knipp"
email="aknipp\@myco.com"/>

};

sub find_name {
my ($xml, $find) = @_;

if ($xml =~ m|name\s*=\s*["']\Q$find\E["']
\s+
fullname\s*=\s*["'](.*?)["']
\s+
email\s*=\s*["'](.*?)["']
\s*/>|xi){
warn("Find - $find Fullname - $1 email - $2\n");
return ($1, $2);
}
warn("Unable to find fullname and email for '$find'\n");
return(undef, undef);
}

($fullname, $email) = find_name($xml, "aknipp");
($fullname, $email) = find_name($xml, 'domain\user7');

exit(0);
[/html](the above still assumes the attributes are in the oder shown)

nipper

Cool fish, thanks. I forgot about using the \s. I have used the \Q before, but it was not working in this example because I did not know to use the \E. I had to dig through the ORA regex book to see what this was doing.

BTW, what does the x do at the end of the regex ?

ISCBorisB

Cool fish, thanks. I forgot about using the \s. I have used the \Q before, but it was not working in this example because I did not know to use the \E. I had to dig through the ORA regex book to see what this was doing.

BTW, what does the x do at the end of the regex ?

x Modifier is for readability, it allows you to use whitespace and comments in the pattern

nipper

Strange, I ran this myself and it worked for the user without the domain, but not with the domain. I did have to escape the @ sign and tried to do the same with the backslash. That was why I used 2 regexes instead of 1.I could not get it to work with both.

Find - aknipp Fullname - Andy Knipp email - aknipp@myco.com
Unable to find fullname and email for 'domain\user7'

ISCBorisB

Strange, I ran this myself and it worked for the user without the domain, but not with the domain. I did have to escape the @ sign and tried to do the same with the backslash. That was why I used 2 regexes instead of 1.I could not get it to work with both.

Find - aknipp Fullname - Andy Knipp email - aknipp@myco.com
Unable to find fullname and email for 'domain\user7'

That's Forum's software quirk. Take a look at Adam's code in the "Quote" mode, you'll see extra escapes that are invisible in standard (html Tag) rendition.

nipper

Got it, thanks.

Andy