Exclude regex help needed

We are trying to create an exclude regex that will do the following:

Exclude any file ending in ".jhtml" except do not exclude "index.jhtml".

I've tried as many possibilities as I can think of, but I can't do it in a one-liner and I am not exactly a regex expert. If anyone has an example of how to pull this off in one sweet regex, I would appreciate it.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Find more posts tagged with

Comments

Migrateduser

The good news - this is trivial with a negative lookbehind assertion.

The bad news - negative lookbehind didn't come until perl 5.6, and the regexes in TS are perl 5.005_03 (and mostly compatible where not implemented via perl).

The best I can think of without it would be something really ugly like:

[^i]?[^n]?[^d]?[^e]?[^x]?\.jhtml

which would be any zero or one characters other than i, followed by zero or one characters other than n, zero or one characters other than d, zero or one characters other than e, zero or one characters other than x, followed by a literal dot, then the string jhtml.

Mind you, I haven't tried that, but logically it seems to be the right sort of thing.

Cheers
Rob
rob@bagpipe.com

Migrateduser

I was just laying in bed, feeling bothered by my answer, but unable to put my finger on it, when it came to me. The regex would not match index.jhtml, but it would also not match these:

indus.jhtml, aspen.jhtml, sussex.jhtml, hoax.jhtml, pandas.jhtml, and rodeo.jhtml

So, it's not clever enough by half. Without negative lookbehind, it might not be doable in a single regex. Hmmm, just a thought - I think zero-width negative lookahead is available in 5.005_03 - you'll have to check the perlre perldoc on your TeamSite box to be sure - if it is, you could combine it with an alternation, something like this (note, I haven't actually checked this, so it might work, might not):

(i(?!ndex).*|[^i].*)\.jhtml

which would be
an i, as long as it isn't followed by the string ndex, followed by zero or more non-newlines

or
a not-i followed by zero or more non-newlines

followed by a literal dot, followed by jthml. Mind you, this is assuming that the regex syntax used in OD is fully perl5 compatible. I've never tried to do anything that complex with an exclusion regex in OD.

Cheers
Rob Huffstedtler
rob@bagpipe.com

Brett_Woven

I tried using Rob's regex in a little test program and it did not
work (index.jhtml was excluded too). And so I was thinking
this was a decent challenge - sticking to 5.005 regex's
being the big caveat.

The following seems to work, but I only tested it a little;
and it's big and ugly. I'd love to see someone do it with a
smaller pattern.

Here's mine:

m/^(.{1,4}|.{6,}|[^i].{4}|i[^n].{3}|in[^d].{2}|ind[^e].{1}|inde[^x])\.jhtml$/

This says: exclude all .jhtml files whose basename is
4 chars or less; or 6 chars or more. If those tests fail,
we know the basename is exactly 5 chars long - and
it iterates through making sure it is any 5 char sequence
but the word "index".

Since this is perl I'm sure there is another way to do it :-)

Test Program:
----------------------------------------------------------------------

#!/usr/bin/perl

while (<>) {
last if (m/^quit$/);
if (m/^(.{1,4}|.{6,}|[^i].{4}|i[^n].{3}|in[^d].{2}|ind[^e].{1}|inde[^x])\.jhtml$/) {
printf(">>> EXCLUDE\n");
}
else {
printf(">>> KEEP\n");
}
}

exit(0);

Note that this matches using a (^) for beginning of line; you
could substitute a forward slash if the source line has a
directory path ahead of the filename.

-- Brett
Interwoven Inc.

Brett_Woven

The following is slightly smaller, not sure if it's more
efficient though:

m/^(.{1,4}|.{6,}|([^i]|i[^n]|in[^d]|ind[^e]|inde[^x]).*)\.jhtml$/

-- Brett
Interwoven Inc.

Jeremy

Hi,

Just a suggestion, have you thought about putting a post up on somewhere like www.perlmonks.org ?
They are ussually pretty helpful and you may be able to get a very short oneliner.

Just a thought.

Jeremy

Migrateduser

Looks like Brett already caught my mistake, and had a pretty good go at it - he actually tested his (isn't that cheating Smiley Happy

).

Anyway, I think you can fix mine by putting a positional anchor (^) at the front. Following Brett's footsteps, I put together a little test script:

my $string = shift;
if ($string =~ /^(i(?!ndex).*|[^i].*)\.jhtml/) {
print "BING!\n";
}

Just invoke it and pass the string you want to match against as an argument.

\>bar.ipl index.jhtml

\>bar.ipl foo.jhtml
BING!

\>bar.ipl foo.html

I think that was what you wanted. Mind you, that's still making the assumption that negative lookahead is valid in this context.

Now, a correction about my original assertion that my first regex would not match the list I provided. It would, and it would also match index.jhtml. Here's why (which will also explain why the positional anchor is necessary in the above):

As you may recall, the original attempt was
[^i]?[^n]?[^d]?[^e]?[^x]?\.jhtml

In trying to match that against index.jhtml, here is what would happen. First, the not-i would try to match the first character, in the string, the i. This fails, so (since there is no positional anchor), the engine moves on and tries to match beginning with the second character of the string. The not-i matches the n, the not-n the d, and so on until the not-x matches the dot.

Then, the literal dot tries to match against the j and fails. The engine backtracks since the character classes were optional. The optional not-x ends up not being used, the dot matches the dot and jhtml matches jhtml.

The natural reaction to this would be to use a positional anchor, but that is a bad idea in this case, because it would limit us to strings with no more than 5 characters in their basename.

Migrateduser

Thanks for all the work to try and help out with this. I tried your script and it does not catch index1.jhtml. Or for that matter indexthissucks.jhtml. I painstakingly tried to use lookbehind to try and solve this, but it just doesn't work the way I need it to work. It's nice that they added it to perl 5.6, but I think I'm screwed with this version of perl. I'm going to try Brett's solution and see if it works. Again, thanks for taking the time.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

Plus, I need it to allow for a path leading up to the filename, so I can't use the beginning of line indicator (^), I need it to just look after the last forward slash (/).

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

This seems to work as long as the list of files contains only filenames, however there may or may not be a path leading up to the filename, so it needs to work for all of these cases:

index.jhtml
doc/index.jhtml
nike/doc/index.jhtml

Nothing like adding a little more complexity into the mix. I tried replacing the beginning of line character (^) with some variations of forward slashes, but I can't seem to get it to work. This is so frustrating.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

4 reasons why I didn't go to a straight perl newsgroup for help:

1. Ever ask a question in comp.lang.perl? Those pompous **** burn you from the stake if you dare ask them a question without knowing the pompous **** secret handshake.

2. I didn't want to have to explain why I needed this in a one-liner. No doubt they would have asked why I needed to do this in one line.

3. I figured there are a lot of smart people that use OpenDeploy and have had to come up with some pretty tricky regex's themselves. This seemed like a kind of problem someone else might have come across using this product.

4. People are much nicer here. See reason #1.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

OK - good catch, but easily solved. Here's my latest:

my $string = shift;
if ($string =~ /\/(i(?!ndex(?=\.)).*|[^i].*)\.jhtml/) {
print "BING!\n";
}

I've replaced the positional marker with a slash (which I had to escape - OD may escape slashes in regexes automatically - I vaguely recall that it does). Then inside the negative lookahead, I added a positive lookahead, so that we check to see if the character following ndex is a dot without gobbling up the dot (so the dot further on still matches).

Here's the result:

\>bar.ipl asff/foo.jhtml
BING!

\>bar.ipl asdf/indexthissyucks.jhtml
BING!

\>bar.ipl asdf/index.jhtml

\>bar.ipl asff/foo.html

Migrateduser

Yep, that definitely works better, as long as there is a path preceding the filename. Unfortunately, there may or may not be a path preceding the filename. I need it to work for all 3 of these cases:

index.jhtml
doc/index.jhtml
nike/doc/index.jhtml

and still exclude any other .jhtml file. Fun stuff.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Bowker

Is there any chance that the slashes could be in the other direction? / or \ ?

Migrateduser

Write better requirements, man! Smiley Wink

Seriously, that one's easy. Just do an alternation at the beginning of the pattern so it will match either from the beginning of the string, or from the last slash.

/(^|\/)(i(?!ndex(?=\.)).*|[^i].*)\.jhtml/

Migrateduser

I'm thinking rather than change the requirements I need to change my career. When I make this change, it now doesn't work for:

doc/index.jhtml

I'm just going to keep bugging you until you either get it or give up.

I tried:

/(^|\/)(i(?!ndex(?=\.)).*|[^i].*)\.jhtml/
/(^|i^.*\/)(i(?!ndex(?=\.)).*|[^i].*)\.jhtml/

Neither works for that case.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

That would imply that someone uses NT. Solaris is the only way to go! Smiley Happy

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

I'm thinking this one fails for doc/index.jhtml because the beginning of string always matches, so it won't even try to look for the "/" since the "^" is true. Right?

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

No - here's what's happening - geez, this is painful.

The ^ in the first alternation matches. Then the i from the second doesn't. Since it doesn't have any backtracking to do in the second alternation, it goes to the second choice. The not-i matches the slash, and the .* grabs everything to the end - some backtracking takes place, and eventually the .jhtml matches the .jhtml. How about we change the .* to zero or more not-slashes [^\/]*

Hmmm - that works for every case except

\>bar.ipl /index.jhtml
BING!

Oh, because now the ^ mathces the beginning, / matches the not-i from the second alternation and so forth. Let's alter the not-i to be a not-i-or-/

my $string = shift;
if ($string =~ /(^|\/)(i(?!ndex(?=\.)).*|[^i\/][^\/]*)\.jhtml/) {
print "BING!\n";
}

I think that has it:

\>bar.ipl index.jhtml

\>bar.ipl /index.jhtml

\>bar.ipl docs/index.jhtml

\>bar.ipl /foo.jhtml
BING!

\>bar.ipl /foo.html

\>bar.ipl docs/foo.jhtml
BING!

Mind you, if for some stupid reason you got a path that had a double slash in it - i.e. OD mangled the path before feeding it to the regex, that wouldn't work

Migrateduser

Dude, you are my hero. Many thanks for this. This looks like it's going to do the trick. Awesome!!

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Brett_Woven

Well you guys beat the tar out of that problem :-)

I forgot about the extended pattern constructs like: (?!pattern)
Late last night I thought that was either a typo or a perl 5.6
feature. It looks so much nicer in the waking hours.

-- Brett
Interwoven Inc.

Migrateduser

I think I posted this in another post somewhere else, but the regex that Rob worked so hard on did not work in an OpenDeploy excludePattern regex because OpenDeploy does not utilize the perl version of regex and therefore the lookahead syntax does not work for OpenDeploy regexes. Bummer. So I sat down and used some of Rob's original ideas from this thread and removed the lookahead syntax from the regex and came up with something that appears to work. So the current version of the regex that seems to work in OpenDeploy to exclude *.jhtml except do not exclude index.jhtml is this:

<excludePattern regex="(^|.*\/)(i.{1,3}|i.{5,}|i([^n][^d][^e][^x]\.).*|[^i\/][^\/]*)\.jhtml" >

There may be other ways to do this, but this one, so far, seems to work. The strange thing is that inserting this regex into the Regex Validator on the support site produces an "Invalid Input" error. Anyway, just wanted to update the thread in case anyone ever needs something like this.

I found one case where this won't work - that's when you have an intermediate subdirectory that begins with the letter "i" that contains index.jhtml. The regex will end up matching and will exclude the file. For instance:

items/index.jhtml will match and subsequently be excluded. Oh well.

I was able to fix the issue I described in red above with this regex:

(^|.*\/)(i.{1,3}|i[^\/]{5,}|i([^n][^d][^e][^x]\.)[^\/]*|[^i\/][^\/]*)\.jhtml

So far every test case passes for this.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Edited by Smitty77 on 10/25/02 01:52 PM (server time).

Migrateduser

Yikes. This was an interesting thread.

While the regex discussion was interesting to read post-gate, I was wondering why you didn't solve this with a deployment spec that has 2 source areas, both pointed at the same source area. One source would have an exclude pattern for /\.jhtml$/, and the other would have an exclude pattern for everything _but_ index.jhtml (ie, /[^i][^n][^d][^e][^x][^\.][^j][^h][^t][^m][^l]$/).

You could also have a source area that does a file list deployment, to avoid clunky regexes like my 2nd one above. That way you'd have a guaranteed set of files that are always deployed, independent of any exclude rules in other sources.

bw

Bob Walden [bob.walden@interwoven.com]
Interwoven Education Group
IM: Yahoo, MSN bob_walden

Migrateduser

Geez, Bob, where's your sense of adventure? Smiley Happy

I guess the reason I didn't go the way you suggested was because I never considered it. I'm sure that could work, I'm just not sure your regex would work...I'll have to give it a try. Interesting idea.

We do use a filelist deployment always. But the exclude rules still work on filelist files, thank goodness. This is significant because we deploy the same set of files to both our web server and app server. However you want files with certain extensions to only go to the web server and files with other extensions to go to the app server. Using exclude patterns allows us to do that with the same filelist.

Dave Smith
Sr. Software Engineer
Nike, Inc.
(503) 671-4238
DavidH.Smith@nike.com

Migrateduser

Bear in mind that with seperate <sourceFilesystem>'s in the same <source>, your excludes only apply to their own <sourceFilesystem>'s, so if you have one <sourceFilesystem> doing a dir comparison and using an exclude, it won't affect the other <sourceFilesystem> in the same <source> element from deploying a fixed list of files using file list deployment.

bw

Bob Walden [bob.walden@interwoven.com]
Interwoven Education Group
IM: Yahoo, MSN bob_walden