UTF-8 problem with <iw_ostream>

agould

Hi,

This is with IW support but I wondered if anyone here has a view

He have the egrave char in a dcr.

Within the tpl file there is an <iw_ostream filter="HTML::Trimmer2->iw_ostream_filter">

This filter is listed below

The problem is that it is converting the unicode characters in the dcr into UTF-8 again even though the DCR is already in UTF format.

note if I comment out the code in the iw_ostream_filter sub then the dbl conversion doesn't happen.

This didn't happen on V5 but has happened on or new V6.1 upgrade

System is set to UTF-8

package HTML::Trimmer2;

use strict;
use vars qw(
%IS_NO_TRIM_TAG
);

use base 'HTML:arser';

# include here all the tags that enclose
# text that should be left untouched
%IS_NO_TRIM_TAG=map {(lc($_), 1)} qw(

pre
script

);

=head1 NAME

HTML::Trimmer2

=head1 DESCRIPTION

An HTML processor that subclasses HTML:arser to strip
unnecessary whitespace from HTML files in order to reduce
their filesize.

=head1 USAGE

In "regular" perl code:

use HTML::Trimmer2;

my $trimmer=new HTML::Trimmer2;

my $trimmedHTML=$trimmer->filter($htmlString);
my $trimmedHTML=$trimmer->filterFile($filename);

In a TeamSite presentation template:

Surround any HTML with an E<lt>iw_ostreamE<gt>

<iw_ostream filter="HTML::Trimmer2->iw_ostream_filter">
<html>
<head>
<title>This will be trimmed </title>
</head>
<body>
All the whitespace (including newlines and TAB indentation)
will be stripped from this HTML content
</body>
</html>
</iw_ostream>

=head1 METHODS

B<HTML::Trimmer2-E<gt>new>

Constructor, takes no arguments

B<$trimmer-E<gt>filter>
B<$trimmer-E<gt>filterFile>

Instance methods, both return a string containing stripped HTML;
the first takes a string in HTML format, the second a string
containing a filename which will be opened for reading and parsed.

B<HTML::Trimmer2-E<gt>iw_ostream_filter>

Static method, used in TST TPLs. Takes $_ as an HTML string, filters
it and puts the result back into $_, as per the iw_ostream spec.

=head1 VERSION

Version 0.01

=head1 AUTHOR

Steve Martina, Interwoven (c) 2002 - steve@interwoven.com

=cut

sub new{
$_[0]->SUPER::new(
unbroken_text=>1,
handlers=>{
# extra custom routine to strip whitespace from tags also
start=>[trimStart=>'self,tagname,attr,attrseq'],

# end tags untouched - unlikely to make a difference
# end=>[_bufferAppend=>'self,text'],
end=>[doEndTag=>'self,tagname,text'],

# trim whitespace in text content
text=>[trimText=>'self,text'],

comment=>[_bufferAppend=>'self,text'],
},
);
}

sub doEndTag{
my ($self, $tagname, $text)=@_;

$self->{isWithinNoTrimTag}--
if (
$IS_NO_TRIM_TAG{$tagname}
and
$self->{isWithinNoTrimTag}>0
);
if ($text !~ /<\/u>/i) {
$self->_bufferAppend($text);
}
}

sub trimText{
my ($self, $text)=@_;
unless ($self->{isWithinNoTrimTag}){
# trim leading whitespace
$text=~s(^\s+)( )msg;# and $self->_bufferAppend('Matched trim leading');
# trim trailing whitespace
$text=~s(\s+$)( )msg;# and $self->_bufferAppend('Matched trim trailing');
# compress repeated whitespace
$text=~s( (\s)+)( )msg;# and $self->_bufferAppend('Matched compress within');
# wipe all non-HTML-significant whitespace (CR, LF and TAB)
$text=~y(\r\t\n)()d;
}

$self->_bufferAppend($text);
}

sub trimStart{
my ($self, $tagname, $attr, $attrseq)=@_;

# rebuild the tag from scratch with the bare
# minimum of whitespace and append it to the buffer UNLESS it is an Underline Tag
if ($tagname !~ /^u$/i) {
$self->_bufferAppend(join(' ', "<$tagname", map(qq($_="$attr->{$_}"), @$attrseq)).'>');
}

# also arrange NOT to trim text in certain tags
$self->{isWithinNoTrimTag}++
if $IS_NO_TRIM_TAG{$tagname};
}

sub iw_ostream_filter{
my ($self)=@_;
# attempt not to die if called as a function (this is a *static METHOD*)
$self||=__PACKAGE__;
# create a default instance if necesary
$self=$self->new unless ref($self);
$_=$self->filter($_);
}

sub filter{
my ($self, $html)=@_;
$self->_clearBuffer;
$self->parse($html);
# my $spanparse = $self;
# $spanparse =~ s/<span/ <span/gi;
return $self->_getBuffer;
# return $spanparse;
}

sub filterFile{
my ($self, $file)=@_;
$self->_clearBuffer;
$self->parse_file($file);
# my $spanparse = $self;
# $spanparse =~ s/<span/ <span/gi;
return $self->_getBuffer;
# return $spanparse;
}

sub passthrough{$_[0]->_bufferAppend($_[1])}

sub _bufferAppend{$_[0]->{_buffer}.=$_[1]}
sub _getBuffer{$_[0]->{_buffer}}
sub _clearBuffer{undef $_[0]->{_buffer}}

1;