Perl incompatibilities

We're upgrading from TS 5.5.2 to TS 6.1, which comes with a newer version of Perl.

We were having trouble Previewing a templating type (PressRelease) that is rendered via an XSL stylesheet. That is, the PressRelease Presentation template (PressRelease_www.tpl) consists of basically just this:

<my_xslt xsl_file="standard/xsl/PressRelease_www.xsl" mode="admin"><iw_include file ='$dcrPath'/></my_xslt>

Looks like my_xslt.pm was created from Interwoven's iwov_xslt.pm.

Anyway, my_xslt.pm invokes the Java-based Xalan XLST processing engine to render the content of the DCR at hand using the PressRelease_www.xsl stylesheet. Xalan was choking on at least one DCR. That's because there was a character in the DCR that was made "UTF-8 char-centric" by the new Perl. From the same KB article: "... the TeamSite Presentation Templating engine canonicalizes data as 'char-centric,' so that customers can take advantage of char-aware regexes."

So I ended up making this minor change to my_xslt.pm:

From:

print IW_XSLT_TMP_XML $_;

To:

print IW_XSLT_TMP_XML Encode::encode_utf8($_);

Xalan was pleased enough.

End of story? Heck no. That PressRelease template turned out to be a bear.

We were seeing strange characters (which weren't showing up on TS 5.5.2) when Previewing. I tried to tweak my_xslt.pm for a long time, but to no avail.

Looks like my_xslt.pm is returning the XSL transformation output as byte-centric characters, but the Presentation Template engine was assuming that they were char-centric characters. So I ended up modifying PressRelease_www.tpl as such:

<iw_ostream filter='use TeamSite: Smiley Tongue

T::my_sg; TeamSite: Smiley Tongue

T::my_sg::backToCharCentricUTF8Rep()'><my_xslt xsl_file="standard/xsl/PressRelease_www.xsl" mode="admin"><iw_include file ='$dcrPath'/></my_xslt></iw_ostream>

That is, I added a filter around the results of the my_xslt tag, to convert from byte-centric back to char-centric characters. I added the backToCharCentricUTF8Rep() function to the my_sg.pm module as follows:

sub backToCharCentricUTF8Rep {
$_ = Encode::decode_utf8($_);
} # End of sub backToCharCentricUTF8Rep {

You might rightfully wonder whether I couldn't have modified my_xslt.pm itself to return char-centric characters. Then I wouldn't have to change the Presentation template. But my_xslt.pm is frightfully complex. I'm not quite sure I understand what's going on in there.

Are you familiar enough with iwov_xslt.pm to recommend where to change it so as to return char-centric characters? More importantly, is my understanding correct that returning char-centric characters from my_xslt (via Encode::decode_utf8()) would have been beneficial? I'm not even sure I understand why those strange characters which were appearing on TS 5.5.2 started appearing on TS 6.1.

Can anyone shed some light?

Thanks,

S.

Find more posts tagged with

Comments

hgbr

Well, the problem is, that beginning with perl 5.6 perl thinks data printed to stdout should be in ISO8859-1. So if you print out your utf8 characters to stdout, perl converts them to ISO even when you want them in utf8. The solution to this problem is simple : add the line use bytes; in the script/tpl before any output occurs and perl will not modify the output.

hgbr

update:
insted of use bytes; use the pragma use open ':utf8'; and ewerything should be fine.

rodigz

hi,

where to put the use utf-8 thing??


<iw_ostream buffer='bodyBuffer'>
<iw_include pt_val='$body_hash{$curr_key}' mode='cwd'>
<![CDATA[
$iw_param{useLinkTargets} = 1;
$iw_param{showPageTitleInHeader} = 1;
]]>
</iw_include>
</iw_ostream>


thanks.

Frederik

Anyone having some fresh experiences or solutions on this one? We've just found out that we're running into the same issue on our 5.5.2 to 6.5 TS upgrade. (Just one of many

)

I've been looking at the code, and managed to reproduce the solution (iw_ostream filter=decode_utf8) of the first poster, but that would mean changing all our .tpl

-thx !

Johnny

Have a look at this post.
It explains things.

In a nut shell you need to tell PERL to read/write files as UTF-8 instead of the default iso as well as the 3 open handles STDIN, STDOUT, STDERR which is what's used when executing commands in PERL via backticks etc.


use open ':utf8';

# TeamSite files are mostly UTF-8 so we'll default our file handles to UTF-8
# Note: the user of "use open ':utf8'" above...
binmode (STDOUT, ":utf8");
binmode (STDIN, ":utf8");
binmode (STDERR, ":utf8");

John Cuiuli

Frederik

Hi all,

I finally believe I've gotten to the very bottom of this ;-)

First of all, I think some posters above have gotten the wrong impression about what the problem is

In reply to:

Well, the problem is, that beginning with perl 5.6 perl thinks data printed to stdout should be in ISO8859-1. So if you print out your utf8 characters to stdout, perl converts them to ISO even when you want them in utf8.

While this is often a problem on perl scripts, here it's not the issue. The problem I'm seeing, is that our characters are double UTF8-encoded.

Example :
ISO-8859-1 : 0xE9 (é)
from ISO-8859-1 to UTF-8 : 0xC3 0xA9
double encoded from ISO-8859-1 to UTF-8 : 0xC3 0x83 0xC2 0xA9

SUMMARY:
what happens is that <iwov_xslt> is executing any XSL transformation. That means that the inputs are an XML file (could be any encoding) and an XSL file (could be any encoding). More importantly, the output of the XSL transformation could be ANYTHING. It could be ascii text, it could be xml of any encoding, it could be binary (Word, PDF, image, ...). So <iwov_xslt> makes no assumption whatsoever about the encoding, and delivers the result in raw format (iw-perl/vendor/lib/TeamSite/PT/iwov_xslt.pm line 411 : return $xslt_data Smiley Wink

The problem then is that the Presentation Template engine, now running in Perl 5.8, is unaware of the encoding. Previously (TS 5.5, Perl 5.0.x.x ?) , it would ignore that. But in Perl 5.8, apparently it tries to convert the incoming result to UTF-8. Ignorant of the incoming encoding, it assumes ISO-8859-1 and convert to UTF-8 (thus : double encoding to UTF-8).

WHAT TO DO?
1) you could modify iwov_xslt.pm line 411 : return Encode::decode_utf8($xslt_data);

This works, but then your Generic <iwov_xslt>, previously capable of any XSLT, now only works for XSLT where the output is in fact UTF-8

2) or, inform Perl in the .tpl of the output of the <iwov_xslt> (the method described by the first poster).

<iw_ostream filter='$_ = Encode::decode_utf8($_)'>
 <iwov_xslt xsl_file='/templatedata/xsl/news.xsl'>
  <![CDATA[ ... here the XML ... ]]>
 </iwov_xslt>
</iw_ostream>

Works, but you'll have to modify all your templates, and add weird techy iw_ostream

3) or, copy iwov_xslt.pm to iwov_xslt_utf8.pm, replace all occurences of iwov_xslt by iwov_xslt_utf8, and add the solution (1) (above) to that file. Then use the tag <iwov_xslt_utf8> instead of tag <iwov_xslt>, any time you are XSLT'ing with UTF8 output. You'll still need a variant of solution 2 on output that has another encoding, probably.

Well, hope that stand the test of peer-review, and helps someone :-)

So what could Interwoven have done to make this easier on us?
Could it be fixed with a modification of the <iwov_xslt> or the PT engine? I think not... it would probably break the genericity of their code.
At least flag this issue in documentation !

cheers, Fred

Frederik

FYI : reaction from Support.

This is a known bug Bug 62530 which has been fixed in TS 6.7 , I am going to find out if there could be a fix for an earlier release, but at the moment the workaround is the one mentioned in devnet. I will let you know as soon as I find out.