XML Parser problem

We have migrated from TS 5.5.2 to 6.5 on solaris 8.

We are facing problem with XML parser module when parsing dcrs containing polish characters.

If the dcr contains a polish charater "ó" in the string as TESTóTEST. We use our own perl module with the help of XMLParser.pm and try to retrieve content of the dcr.

In both version TS 5.5.2 and 6.5 , value stored in dcr is TESTÃ³TEST

In Teamsite 5.5.2, dcr node value from XMLparser.pm is returned as TESTÃ³TEST, when seen in the browser is displayed as TESTóTEST
but in Teamsite 6.5, dcr node value is returned as TESTóTEST ,when seen in browser is displayed as TEST????T

is this problem because of change in the perl version

-Ganesh

Find more posts tagged with

Comments

Migrateduser

Hi,

Could you tell us which encoding are you using and also please paste us the perl module line which opens the file and parses it.

-Rahul.

cute_guy

Encoding is UTF-8 and here is the code to parse dcrs


sub parse_dcrs
{
 my ($vpaths,$hash) = @_;
 # create the XMLparser object.
 my $parser = TeamSite::XMLparser->new();
 # make sure that a reference was passed.
 if(!(ref $vpaths) || !(ref $hash)) { return "ERROR: Bad reference"; }
 # loop through the list of vpaths and process.
 foreach my $vpath (@{$vpaths})
 {
# if the hash already contains this file, skip it.
next if $$hash{$vpath};
# store this round's list of dcrs here.
my @dcr_list = ( ); 
# if it's a directory.
if(-d $vpath)
{
  # read a grab only files ending in '.dcr'
  opendir DCRHNDL, $vpath or return "ERROR: Failed to open directory $vpath ($!)";
  push @dcr_list, (grep /\.dcr$/, readdir DCRHNDL);
  closedir DCRHNDL;
}
elsif(-f $vpath)
{
  $dcr_list[0] = $vpath;
}
foreach my $dcr (@dcr_list)
{
  # if necessary, add on the vpath.
  if(!($dcr =~ /^\//))
  {
$dcr = "${vpath}/${dcr}";
  }

  my $dcr_file_data = "";

  # read the file.
  open HNDL, $dcr or return "ERROR: Failed to open file $dcr ($!)";
  while(HNDL)
  {
  $dcr_file_data .= $_;
  }
  close HNDL;
  # parse the file.
  my $rootnode = $parser->parse($dcr_file_data);
  # make sure the parse was successful.
  if(defined $rootnode) 
  {
my @items = $rootnode->list('*');  


# loop through the items and populate the hash.
foreach my $item (@items)
{

#Value from DCR is read
  $$hash{$dcr}{$item->type()} = $rootnode->value($item->type()); 

  # XML is automatically escaped by the XML:arser. 
  # We don't want them to mess with our en-****-y refs 

  $$hash{$dcr}{$item->type()} =~ s/\Q&#/&#/gs;
}
  }
  else { return "ERROR: Could not parse file $dcr"; }
} 
 }
undef $parser;
 return "";
}

Migrateduser

Hi,

try to open the handle of the file specifying the encoding as utf 8

open HNDL, $dcr

to

open HNDL, ">:utf8" , $dcr

Migrateduser

Hi,

try to open the handle of the file specifying the encoding as utf 8

open HNDL, $dcr

to

open HNDL, ">:utf8" , $dcr ##check the syntax, dont remember off-hand.

but specifying the encoding should resolve the issue. If this doesd not work out then let us know.

-Rahul

Adam Stoller

...
  open HNDL, $dcr or return "ERROR: Failed to open file $dcr ($!)";
  while(HNDL)
  {
  $dcr_file_data .= $_;
  }
  close HNDL;
...

Try changing that to:

...
  open(HNDL, '<:utf8', $dcr) or return "ERROR: Failed to open file $dcr ($!)";
         $dcr_file_data = do{local $/; <HNDL>};
  close HNDL;
...

The first change (ignore he parens if you wish, I like having them) is to explicitly state that you want to read the file in UTF-8 mode - which might fix your problem.
NOTE: the direction of the angle bracket in the 'open' statement - everyone else who responded thus far had it pointing the wrong way!

The second change is just more efficient than a 'while' loop.

Migrateduser

Hi,

thanks for correcting the angle bracket ghoti, as I mentioned I dont remember it off-hand since I provided him the solution in the hurry as I was about to leave. But I seriously believe that only the correct info. must be passed on the thread.

Once again thanks for correcting me!!

Rahul.

cute_guy

Even after changing the code as said by Adam, there is no change in the value read from the dcr it shows TESTóTEST and not TESTÃ³TEST.

However i tried to read the dcr in binary mode as below



open HNDL, $dcr or return "ERROR: Failed to open file $dcr ($!)";
# open (HNDL,'<:utf8' , $dcr ) or return "ERROR: Failed to open file $dcr ($!)";
  binmode(HNDL);
  while(HNDL)
  {
  $dcr_file_data .= $_;
  }
  close HNDL;

Now the values from dcr is read as TESTÃ³TEST as expected, but when this xml is passed to the XMLparser to be parsed to obtain only the value

 

#Value from DCR is read
$hash{$dcr}{$item->type()} = $rootnode->value($item->type());

The value returned is TESTóTEST and not TESTÃ³TEST.

-Ganesh Smiley Happy

cute_guy

Thanks Guys for your agile response..

The Cause of the problem was the perl difference...
there are actually 2 distinct ways in which a $scalar may contain "utf8":

o UTF-8 as a dumb byte-centric rep (the old way)(Perl 5.005_03)
o UTF-8 as internal char-centric rep (the new way)(Perl 5.8.x)

https://support.interwoven.com/kb/kb_show_article2.asp?ArticleID=51304

So to fix this in our code we used the below piece of code once dcr is read.

$dcr_file_data = Encode::encode_utf8($dcr_file_data);

Thanks,
- Ganesh :cool:

Adam Stoller

Now the values from dcr is read as TESTÃ³TEST as expected, but when this xml is passed to the XMLparser to be parsed to obtain only the value
#Value from DCR is read
$hash{$dcr}{$item->type()} = $rootnode->value($item->type());
The value returned is TESTóTEST and not TESTÃ³TEST.

-Ganesh

This may be a strange question - but why is it that you are accessing type and getting what appears to be a value? Shouldn't type be something like 'text', 'textarea' or something generic like 'item', 'replicant', 'container'?

Can you post the DCT and DCR that's being processed?