Encoding problem in perl

[color=black]Hi,[/color]

[color=black]I have a callout from a DCT/form which calls a perl script. This script connects to MSSQL database, reads data and populates it in a pop-up window. Everything is working fine except that the special characters (like chinese/japanese language characters) are NOT getting displayed properly (?? are getting displayed instead). In other words, the encoding is getting lost.[/color]
[color=black]

[/color]

[color=black]The special characters in the database looks fine. The DCT and the html pop-up window are UTF-8 encoded. So, its the perl script thats causing the issue.[/color][color=black] [/color]
[color=black]

[/color]

[color=black]I know that this type of encoding issues are discussed several times on DevNet but, all/most of them are regarding to ORACLE DB or setting up proper environment variables in the script. Since I am using SQL DB, am wondering is there any environment variable or special code that I need to be aware of.

[/color]
[color=black]

[/color]
[color=black]I am attaching the perl script for you reference purposes. Thanks in advance!!

[/color]
[color=black]

[/color]
[color=black]Environment: TS 6.7.2 SP2 on Windows 2003[/color]

Find more posts tagged with

Comments

Jamik

are you positive your database is encoded UTF-8? did you look at the db properties to verify that?

ISCBorisB

Chances are, you are getting your SQL values in UTF-16LE, not UTF-8.
It is easy to check if the return is a valid UTF-8, look for example here.

BlueWhale

I got the information from the sql admins that the DB is installed with default encoding for sql and has the collation: SQL_Latin1_General_CP1_CI_AS and they say that this represents UCS-2 and/or UTF-16.

Considering above the case, please let me know what kind of changes needs to be done to the script?

Thanks.

ISCBorisB

I got the information from the sql admins that the DB is installed with default encoding for sql and has the collation: SQL_Latin1_General_CP1_CI_AS and they say that this represents UCS-2 and/or UTF-16.

Considering above the case, please let me know what kind of changes needs to be done to the script?

Thanks.

You'd have to encode your raw DB Data to UTF-8, see the sample below. Note that it has not been tested.
Note also that you should not try to run that just once to "transcode" the whole result, it won't work. The reason
is that right now what you really generate is a mix of ISO 8859-1 (aka Latin-1) and UTF-16

use Encode;
eval {Encode::from_to($your_string, 'utf-16le', 'utf-8', Encode::FB_CROAK);};
if ($@) {
   my $error_message = qq(Can Not Encode to UTF-8: $@);
   # Log out $error_message
}

BlueWhale

Thanks Boris.

When I do the encoding conversion from 'utf-16le' to utf-8', I am getting the following error:

Can Not Encode to UTF-8: Unknown encoding 'utf-16le' at

\Programs\Interwoven\TeamSite\httpd\iw-bin\test_db_enc_script.ipl line 186

But, when converting from ucs-2 to utf-8, there are no such errors but, the characters still doesn't display well.
Below is the updated code that I am testing with:

 
my $count=0;
foreach my $row (@items)
{ 
  my $id = @$row[0];
  my $name = @$row[1];

  $logger->debug("INITIAL ID: $id");
  $logger->debug("INITIAL Name: $name");
  eval {Encode::from_to($name, 'utf-16le', 'utf-8', Encode::FB_CROAK);};
  #eval {Encode::from_to($name, 'ucs-2', 'utf-8', Encode::FB_CROAK);};

  if ($@) {
    my $error_message = qq(Can Not Encode to UTF-8: $@);
    $logger->debug("$error_message"); # Log out $error_message
   }


print <<"END";
$name
END
$count=$count+1;
}

Do I need to do anything more or am I missing something here?

ISCBorisB

Thanks Boris.

When I do the encoding conversion from 'utf-16le' to utf-8', I am getting the following error:

Can Not Encode to UTF-8: Unknown encoding 'utf-16le' at \Programs\Interwoven\TeamSite\httpd\iw-bin\test_db_enc_script.ipl line 186

Try to drop LE ( Little Endian ) Suffix, eg use just utf-16. See if that helps

BlueWhale

hmmm...didn't work out Boris.

Result: Can Not Encode to UTF-8: Unknown encoding 'utf-16' at

\Programs\Interwoven\TeamSite\httpd\iw-bin\test_db_enc_script.ipl line 188

Am wondering is there any environment variable that I need to set.

ISCBorisB

hmmm...didn't work out Boris.

Result: Can Not Encode to UTF-8: Unknown encoding 'utf-16' at \Programs\Interwoven\TeamSite\httpd\iw-bin\test_db_enc_script.ipl line 188

Am wondering is there any environment variable that I need to set.

I've used something like the following without any problems in one of my Projects. Please note that:

- it is a reverse of what you need
- Inconsistent Capitalization and dash absence/presence (utf8 - Lower Case, no dash UTF-16LE, Capital with Dash). Try to use it exactly like this.
- To check encoding Modules included into your Version get to Encode::Config.pm, see %ExtModule Hash. Look at the Code, not perldoc

use Encode 'from_to';
. . .
eval { Encode::from_to($rs, 'utf8', 'UTF-16LE', Encode::FB_CROAK); };
if ($@) {
# Report Problems
}