The diacritical characters

I have users who cut and paste in to dcrs., You can get some pretty funky chars in them.

I'm using UTF-8 encoding
They have a single quote that comes in as garbage â€™

I'm doing a
sub doConversion
{
my ($content) = @_;
$content =~ s/â€™/\&\#8217\;/g;

return $content;

}

What's your thoughts?

Thanks in advance!
WIN2K TS/TST 5.5.2 sp5; OD 5.6.0.1; DD 5.5.2
The Professor-

Find more posts tagged with

Comments

lhdavis

One issue with this approach is that's it's not only the Microsoft Word single quote (which I think is the one you're referring to) that displays as garbage. I've also seen it with the mdash and several other Microsoft Word characters.

I'd definitely be interested in how other people have resolved this. In the past, I've been told content contributors to use WordPad to keep formatting but avoid some of these special characters or just use NotePad and format in the VisualFormatEditor (if enabled).

Luke Davis
Open Technology Group, Inc.
luke.davis@med.va.gov

iwovGraduate

Do you see the "funky" chars in the DCR itself ? or when you generate a page ? or/and when you deploy to a DB ?

The code you pasted looks like Perl. Is this in your PT ?

Migrateduser

I would put the cleanup in JavaScript or VFE config because if you put it in the PT, then other downstream systems that read the DCR (datadeploy, etc.) will still see the microsoft markup. It's better not to store that garbage to the DCR in the first place.

WORK_XML_Report.jpg

ProfessorX

The funky chars are in the dcr.

The code above is in a inclue pm (because I will probably use it other places as well) in the tpl.

I found out as well that if I use VF and view&save as HTML I get ’

But I cant very well tell the CC to save all thier DCR's viewing in html mode now can I?

Thanks in advance!
WIN2K TS/TST 5.5.2 sp5; OD 5.6.0.1; DD 5.5.2
The Professor-

bboyle

VisualFormat can convert those characters to entities which works pretty good for us: <clean charencode="charref" ... (in visualformat.xml configuration file).

And in a script that extracts existing HTML content to create DCRs we use this code (which may help):

# character encoding
if ($_ =~ m/[0x80-0x9F]/) {
# assume windows encoded?
$_ = TeamSite::I18N_utils::utf8_normalize_string('WINDOWS1252', $_);
} else {
# assume ISO-8859-1 encoded?
$_ = TeamSite::I18N_utils::utf8_normalize_string('ISO-8859-1', $_);
}

Good luck!