Special characters displaying wrong

I copied some content from MS document

ex: Swimin’ in dirty water (single quote from MS word)

and pasted in plain text field in the DCT (utf-8 encoding) and when saved and previewed I got some funky letters for the appostraphie, if i do not copy and write it manually everything is ok. I realized the preview tpl had iso-8859-1 encoding in the meta tags

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

i changed the charset to utf-8 and preview worked fine I did the same thing with the tpl that generates jsp content the one that actually publishes to the site and not the preview

<%@ page contentType="text/html; charset=UTF-8" %>

the funky letters disappeared but it a question mark(?) in the place of the apostraphy

the iwpt_encoding returns utf-8

and when i looked at the JSP pages in the WWW folder

<%@ page contentType="text/html; charset=UTF-8" %>

so i know it's encoding in utf-8 but it won't read teh apostraphy and it's replacing it with a question mark, however the preview works fine and shows the apostraphy. Can some help me with this issue. thanks in advance

Find more posts tagged with

Comments

Migrateduser

We've battled that same issue for years now, we just have the users fix the bad apostrophes. Hopefully someone else has a solution.

ISCBorisB

We've battled that same issue for years now, we just have the users fix the bad apostrophes. Hopefully someone else has a solution.

You may want to try this little gem (originally by Paul Graham I beleive) in your PT. I use it for years.
Make sure you read an article as well, it explains what causes MS apostrophes and quote marks issues, etc

GriffTheDog

I'm sure there is a better solution and I hope someone posts it, but what I do is:

keep a list of the character that have caused problem, expanding it as I find new ones.

I grab the content into a formAPI variable, escape the variable, do a search on known problem characters such as %u2019 for a problem single quote, replace it with an ordinary single quote, unescape the variable and write the content back to the DCT.

nipper

You may want to try this little gem (originally by Paul Graham I beleive) in your PT. I use it for years.
Make sure you read an article as well, it explains what causes MS apostrophes and quote marks issues, etc

you have to like an application called a demoroniser.

paruntha

i tried using this code to subsitiute the single quote

$xml =~ s||'|sg;
or
$xml =~ s||'|sg;

but it still didn't work I don't think the tpl is reading the single quote for it to subsititute. I even copied the version of the single quote of MS doc

$xml =~ s//’/sg; (that single quote is copied from MS word)

and it still didn't work

Johnny

Have you tried using the regular html meta tag inside your jsp page?

< meta http-equiv="Content-Type" content="text/html; charset=UTF-8" / >

Also watch your request with a http sniffer (firebug on firefox is easy enough) to view the http headers - you can confirm if the application server is actually saying that the content type is UTF-8

I copied some content from MS document

ex: Swimin’ in dirty water (single quote from MS word)

and pasted in plain text field in the DCT (utf-8 encoding) and when saved and previewed I got some funky letters for the appostraphie, if i do not copy and write it manually everything is ok. I realized the preview tpl had iso-8859-1 encoding in the meta tags

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

i changed the charset to utf-8 and preview worked fine I did the same thing with the tpl that generates jsp content the one that actually publishes to the site and not the preview

<%@ page contentType="text/html; charset=UTF-8" %>

the funky letters disappeared but it a question mark(?) in the place of the apostraphy

the iwpt_encoding returns utf-8

and when i looked at the JSP pages in the WWW folder

<%@ page contentType="text/html; charset=UTF-8" %>

so i know it's encoding in utf-8 but it won't read teh apostraphy and it's replacing it with a question mark, however the preview works fine and shows the apostraphy. Can some help me with this issue. thanks in advance

paruntha

yes i tried using the meta tags in the JSP pages but it still did not work. I am trying to figure out a way to convert the MS word double and single quotes to the ones that TPLs can recognize and print them out properly

“Hey” into "Hey"

does anybody know how to this because I tried doing this in the datacapture itself trying to look for “ or ” and change but it won't recognize it in both UTF-8 and ISO-8859-1. Thanks in advance

Adam Stoller

yes i tried using the meta tags in the JSP pages but it still did not work. I am trying to figure out a way to convert the MS word double and single quotes to the ones that TPLs can recognize and print them out properly

“Hey” into "Hey"

does anybody know how to this because I tried doing this in the datacapture itself trying to look for “ or ” and change but it won't recognize it in both UTF-8 and ISO-8859-1. Thanks in advance

You might also consider trying to "train" the users to configure Word not to use "smart quotes" and such - it will save a lot of headaches down the road and it doesn't really make the text look that bad.

ISCBorisB

yes i tried using the meta tags in the JSP pages but it still did not work. I am trying to figure out a way to convert the MS word double and single quotes to the ones that TPLs can recognize and print them out properly

“Hey” into "Hey"

does anybody know how to this because I tried doing this in the datacapture itself trying to look for “ or ” and change but it won't recognize it in both UTF-8 and ISO-8859-1. Thanks in advance

In Perl context:

$s =~ s/\x91/`/g;
$s =~ s/\x92/'/g;
$s =~ s/\x93/"/g;
$s =~ s/\x94/"/g;

paruntha

thanks i'll look into this , i have found a solution by writing the code in the TPL it self by replacing the funky letters with the corresponding smart quotes.

Surender

Hi Paruntha,

I am also facing the same problem.
Please let me know if you find any solution.

Thanks,
Surender