UTF-8 encoding not working

Hi there,
we are using TS 6.5 on Solaris.
We create HTML page snippets which are picked up and displayed by the application server.

The customer needs to use special characters (like the ë character) on the website. So, we use the following line in the datacapture.cfg:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

In the DCR every ocurrence of the ë character is stored as Ã« so the above line doesn't seem to work (or at least it doesn't work as I would expect it to).
From there on, the problem is of course passed on to the presentation template and, eventually to the web application server.

This only applies to the content of text fields. The encoding of textarea fields' content is working fine, thanks to the appropriate settings in the VF configuration file.

I would appreciate any suggestions and comments on this issue.

Find more posts tagged with

Comments

Frederik

(taking this from memory, one should really RTFM to verify ;-) )

uhm, I think DCR files are ALWAYS saved as UTF-8. The first line in a DCT has no effect on that. It simply states the encoding that the DCT file itself is in (as for any XML file)

When you see the Ã« for ë, it seems to me that Ã« is probably the latin1 representation of the UTF-8 2-byte encoding of ë.

How (with what program) are you reading the DCR file, when you see the Ã« ? If you are simply using cat or vi on solaris, I'm not surprised, as probably your Solaris OS isn't using UTF-8 as its encoding.

Sailesh

Hi there,
we are using TS 6.5 on Solaris.
We create HTML page snippets which are picked up and displayed by the application server.

The customer needs to use special characters (like the ë character) on the website. So, we use the following line in the datacapture.cfg:
In the DCR every ocurrence of the ë character is stored as Ã« so the above line doesn't seem to work (or at least it doesn't work as I would expect it to).
From there on, the problem is of course passed on to the presentation template and, eventually to the web application server.

This only applies to the content of text fields. The encoding of textarea fields' content is working fine, thanks to the appropriate settings in the VF configuration file.

I would appreciate any suggestions and comments on this issue.

Teamsite dct's by default saves every dcr as utf-8 format, you need to check if you have some OS issue?

ISCBorisB

When you see the Ã« for ë, it seems to me that Ã« is probably the latin1 representation of the UTF-8 2-byte encoding of ë.

It absolutely is. UTF-8 ë representation is two bytes: 0xC3AB, see enclosed picture. So, non-unicode software shows (correctly) Ã«.

ccprofessional

This is an example of 'transcoding'.

The OS does not store dcrs in utf-8, but rather using its own encoding (depending on the OS).

For example, a dcr with latin characters encoded in utf-8 will show up one way in the DCT or the browser, but when looked at using 'vi' it may look different.

The question to the original poster is - Where do you see the problem? In the page rendered by presentation template? or when viewing the dcr from the terminal?

ISCBorisB

This is an example of 'transcoding'.

This is not an example of transcoding! Transcoding is the direct conversion between various codeces.
Here we have the same encoding that is shown differently by different software.

tobias

Thanks for this information.

I see the "problem" in the DCR, which I open with UltraEdit. If I use UltraEdit's convert function and convert the file from UTF-8 to Unicode, the special character is displayed correctly.
This is also the case for the generated HTML page.

However, on the website the special character is not displayed properly. Instead of the ë the Ã« is shown.

Does that mean that files on the servers are okay, but the web application server isn't configured properly?

ccprofessional

to ICSBoris - are you saying my statement earlier is wrong:

The OS does not store dcrs in utf-8, but rather using its own encoding (depending on the OS).

ok, I may be using the transcoding term losely - only the last statement from tobias confirmed that.

tobias, if the generated page is using utf-8 and showing the character properly, then it means that dcr's are fine and there's no problem in the encoding within teamsite

ISCBorisB

Does that mean that files on the servers are okay, but the web application server isn't configured properly?

That depends on your site encoding. Assuming for simplicity that it's UTF-8 and no conversion is needed you can do either:

ISCBorisB

to ICSBoris - are you saying my statement earlier is wrong?

No. Your statement is impresise but basically correct.
What I'm saying is that you should not use nice sounding terminology if you are not sure what it means.

Frederik

I see the "problem" in the DCR, which I open with UltraEdit.

I suspect you are using a fairly old version of UltraEdit (UE). Until recently, I was using v7.x here, and would have the same thing: it didn't recognize automagically the charset of an XML file. I believe the v10 at home, and certainly the v12 now at work, will recognize the encoding="UTF-8" attribute of the XML header, and act/display accordingly. (could be a configuration setting involved too)...

As for your webpages, others have already answered: you have to properly configure the encoding on your webserver or html-pages.

You may also need to check into why there is a different behaviour in text fields and textarea fields... possibly your visualformat settings (< clean charencode="utf-8" ... >) are set up to produce latin instead of utf8.

Valentine

Thanks for this information.

I see the "problem" in the DCR, which I open with UltraEdit. If I use UltraEdit's convert function and convert the file from UTF-8 to Unicode, the special character is displayed correctly.
This is also the case for the generated HTML page.

However, on the website the special character is not displayed properly. Instead of the ë the Ã« is shown.

Does that mean that files on the servers are okay, but the web application server isn't configured properly?

Make sure that your web site/presentation templates give out proper "content-type" encoding.