How to save the DCR content in UTF8 format?

Hi,

Env: Windows 2003 & TS 6.7.1

Some of the characters are not saved in UTF8 format in the DCR. It accepts the characters as it is. These characters are entered thru a TEXT FIELD or TEXTAREA(with VFE & without VFE).

For the TEXTAREA, i am using the following configuration in VFE.
<clean charencode="charref" cr="cr" lf="lf" showonsize="5000" preferfonttag="false" reducetags="true" showdonemsg="true" prompt="true" hideobject="true" mswordfilter="true">

I tried with the clean charencode="utf8", but that doesn't convert the non-break special character (Â) to   . So i am using CHARREF, but that even sometime skips the Â character.

I have specified the following in the DCT.
<?xml version="1.0" encoding="utf-8" standalone="no"?>

These are the few characters that got stored in the DCR.

“ ”  &  ‘  ’  …

How to save the DCR content only in UTF8 format? Is there anyway to specified the charset for TEXT FIELD too?

Kindly advice.

Thanks in advance.
John

Find more posts tagged with

Comments

nipper

Why is this an issue ? Is it breaking something ?

I keep special characters in my DCR all the time: Service à l’aéroport

My TPL has the correct BOM and displays that data correctly.

It gets parsed correctly with TeamSite::XMLnode->new($xml);

jstephan

Why is this an issue ? Is it breaking something ?

I keep special characters in my DCR all the time: Service à l’aéroport

My TPL has the correct BOM and displays that data correctly.

It gets parsed correctly with TeamSite::XMLnode->new($xml);

When we deploy the content to DB using datadeploy, it deployes with those special characters. And when we pull data from DB for an XML output, it breaks.And for HTML, it shows some boxes instead of the characters.

nipper

When we deploy the content to DB using datadeploy, it deployes with those special characters. And when we pull data from DB for an XML output, it breaks.And for HTML, it shows some boxes instead of the characters.

I thought Â was a valid character in HTF-8 and did not need to be encoded.

For the TPL you may not have the BOM set, use something like this in the TPL :

[html]

my $UTF8_BOM = "\xEF\xBB\xBF";
iwpt_output($UTF8_BOM);

[/html]

I also found a good regext you can use in FormAPI here

jstephan

I thought Â was a valid character in HTF-8 and did not need to be encoded.

For the TPL you may not have the BOM set, use something like this in the TPL :

[html]

my $UTF8_BOM = "\xEF\xBB\xBF";
iwpt_output($UTF8_BOM);

[/html]

I also found a good regext you can use in FormAPI here

Thanks Nipper for your responses.
But we are not using any TPL. Just datadeploy the DCR contents to DB. So i am not sure how to implement the TPL fix?

nipper

Well if the DB is not complaining about invalid data then you can put the BOM in the page being processed (JSP/ASP/ASPX). I have placed those 3 HEX characters in the very beginning of an ASPX and it works without any issues.

jstephan

Well if the DB is not complaining about invalid data then you can put the BOM in the page being processed (JSP/ASP/ASPX). I have placed those 3 HEX characters in the very beginning of an ASPX and it works without any issues.

While inserting the data into DB, the log file shows the data are inserted as it is. But while we retrieve the data, its not displaying as in DCR thou we are using the BOM in the frontend. The DB is set to accept the UTF8 format and regular insert works well in DB.

As the characters are not converted to UTF8 format in the DCR, the datadeploy inserts the content as it is and DB stores the value as special characters. So i am looking for a way to store the values in UTF8 format in the DCR. How to store the data in UTF8 in DCR?

Valentine

I believe there is also charencode="raw"

jstephan

Is there anyway for TEXT FIELDS to specify the UTF8?

jstephan

Is it possible to convert to UTF8 before we save the DCR thru Form API?
Anyone tried to specify the TEXT FIELDS in UTF8?

nipper

I do this in Perl all the time, never had to with java. That being said, there must be a way to do it with either formapi or with a datadeploy preprocessor, I would think the latter being the better approach.

jstephan

I do this in Perl all the time, never had to with java. That being said, there must be a way to do it with either formapi or with a datadeploy preprocessor, I would think the latter being the better approach.

DataDeploy preprocessor : Writing a custom tuple processor to convert the content to support UTF8?

Have anyone did this kind of tuple processor to convert to UTF8?
So there is no way right now in the Teamsite build-in functions to complete the task in DCR level?!!

nipper

maybe, maybe not. I do not see this as a TeamSite issue. Nor is it a DB issue. You said the is inserted properly in the DB, it is the application that reads the data is not displaying correctly. I do this all the time with ASPX files and it displays the non-encoded data correctly.

FurnaceFace

@ jstephan

For the VF values that are not being changed, I'm not sure if it was finally resolved for you but I do know that if you change the "UNICODE" value within the visualformat code to "true", Non breaking spaces are converted to

I believe (it's been a while) that your curly quotes will also be converted to their numerical values as well.

Hope this helps,
/FF

Evani20

I am not sure this issue has been resolved or not, still, I did something like putting a CDATA after *value* in DCR. It saves from stripping any special character. Let me know if you have any question regarding this.

Good luck.

jstephan

I am not sure this issue has been resolved or not, still, I did something like putting a CDATA after *value* in DCR. It saves from stripping any special character. Let me know if you have any question regarding this.

Good luck.

Hi Evani, could you please elaborate your method. I am not quite following how to use CDATA after *value*. Is that something like <tag><![CDATA[the value of the tag]]></tag>?
And kindly explain about how to implement?

jstephan

@ jstephan

For the VF values that are not being changed, I'm not sure if it was finally resolved for you but I do know that if you change the "UNICODE" value within the visualformat code to "true", Non breaking spaces are converted to

I believe (it's been a while) that your curly quotes will also be converted to their numerical values as well.

Hope this helps,
/FF

Thanks FF. <viewas enabled="true" publish="cleanhtml" mode="body" unicode="true"> - This is what i have on the vfe config. But still the non-breaking spaces are not get converted. Also is there any way to specify for a TEXT FIELD?

Evani20

Hi, sorry for that confusion; no CDATA in DCR should be...

[html]
evan
222-333-3333
cell

[/html]

What I was doing I use a callServer on onSaveDone event in order to call a perl script, now that perl script will read that DCR and add CDATA right after the *value* tag as you see on the top. Let me now if you still have question about this.

Thanks and good luck,

jstephan

Hi, sorry for that confusion; no CDATA in DCR should be...

[html]
evan
222-333-3333
cell

[/html]

What I was doing I use a callServer on onSaveDone event in order to call a perl script, now that perl script will read that DCR and add CDATA right after the *value* tag as you see on the top. Let me now if you still have question about this.

Thanks and good luck,

Thanks Evani for your reply. You are saving the DCR in IWOV type. I am using regular XML. And i can use the CDATA.