VFE, UTF-8, and special characters

[color=black]Hey Everyone,

[/color]

[color=black]

[/color]

[color=black]This is a general question regarding a behavior we are experiencing with our VFE’s in teamsite. We are using TS 6.5 SP2 in one environment and SP1 in another and we seem to experience the same issue.

[/color]

[color=black]When the user enters information in the VFE, it is being placed in the DCR, but it seems like certain spaces (i.e. after links, etc’) are not interpreted as spaces rather as \302\240 which in turn is displayed on the page as Â. The weird thing is that our generated page, which contains the problematic characters, \302\240, gets deployed to both a web server and an app server. The web server view is fine, and the spaces come out as expected. It is in the app server view that we see Â instead of spaces. [/color]

Now, it seems like the junk characters are already in the DCR, so we need to get rid of them at that level. We currently have a regex to do that, but I am not happy about this solution. Has anyone else experienced this issue, where VFE’s introduce junk characters into the DCR’s?

I have looked on similar postings and all our encoding for the VFE are utf-8. We have also tried to solve this on the app server side by imposing utf-8 encoding in the app server via jsp directives, but that does not seem to help because the characters are already embedded in the file.

Any ideas?

Thanks,

Ido

Find more posts tagged with

Comments

Migrateduser

Are you sure it's the VFE that's at fault? Is your user copying and pasting from another program, like Microsoft-anything, for instance?

Migrateduser

Nope. User used to copy and paste and that caused an error, but we ended that ;-)
Now the user types in the information "manually" but we still get junk characters.

Migrateduser

Hmm... I *think* the \302, etc. is EBCDIC (Extended Binary Coded Decimal Information Code) and is pretty much found only on mainframes.

I won't bother to ask if your user is accessing the TeamSite application from an HP mainframe because I can just guess as to the answer and Jeff will say once again that I'm on crack, so I'll skip it.

I'd be interested to know if the user is causing these characters or if something in your visualformatconfig.xml or character transaction is doing it. Either way, why are you against solving the problem with a regex? You don't even need to rely on Perl -- you could fix it with a regex in FormAPI, before the DCR gets saved.

Migrateduser

Hey, here's a question -- you didn't mention either way but are you on a Unix platform and your user is on Windows? Of course, this is a common scenario but the \302 character is actually not used but \240 translates into an empty square graphic which I've noticed frequently when a carriage return is misappropriated which can frequently occur when a file is edited in Windows but viewed on Solaris (I don't think I've ever noticed it the other way around).

Migrateduser

Hey Dave,

Thanks for your reply. Yes, you are correct...the user is using windows and teamsite is on a unix platform.
As far as why I am opposed to using a regex - it's not that I am against it, but I feel like a regex is a point solution to this problem, and there has to be a more concrete (if you will) solution for this type of problem. From the postings I have read, it seems like a lot of other users are faced with similar problems in VFE's, and they also take the regex approach. My question is, why does the webserver display the content fine and the app server does not? We tried to fix this on the app server but that did not work, and that's why I want to try and see if we can fix it from the teamsite end by eliminating the junk characters from the DCR's (through possibly a solution which does not involve a regex).

Thanks,
Ido

Migrateduser

It's just a characteristic of control characters and other meta characters -- that is, they don't necessarily show up as anything or appear as invisible (hmm, is it a contradiction if something were to "appear invisible"?) However, if one were to view the source code in the VFE, I believe that the character(s) would be apparent provided the content were pasted in WYSIWYG mode.

Think of the source code view as a text-only repository. Theoretically, nothing can hide from it. I know, that doesn't help your situation.

Regexes are perfectly fine to use but let me ask you this: are you translating only the characters you come across (i.e. \302, \240) or are you translating the entire class?

Migrateduser

We are only checking for the characters we came across.
We use the following regex to do so: $decp =~ s!\302\240|Â ! !g

Migrateduser

Okay, then I can see why you don't want to use that approach.

Check out this link: http://seth.positivism.org/man.cgi/1/perlebcdic It guides you through how to take care of all of these characters with one regex and an uncomfortably long string.

Dave

Migrateduser

Okay.. I like the approach of this regex much more. I'll take a look in more detail later on, and I'll post here again incase I have any questions. Thanks again for all the help.

Johnny

Hey Ido,

It's not EBCDIC or anything like that. They are valid Unicode/UTF nonbreaking space characters. It is a double byte character - hence you see two separate characters if you view the content as non UTF (EG ISO 96....).

You'll find a few posts on the forum regarding VFE and it's habit to pad text with nonbreaking space. It seems to be a bug with VFE.

If you make sure that all your content points (web servers, scripts that may read the text etc) are all specified to read and display UTF then you should have no problem.

What exactly "does not work" with your app server. Can you verify that you are setting it up correctly to parse the content as UTF and then subsequently display it as UTF to the browser. I would speak with BEA support to get more help there.

Johnny

Just be sure to note that there are at least two points where you may have an issue with the portal.

1. The portal server parses (reads) all content before sending it to the client.
2. The portal server passes http headers specifying the encoding to the webserver which then should pass onto the client. Swapping between the portal and webserver might be another issue (3rd).