nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

VFE inserts BOM ?

nipper

TS 6.5 SP2 on Windows. Moved servers from an old box to a new one. Moved all configurations (including http/iw/visualformat.xml - which was unchanged since 2005).

One (or more) DCTs with a VFE are inserting a BOM (byte order mark) in the DCR XML.

THe DCT has no configuration: textarea required="f" rows="26" cols="73" wrap="virtual" external-editor="visualformat"

Any ideas ?

Find more posts tagged with

Comments

ISCBorisB

TS 6.5 SP2 on Windows. Moved servers from an old box to a new one. Moved all configurations (including http/iw/visualformat.xml - which was unchanged since 2005).

One (or more) DCTs with a VFE are inserting a BOM (byte order mark) in the DCR XML.

THe DCT has no configuration: textarea required="f" rows="26" cols="73" wrap="virtual" external-editor="visualformat"

Any ideas ?

Simple check first... Are you sure that's not an artifact of whatever Editor your Client is using to see the XML?
Create and save brand new DCR and without opening it up in any Editor, run from the Server:

type Y:\FullPath

Do you see BOM's "garbage" three bytes right before first "<"?

nipper

It is actually not the BOM (though it certainly looks like it).

The DCR has 3 Hex characters, FF FE 20
then it has some ektron junk <meta.... i="" can="" strip="" out="" all="" bu="" the="" 1st="" three="" character="">

ISCBorisB

It is actually not the BOM (though it certainly looks like it).

The DCR has 3 Hex characters, FF FE 20
then it has some ektron junk <meta....>

I can strip out all bu the 1st three character, I have tried"

$body =~ s|\xff\xfe|;

No go, I should be able to use ^\xff\xfe because it is the 1st characters of the field but nothing is working.

WTF ?

Strange... Something like the following works flawlessly on this little test:

my $body = "\xff\xfe whatever";
print qq(before - $body\n);
$body =~ s|^\xff\xfe||;
print qq(after - $body\n);

Try to convert $body to hex to make sure that you are matching what's "really" there

(my $bodyX = $body) =~ s/(.)/sprintf("%X", ord($1))/eg;
print qq($bodyX\n);

Btw, 0xFFFE is a BOM - only for UTF-16LE, not UTF-8. Perhaps your users copy/paste from some UTF-16 encoded source?

nipper

OK, this is driving me crazy.

Here is the code:


my $xml = _getXML($file);
 $rootnode = TeamSite:CRnode->new($xml);
 $body  = $rootnode->value('body');
(my $bodyX = $body) =~ s/(.)/sprintf("%X", ord($1))/eg;
print "Pre regex \n";
print qq($bodyX\n);

$body =~ s|^\xfe\xff||;
(my $bodyY = $body) =~ s/(.)/sprintf("%X", ord($1))/eg;
print "Post regex \n";
print qq($bodyY\n);

print $body."\n";

Here is my output:

[html]

Pre regex
FEFF203C4D455441206E616D653D2247454E455241544F522220636F6E74656E743D224D5348544D
4C20382E30302E363030312E3138393339223E3C4C494E4B207469746C653D222F69772F65776562
6564697470726F32302F656B746E6F726D616C2E637373222072656C3D227374796C657368656574
2220687265663D222F69772F657765626564697470726F32302F656B746E6F726D616C2E63737322
3E4869
Post regex
FEFF203C4D455441206E616D653D2247454E455241544F522220636F6E74656E743D224D5348544D
4C20382E30302E363030312E3138393339223E3C4C494E4B207469746C653D222F69772F65776562
6564697470726F32302F656B746E6F726D616C2E637373222072656C3D227374796C657368656574
2220687265663D222F69772F657765626564697470726F32302F656B746E6F726D616C2E63737322
3E4869
Wide character in print at test.ipl line 16.
∩╗┐ <meta name="GENERATOR" content="MSHTML 8.00.6001.18939" />Hi
[/html]

nipper

So the regex did not do squat. I think you are correct on the 0xFEFF may be the UTF-16LE BOM, but why VFE is inserting that I have no clue. There was no cut and paste, I typed Hi in to the VFE.

Adam Stoller

Perhaps it's a question of interpretation - you could try a simpler / dumber approach like:

while ($body !~ /^</sm) {
    $body =~ s|^.||;
}

This is based on the premise that the first "legal" character in an HTML file is an '<' - in the case of a VFE you might have to expand that to something like:

while ($body !~ /^[-<a-z0-9]/smi) { ... }

Or something like that.

ISCBorisB

So the regex did not do squat. I think you are correct on the 0xFEFF may be the UTF-16LE BOM, but why VFE is inserting that I have no clue. There was no cut and paste, I typed Hi in to the VFE.

"Wide character in print at..." may perhaps give us a clue...

The string is already recognized as the "whole sequence" of Octets, not just bytes.
Try $body =~ s|^\x{fffe}||; or perhaps $body =~ s|^\x{feff}||;
(It's UTF-16LE and UTF-16BE BOMs respectively, you've indicated both in different parts of this thread)

I have no idea what setting or default makes VFE insert this garbage, never saw anything like it Smiley Sad

nipper

Thanks Bo, the \x{FEFF} worked. No clue what is causing it.

Frederik

This looks a lot like the garbage that gets inserted in VFE if the *client* Windows computer has a security fix applied that was released in Q4 (Sep?) 2009.

I've not heard of this as a result of moving *server*.

There was a storm on devnet back then, because of how it broke VFE on 6.5, and Interwoven refusing to fix it. Their response was "move over to 6.7". Great round-about way of announcing that 6.5 was EOL.

https://customers.autonomy.com/support/secure/docs/FAQ/Interwoven/63553.html

ISCBorisB

This looks a lot like the garbage that gets inserted in VFE if the *client* Windows computer has a security fix applied that was released in Q4 (Sep?) 2009.

I've not heard of this as a result of moving *server*.

There was a storm on devnet back then, because of how it broke VFE on 6.5, and Interwoven refusing to fix it. Their response was "move over to 6.7". Great round-about way of announcing that 6.5 was EOL.

https://customers.autonomy.com/support/secure/docs/FAQ/Interwoven/63553.html

You've nailed it. Good catch!

The Field  mentioned in the article is after all 0xFEFF. Move itself probably did not expose the problem.
New/Different Contributors *might* have had MS Patch 956844 installed while existing Group has not, etc

Frederik

out of curiosity, are you planning to deal with this, and how (remove in formapi, iwat, wf task, tpl, batch, ... or uninstall security patch, ...)?

We've got this in a number of websites (for which EOL is around the corner, though undefined) managed in TS65 and left it alone for now. The effect after rendering to html is annoying (extra whitespace) but not bad enough to spend money on fixing it.

nipper

Well I hope that an upgrade to 6.7.2 will fix it. In the mean time, we updated the TPL to regex out the characters. Built a PM to filter out stuff so we can remove it.

Andy