BOM in CS16 Text Documents

I see a change was made with how text documents are saved in CS16 (via llnode.NodeAddVersion() on $TypeTextDocument). There's now a hardcoded insertion of a byte-order marker (BOM) for text/plain mimetypes like this:

if ( version.pMimetype == "text/plain" )    dataLen += 3    bom = Bytes.Allocate(3)    Bytes.PutU1( bom, 0, 0xEF, Bytes.InPlace )    Bytes.PutU1( bom, 1, 0xBB, Bytes.InPlace )    Bytes.PutU1( bom, 2, 0xBF, Bytes.InPlace )end::written = DAPI.VerStreamWrite( stream, bom )

Why was this added and why is it limited to text/plain mimetypes? Furthermore, the BOM insertion doesn't check if the BOM already exists and will therefore keep adding additional BOMs on each consecutive save.

Find more posts tagged with

Comments

Hans Stoop

Hi Chris,

I thought that text document are just normal documents. The special node type is only used for adding a special entry in the Add Item list.

Hans

From: eLink Entry: Content Server Development Forum [mailto:development@elinkkc.opentext.com]
Sent: Montag, 26. September 2016 17:35
To: eLink Recipient
Subject: BOM in CS16 Text Documents

BOM in CS16 Text Documents

Posted by Meyer, Chris On 09/26/2016 11:29 AM

I see a change was made with how text documents are saved in CS16 (viallnode.NodeAddVersion() on$TypeTextDocument). There's now a hardcoded insertion of a byte-order marker (BOM) fortext/plain mimetypes like this:

if ( version.pMimetype == "text/plain" )

    dataLen += 3

    bom = Bytes.Allocate(3)

    Bytes.PutU1( bom, 0, 0xEF, Bytes.InPlace )

    Bytes.PutU1( bom, 1, 0xBB, Bytes.InPlace )

    Bytes.PutU1( bom, 2, 0xBF, Bytes.InPlace )

end

written = DAPI.VerStreamWrite( stream, bom )

[To post a comment, use the normal reply function]
Forum:	Content Server Development Forum
Content Server:	Knowledge Center CS16

David Templeton

Chris

You are correct, the existence should be checked before adding the BOM, listed as LPAD-51600. As for the text/plain, it is the only type in the list that needed to indicate being UTF-8. Unless there is a reference that the others need it.

David

Chris Meyer

Hi Hans,

I thought that text document are just normal documents. The special node type is only used for adding a special entry in the Add Item list.

That's true, but it's also used when editing and saving text documents and custom views from the online text editor. If you look at the TextEdit2 webNodeAction:

Object llnode = $LLIApi.LLNodeSubsystem.GetItem($TypeTextDocument):result = llnode.NodeAddVersion(node, addVerInfo)

The BOM gets inserted in the llnode.NodeAddVersion() call.

In a similar way I use this function to add new versions with the content of a string. It worked well, but this change now forces the BOM characters and causes other problems with my module. It should be made into a parameter (and defaulted to false for backwards compatibility).

Either way, the ckeditor text editor needs to be updated to accomodate this. If you edit a text document with a BOM marker there will be an invisible first character. You'll notice the problem if you place the cursor at the first position of the document and try to move the cursor to the right of the first document character. It takes two key strokes (one to move passed the invisible character and one to move passed the document character).

A better solution might be to place all BOM logic in ckeditor editor and not force it on the server. This would address both problems mentioned above.