nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

copying metadata behavior - browser dependent?

Adam Stoller

Environment:

Preface:

I started working on trying to fix something that my customer had noticed with one of our customizations, and I've run into a strange aspect upon which I'm hoping someone here can shed some light...Background:

Scenario:

Conundrum

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Find more posts tagged with

Comments

Dwayne

What is the immediate source of the metadata? I know that the "true" source of the metadata is the EA data from the selected file. But how is that data transferred to the target? Is it being stored as hidden variables within the HTML form, or is the underlying CGI code directly reading the EAs and writing them in one step?

If the former, then is there any way to change it to the latter? And, if you're already doing the latter, then I'm really stumped. The browser shouldn't have any impact on a strictly server-side process, unless you're changing behavior based on the client. And I assume that's not the case here.

--

Current project: TS 5.5.2/6.1 W2K

Dwayne

Never mind - I'm stupid. Of course you can't do what I described, as this is a metatagger callout, and is not setting the data directly.

--

Current project: TS 5.5.2/6.1 W2K

Adam Stoller

Good question. I didn't write this code originally - but looking at it I believe it's using iwextattr directly to *grab* the EAs and stick them into a hash, and then it's using javascript to populate all the fields in the Set Metadata form with the values stored in the hash. (the idea is that we don't commit the copying of the metadata based on the callout - we only populate the fields and wait for them to explicitly click the Set Metadata button at the bottom of the window to commit the changes)

So - the suspect here is that the javascript handling in FireFox is UTF-8 aware, and the javascript handling in IE is *not* UTF-8 aware? Hmm, I suppose that would explain things, but it sure would be nice to understand if/how that's possible or if there's something I can "fix" in the javascript portion of the code to explicitly make it UTF-8 compatible.

Ideas?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Adam Stoller

MetaTagger is *not* involved here - this is the old 5.5.2 OOTB Set Metadata GUI

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Dwayne

It seems to me that there are two possible points where this could be failing:

Dwayne

Well, regardless of whether it's Metatagger or the OOTB tagging GUI, you still need to pass the actual metadata values around from within the HTML page, so the solution I was trying to raise in my original post is still a moot point.

--

Current project: TS 5.5.2/6.1 W2K

brandon1

What is the doctype of the HTML page? Have you tried to change it to strict or to transitional?

Current Project: TS 5.5.2 to 6.1 Solaris, Content Services implementation

Dwayne

Here's something else I found while digging around. It's actually referring to a slightly different problem, but it may lead you down some paths that may prove fruitful:

http://www.worldtimzone.com/res/encode/

The salient part of the page is that it claims the JavaScript function escape() doesn't handle UTF-8 well. I don't know if you're using escape() or not, but it may be something to look at.

--

Current project: TS 5.5.2/6.1 W2K

Edited by Dwayne on 08/12/04 08:43 AM (server time).

Adam Stoller

I tried to attach a JPG with a bunch of snapshots in it, to try and clarify things, but the image is too big, so I'll have to try and capture much of what was in it, here.

First, the actual code (as opposed to the View Source shown in the JPG)
(note, this is abridged - sections of code, blank lines, comments, removed for brevity)sub print_header {
print qq(Content-type: text/html

<HTML>
<HEAD>
<TITLE>Save Selection</TITLE>
<SCRIPT LANGUAGE="JavaScript">
<!--

function set_datacapture_item_value() {
if ((window.opener == null) || (window.opener.closed)) {
return false;
}
var calloutForm = eval($form_name);
if (!calloutForm) {
return false;
}
var calloutElementFound = false;
for ( i = 0 ; i < calloutForm.elements.length ; i++ ) {
if(1==2){} //#... long comment explaining why removed for this post ...
);
while(my ($key, $value) = each(%hashMetaList)) {
#...
chomp $value;
if($value ge " "){
if($key =~ m/Level of Education/){
#...
}
else {
#the rest of the while loop, which tests every other value on the DCT
#escape any chars that may cause problems or do not get otherwise written correctly (',\)
$value =~ s/\\/\\\\/g;
$value =~ s/'/\\'/g;
#Newline chars were breaking the javascript
$value =~ s|\n|\\n|g;
print " else if(calloutForm.elements.name == \"$num" . "_" . "$key\"){\n";
print " calloutForm.elements.value = \'$value\';\n";
print " calloutElementFound = true;\n";
print " }\n\n";
}
}
}
print qq(
}
//#...
//#this will return true when done successfully.
return true;
}
//#...
);
return;
}Second, on both the attempts to copy metadata within IE and Firefox, in the cgi-callout window just prior to submitting the copied metadata to the Set Metadata form - the results of View Source look identical - e.g., the copyright symbol and odd characters and all, look fine. (hopefully the greatly reduced JPG is still readable)

Basically, it's when the data gets copied from the cgi-callout form to the Set Metadata form that the text appears to get munged (the lower-left image in the JPG is IE, the lower-right image is doing the entire thing in Firefox and having done it in Firefox, viewing it with IE.

So again - this seems to come down to some aspect of the javascript code used to transfer the data from one form in one window to another form in the other window - and I guess that it succeptible to differences in javascript processing by the browser - yes?

If anyone has specific pointers for code fixes, I'd be happy to look at them. I'll take a look at the reference you gave me Dwayne, ... a quick check using Firefox and IE from my PC shows the folllowing:

I paste "El Niño is not ©" into the text field on that page and hit the test button.

On IE - I see:New encoding: El%20Ni%C5%84o%20is%20not%20%C2%A9
escape(): El%20ni%u0144o%20is%20not%20%A9On Firefox - I see:New encoding: El%20Ni%C5%84o%20is%20not%20%C2%A9
escape(): El%20Ni%u0144o%20is%20not%20%A9In other words - I see the exact same thing - so I'm not sure I understand what this means with regard to my problem...

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Dwayne

In other words - I see the exact same thing - so I'm not sure I understand what this means with regard to my problem...

Given the code that you've shared, I would agree that the reference I cited is completely irrelevant. However, I don't know that I necessarily argee with this statement of yours:

So again - this seems to come down to some aspect of the javascript code used to transfer the data from one form in one window to another form in the other window

That is one possible explaination. And, for your sake, I frankly hope that it's wrong, because I don't know that there's much you can do about it. But there is another possible explanation. The JavaScript parser may be reading your HTML page in some funky encoding. So even though the constand looks the same in both IE and FireFox, what gets placed in memory may differ.

I would try this: create a text box in your callout, and have the JavaScript send the data, not from the constant that you've calculated, but from the contents of that text box. Then you can paste in some of the "troublesome" strings, and see what happens. If it still shows the problem, then I suspect that you're right, and the issue is with the transfer. But if that comes up correctly, then we may be back to the way IE is handling JavaScript constants.

To be clear, you'd change this line:

                print "             calloutForm.elements[ i ].value = \'$value\';\n";[/bue]

to something like this: (assuming your text field was named "testing")

                print "             calloutForm.elements[ i ].value = document.forms[0].testing.value;\n";

(I think that's the right syntax, anyway)

--

Current project: TS 5.5.2/6.1 W2K

Adam Stoller

Maybe I'm being dense here - but I'm not sure I'm following the logic here.

The line of code:

   print "     calloutForm.elements.value = \'$value\';\n"

is executed correctly without regard to browser, as can be verified by viewing the source of the CGI window prior to having the 'submit' operation go through and execute the command.

However ... I made the change as you suggested, and if I paste the same text in that text box - it does seem to get copied correctly ... so now I'm extremely confused.

What does this tell me? How does this indicate I should change my code?

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Dwayne

You are correct, in that the line of code:

    print "     calloutForm.elements

is being executed correctly by the CGI. But that's not the problem. The problem is that the resulting line of code:

     calloutForm.elements

isn't being executed properly by the browser.

JavaScript is just like Perl in the sense that it takes the source code (the above line) and converts it into machine instructions. The behavior that you've reported indicates that the problem is in the compile phase, where the browser reads the "source code." It appears to me to be a bug in IE, where it isn't properly parsing the source code as UTF-8. When you view the generated HTML, you're viewing the source code sent to the JavaScript interpreter. You have no way of seeing how it was parsed internally.

By using a form value instead of a constant value, you're bypassing the "compile" stage, and using an external data source. And it works.

So I think you're back to the encoding suggestion I made back a few posts ago. Perhaps you can have your CGI spit out, not the actual value, but the result of converting each of the characters in $value into their hex equivalent. Then have a JavaScript function in the generated HTML page "decode" those hex values back into the appropriate binary values before sending them on to the metadata capture screen. Since all of the characters in the hex character set are "normal" ASCII, you won't have the compile-time problem. And you've already shown that, once the data is "in the system." it cam be copied OK.

Does that make any sense?

--

Current project: TS 5.5.2/6.1 W2K

Adam Stoller

I was following you up until about here:
Perhaps you can have your CGI spit out, not the actual value, but the result of converting each of the characters in $value into their hex equivalent. Then have a JavaScript function in the generated HTML page "decode" those hex values back into the appropriate binary values before sending them on to the metadata capture screen. Since all of the characters in the hex character set are "normal" ASCII, you won't have the compile-time problem. And you've already shown that, once the data is "in the system." it cam be copied OK.

How am I going to take the external data (the data retrieved from iwextattr) and turn it into form variables without running into the exact same problem? (I already tried doing this and it didn't change the behavior).

If I encode / translate the EA values into using entities - then those entities will show up in the Set Metadata form - which we don't want - we want to see the WYSIWYG characters in that form.

Maybe I'm just tired - or maybe I just have a mental block with programming languages that begin with the letter 'J' [:-)] - but I'm just not understanding how I can retrofit this code to get around this semi-obscure defect of IE.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Dwayne

How am I going to take the external data (the data retrieved from iwextattr) and turn it into form variables without running into the exact same problem? (I already tried doing this and it didn't change the behavior).

You misunderstood. I didn't mean to create form variables. That was only to test my theory. I was thinking more of something like this:

In your CGI (in perl)

<pre># Assuming that $value contains the value you want to send to the metadata capture screen:
my $encoded = unpack "H*", $value; # Convert the input string to the hex equivalient.
# That is, "1234" becomes "31323334";

Now in the generated JavaScript section, where you currently use something like

    print "      someVarialble.forms[0].elements[ i].value = '$value';"\n;

use instead something like

    print "      someVarialble.forms[0].elements[ i].value = ParseValue('$encoded');"\n;

and then also include the definition of a JavaScript function which will convert that hex string back into the original data.

You of course can use whatever encoding you want. I just chose hex because it seemed easiest.

--

Current project: TS 5.5.2/6.1 W2K

Dwayne

OK, I did a little more digging, and I think something like this might be the easiest. Rather than doing the hex encoding that I was describing, try replacing this line:

    print "      someVarialble.forms[0].elements

with something like this:

(there's probably a use of map that would simplify that, but I didn't feel like working that hard ;-) )

$value would be the unescaped metadata value

This will end up, in the generated page, with code that looks something like this (assuming the metadata to be set was "123")

    someVariable.forms[0].elements

Thus, no "special" characters appear in the HTML, but the actual binary data gets stored in the appropriate variable.

--

Current project: TS 5.5.2/6.1 W2K

Adam Stoller

For those playing along from home ... here's my assessment of the situation (much appreciation to Dwayne for providing assistance):

1. IE's javascript / form processing code doesn't handle UTF-8 / multi-byte characters very well (or at least not as well as Mozilla / Firefox does)
2. Perl 5.005_03 (which ships with TS5, which we are using) also does not support UTF-8 / multi-byte characters very well (neither the 'utf8' nor the 'bytes' pragmas are supported)

My guess is that we might be able to "fix" this in TS6 with the newer version of Perl, but that doesn't help us in the here and now, so I don't think there's anything I can really do here - but I remain open to any ideas anyone wishes to suggest.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Dwayne

Take a look at Unicode:tring, which ships as a part of the Perl package in TS 5.x. It includes the method unpack which converts a Unicode string into a sequence of integers (which is what I was attempting to do with my split logic).

I suspect that will get you working before moving to 6.x

--

Current project: TS 5.5.2/6.1 W2K

Edited by Dwayne on 08/16/04 06:52 AM (server time).

Adam Stoller

Fantastic!

I searched on CPAN for "utf8" - didn't think to search for "unicode" - this appears to have resolved the problem:use Unicode: Smiley Frustrated

tring;
#... get the value from the hash of values extracted via iwextattr - store in $value
my $u = Unicode: Smiley Frustrated

tring->new($value);
my @chars = $u->unpcak;
my $v = join(",", @chars);
print " calloutForm.elements.value = String.fromCharCode($v);\n";NOTE: Unicode: Smiley Frustrated

tring does not come with TS6 / Perl 5.8.2 - so we'll either have download and install it ourselves, or convert the code to use some other technique when we get there - but this does seem to resolve the problem for the here and now.

Many thanks Dwayne.

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

Dwayne

Glad I could be of service.

BTW - once you move to Perl 5.8, I suspect that the split ... logic I used before should work, since the strings will be treated as UTF-8 already, and regexes are multibyte aware.

--

Current project: TS 5.5.2/6.1 W2K