nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

WFT: SetVariable UTF-8 encoding issues in externaltask perl code

Frederik

TS 6.5 sp2, Solaris 9

I'm having issues with the encoding of Workflow Variables.

Suppose a simple workflow: externaltask -> submittask -> End

The submit comment is input on the Instantiator CGI.

[HTML]
owner="__TAG__('iw_user');"
savecomments="t">

[/HTML]

If the user inputs fancy UTF-8 characters (e.g. éàŁšŀňëï½€), and the external task doesn't touch the Submit Comment, all is fine. I can see the comment in the submit log with all the fancy characters.

But I want to slightly modify the Submit Comment in the external task:

my ($success, $submittask) = $job->GetTaskByName("Submit");
if ($success) {
  my $submit_cmt = $submittask->GetVariable("submit_cmt");
  if ($submittask->SetVariable("submit_cmt", "$submit_cmt (external task has run)") ) {
    $logger->warn("failed: $submittask->SetVariable");
  }
}

In the ContentCenter's submit log, I can see the modification of my comment, but the fancy UTF-8 characters are lost, replaced by squares.

Has anyone managed to create or edit UTF-8 strings on the WF Variables?

many thanks

Find more posts tagged with

Comments

ISCBorisB

You may want to try something like the following:
use Encode;
...
$submittask->SetVariable("submit_cmt", "$submit_cmt" . Encode::encode_utf8('(external task has run)'))
Or perhaps
$submittask->SetVariable("submit_cmt", Encode::encode_utf8("$submit_cmt (external task has run)"))

Frederik

I've tried Encode::encode_utf8 like your example, but it doesn't make any difference.

to complete my first post. It's TS 6.5 sp2 patch 1767, the server_locale in iw.cfg is set to US ASCII.

ISCBorisB

I've tried Encode::encode_utf8 like your example, but it doesn't make any difference.

to complete my first post. It's TS 6.5 sp2

I've replicated your issue, exactly as described Smiley Sad

For what it's worth, my little debugging shows that the result of GetVariable (or GetVariables) is already mangled.
Eg, we have a problem right here: my $submit_cmt = $submittask->GetVariable("submit_cmt"); even before
SetVariable. Please let me know if you would be able to go further, I'll do the same

Frederik

For what it's worth, my little debugging shows that the result of GetVariable (or GetVariables) is already mangled.

Not so here.
In my trials, it seems that the GetVariable *is* successful in keeping good encoding.

As test, I'm printing it out to an XML - that way I make sure my not-so-smart editor knows with what encoding to interpret the text file. That XML looks good with all the exotic UTF-8 intact.

    my ($success, $submittask) = $job->GetTaskByName("Submit");
    if ($success) {
      my $submit_cmt = $submittask->GetVariable("submit_cmt");
      if (! defined $submit_cmt) { $logger->warn("failed: $submittask->GetVariable"); }

      use IO::File;
      my $fh = new IO::File "> /tmp/wfvars.xml";
      if (defined $fh) {
        binmode( $fh, ':utf8' );
        print $fh "\n";
        print $fh "$submit_cmt\n";
        $fh->close;
      }
    }

ISCBorisB

Not so here...

You are correct, my mistake. GetVariable is fine - just tested my results more carefully

Frederik

I just realised my code to illustrate the GetVariable good encoding above had some XML tags hidden due to Forum's crappy renderer. Corrected below.

Anyway, I'm sending this issue to Support.

    my ($success, $submittask) = $job->GetTaskByName("Submit");
    if ($success) {
      my $submit_cmt = $submittask->GetVariable("submit_cmt");
      if (! defined $submit_cmt) { $logger->warn("failed: $submittask->GetVariable"); }

      use IO::File;
      my $fh = new IO::File "> /tmp/wfvars.xml";
      if (defined $fh) {
        binmode( $fh, ':utf8' );
        print $fh "<?xml version='1.0' encoding='UTF-8'?>\n";
        print $fh "<submit_cmt>$submit_cmt</submit_cmt>\n";
        $fh->close;
      }
    }

Adam Stoller

A variant of the code posted - not necessarily "better", just a different take on the same code:

my ($success, $submittask) = $job->GetTaskByName("Submit");
if ($success) {
    my $submit_cmt = $submittask->GetVariable("submit_cmt") || q();
    if ($submit_cmt eq q()) {
# $logger is [presumably] a Log::Log4perl object handle
        $logger->warn("failed: $submittask->GetVariable");
    }

if (!open(OUT, '>:utf8', "/tmp/wfvars.xml")){
        $logger->warn(" /tmp/wfvars.xml ($!)");
    }
    else {
        binmode(OUT); [color=orange]# or perhaps still binmode(OUT, ':utf8') ?[/color]
        print OUT "<?xml version='1.0' encoding='UTF-8'?>\n";
        print OUT "<submit_cmt>$submit_cmt</submit_cmt>\n";
close(OUT);
    }
}

Frederik

A variant of the code posted - not necessarily "better", just a different take on the same code:

thanks... I only code in Perl about 10-20 days a year, so every time I pick it up, I need a quick refresher and I usually end up copy pasting bits and pieces from what seems to be the best samples around - so I appreciate the pointers.

In this case though, that code served it's purpose :-) to illustrate the problem with SetVariable.

Frederik

I think I cracked it :-)

The problem seems to be due to the

-a

flag used on the CLT iwtaskvariable (idem for iwjobvariable).

TeamSite::WFtask->SetVariable

basically only calls

TeamSite::WFutils::call_clt("iwtaskvariable", @args)

, which basically does:

system( "$iwhome/bin/$clt", @args );

(in case of UNIX, or the equivalent for Windows) but only after shifting a [COLOR=Teal]-a

flag on the

@args

[/COLOR] !!

Now, this

-a

flag is undocumented in TS 6.5 documentation (ts.650a.clt.unix.pdf, ts.650a.admin.unix.pdf). Smiley Sad

Some comments in the perl code iw-home/iw-perl/vendor/lib/TeamSite/WF*.pm suggests that the

-a

flag causes the CLT to transform its arguments to the encoding set by iw.cfg server_locale (in our case that's the default server_locale=English_UnitedStates.US-ASCII@Binary; ).

I suppose that the transformation fails on UTF-8 exotic chars, and therefore the CLT transforms every character that's unavailable in the target encoding to some exception character.

I tested this

-a

flag via command line (wfvars-utf8.txt is a text file containing some UTF-8 text (UTF test éàŁšŀňë/ï[½/€ test), because I can't type UTF-8 special chars directly in my shell)

$ iwtaskvariable -t mytaskkey "`cat /tmp/wfvars-utf8.txt`" 28323
-> OK, verified with ccpro

$ iwtaskvariable -g mytaskkey 28323
UTF test Ã©Ã ÅÅ¡ÅÅÃ«/Ã¯[Â½/â¬ test
-> OK, the shell is ISO-8859-1 and can't display UTF-8 natively

$ iwtaskvariable -a -g mytaskkey 28323
UTF test Ã©Ã ÅÅ¡ÅÅÃ«/Ã¯[Â½/â¬ test
-> OK, the shell is ISO-8859-1 and can't display UTF-8 natively

$ iwtaskvariable -t mytaskkey "`iwtaskvariable -a -g mytaskkey 28323`" 28323
-> OK, verified with ccpro

$ iwtaskvariable -a -t mytaskkey "`iwtaskvariable -a -g mytaskkey 28323`" 28323
-> NOK! verified with ccpro -> UTF-8 mangled.

$ iwtaskvariable -g mytaskkey 28323
UTF test ??????????????/??[??/??? test

Bottom line:
getting a variable (

iwtaskvariable -g var id

) seems not to care about the added

-a

, but on setting a variable, the

-a

flag ruins the UTF-8 encoding (depending maybe on your server_locale).

I'm not sure if the

-a

flag in WFutils.pm is a remnant from pre-6.1 era (6.1 introduced newer Perl 5.8.2 with all new utf8 string handling). Or if it still has some good use... But I would think it's fairly safe to modify WFutils to avoid

-a

on the iwtaskvariable CLT, if not on all of them.

All the above also applies to iwjobvariable CLT.

EDIT:
PS: I forgot to mention how much I love open-source content management systems ;-) I wouldn't have found this without access to the perl module source.

mdang

I verified that we do have a bug in the code path of
TeamSite::WFtask->SetVariable ===> TeamSite::WFutils->call_clt

I will file a bug and will fix the Perl module in future patch/release. However, the iwtaskvariable CLT works fine with utf8 values. So to work around the issue in the current releases, you can call iwtaskvariable directly as the following example code:

my $iwhome = TeamSite::Config::iwgethome();
my $taskId = $your_WFTask_object->GetId();

system("$iwhome/bin/iwtaskvariable", "-t", $your_variable_name, $your_utf8_content_string, $taskId);