nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Cleaning-up DCRs using HTML Tidy or similar??

System

I need to find a way to use HTMLTidy or any similar tool to clean-up DCR content of invalid tags, the problem is that formatted DCR content, from the visual formatter fields, are stored as entity references and not tags themselves, <bold> are stored within the DCR as <bold> and the clean-up tool does not recognize it as a tags and cannot perform the cleanup.

Has anyone in the field done anything similar that worked? If I have to open every DCR and copy the content from VisualFormatter and paste into another tool for conversion and then paste it back it would take me forever.

Any help or ideas are greatly appreciated.

Cheerio,

A

Find more posts tagged with

Comments

Adam Stoller

Do you have to do it in the DCR - or could you do it in the PT - where you can get a handle on the formatted content?
Alternatively - you could potentially write a PT that processes a DCR and then writes out a modified version of the DCR (taking care to re-entitize content within value tags)

--fish
Senior Consultant, Quotient Inc.
http://www.quotient-inc.com

chuckles

The VF content should already be cleaned. You can even configure for different validations like XHTML. The character encodings ensure that the HTML content is stored as plain CDATA in the DCR. When you access that in your PT using <iw_value> the characters are converted back for you. If you are accessing the DCR through some custom script, it should easy enough to convert if yourself using some regexps. The only characters that get escaped are '<', '>', and '&' I think.

Migrateduser

The problem here is that we are talking about legacy content which was not collected utilizing the clean-up feature of the VF editor.

Migrateduser

Hey Fish, Thanks for the post, but they do not want to perform the clean-up on the resulting HTML, I suggested that one. They want to do it at the source and since we are talking about a publishing company, they have a lot of legacy content.

shanon

albert,

you can use TeamSite::CGI_lite::escape_html_data and unescape_html_data functions to swap out the entities...

use the TeamSite: Smiley Very Happy

CRnode module to parse the dcr and get the XML, then the above function to correct the entities, then clean it in perl ( i think there's a tidy of some sort in cpan ), then escape it and put it back...

probably the easiest way is to use File::Find and build a file of files and then feed them into a foreach to fix them...

here's the perldoc for File::Find:

http://search.cpan.org/~nwclark/perl-5.8.5/lib/File/Find.pm

good luck.

Shanon Levenherz
Mercer eBusiness Team
shanon.levenherz@mercer.com
http://www.mercerhr.com

chuckles

I think that using un/escape_html_data across the board would be overkill and possibly not give you your expected results. Just escape/unescape the characters that affect XML ('<', '>', '&'). For the actually tidying, there was a thread on this subject in the Perl forum recently that might help:
http://devnet.interwoven.com/forums/cgi-bin/showflat.pl?Cat=&Board=perl&Number=50979&page=0&view=collapsed&sb=5&o=7&part=

shanon

um, what do you think escape/unescape html data does?

it escapes >, <, & and the other two characters that you missed, ' and ".

this is exactly what you're looking for regarding the entities...

here's the code from CGI_lite:


sub escape_html_data 
{
    make_static(\@_);
    my($line) = shift;

    if (!defined $line) {return undef;}

    # #39 = Apostrophe. &apos; does not work in some versions of IE.
    my %html_escape_sequences = ('&' => '&amp;',
                                '>'  => '&gt;',
                                '<'  => '&lt;',
                                '"'  => '&quot;',
                                '\''  => '&#39;',
                                );

    $line =~ s/([&><"'])/$html_escape_sequences{$1}/g;
    return $line;
}

Shanon Levenherz
Mercer eBusiness Team
shanon.levenherz@mercer.com
http://www.mercerhr.com

tonyarnold

Here's how I did it:

Inside my presentation template

Keep in mind, I'm parsing my DCR using perl rather than using the templating tags (I find it easier for the way I code, and the output is much cleaner than if you use tags), hence the xml node references.

use xMarkdown;
use University::Tidy;
my $output_text;
my $field_type = $field->type();

if ( $field_type eq 'markdown' )
{
   $output_text = xMarkdown::markdown( $field->value() );
   $output_text = University::Tidy::tidyString( $output_text );
} 
elsif ( $field_type eq 'visualformat' ) 
{
   $output_text = $field->value();
   $output_text = University::Tidy::tidyString( $output_text );
}

The "University::Tidy" perl module (in $IW_HOME/iw-perl/site/lib/University/Tidy.pm)

package University::Tidy;
use Cwd;
#use strict;
# Yeah, I know, I know...

my $g_tidyPath = "/usr/local/bin";
my $g_tidyOptions  = " -quiet";
 $g_tidyOptions .= " --wrap 0";
 $g_tidyOptions .= " --output-xhtml yes";
 $g_tidyOptions .= " --drop-font-tags yes";
 $g_tidyOptions .= " --drop-proprietary-attributes yes";
 $g_tidyOptions .= " --word-2000 yes";
 $g_tidyOptions .= " --bare yes";
# $g_tidyOptions .= " --clean yes";
 $g_tidyOptions .= " --indent auto";
 $g_tidyOptions .= " --indent-attributes no";
 $g_tidyOptions .= " --vertical-space no";
 $g_tidyOptions .= " --drop-empty-paras yes";
 $g_tidyOptions .= " --logical-emphasis yes";
 $g_tidyOptions .= " --wrap-sections no";
 $g_tidyOptions .= " -f /dev/null";

sub tidyString {
 #my ( $parent, $myEmptyAttrib, $myEmptyAttribStr) = @_;
 # usage: tidyString(string STRING[, int NODOCTAGS])
 # result: STRING with tidy-parsed HTML
 die("usage: tidyString(string STRING[, int NODOCTAGS])") unless @_;
 return ($_[0]) if ($_[0] !~ /<[^>]*>/); # return STRING if it has no HTML tags, nothing to parse
 if (!defined $_[1]) { $_[1] = 1; }  # NODOCTAGS shall default to 1, if omitted
           # (set it to 0 if you do want to receive 
           # a complete HTML document; to 2 if SCRIPT and STYLE
           # tags in the header shall be returned with the body)

 pipe (READ_FROM_PARENT, WRITE_TO_CHILD) or die "can't pipe(1): $!";
 pipe (READ_FROM_CHILD, WRITE_TO_PARENT) or die "can't pipe(2): $!";
 $parent = fork();
 die ("fork failed: $!") unless defined $parent;

 unless ($parent) {
  # we are the child

  my $tidy = $g_tidyPath . "/tidy";

  # close obsolete file handles
  close(READ_FROM_CHILD);
  close(WRITE_TO_CHILD);

  # redirect STDIN
  close(STDIN);
  open(STDIN, "<&READ_FROM_PARENT");

  # redirect STDOUT
  close(STDOUT);
  open(STDOUT, ">&WRITE_TO_PARENT");

  select(STDIN);
  $| = 1;

  # switch process to tidy
  exec("$tidy$g_tidyOptions") or die("exec of \'$tidy\' failed: $!");

  # END of child
 } else {
  # we are the parent

  my $tidied_data = "";

  # close obsolete file handles
  close(READ_FROM_PARENT);
  close(WRITE_TO_PARENT);

  # print the to-be-tidied string to the child's STDIN
  print WRITE_TO_CHILD $_[0];
  # close the file handle
  close(WRITE_TO_CHILD);

  # read everything from the child's STDOUT into $tidied_data
  while () {
   $tidied_data .= $_;
  }

  # close the file handle
  close(READ_FROM_CHILD);

  # reap the child
  waitpid($parent, 0);

  if ($_[1] > 0) {
   # remove unwanted document tags
   # (we want only what is between  and )

   if ($tidied_data !~ /]*>/i) {
    # cancel, if no body
    return ("[could not find BODY in message]");
   }

   if ($_[1] > 1) {
    # we'd like to have JavaScript and CSS codes returned
    my $JavaScript = "";
    my $CSS = "";

    $tidied_data =~ s\(<SCRIPT[^>]*>.*?</SCRIPT[^>]*>)\$1\si;
    $JavaScript = $1 . "\n" unless ($1 eq "");

    $tidied_data =~ s\(]*&amp;amp;amp;gt;.*?&amp;amp;amp;lt;/STYLE[^&amp;amp;amp;gt;]*&amp;amp;amp;gt\$1\si;
    $CSS = $1 . "\n" unless ($1 eq "");

    # add the JavaScript and StyleSheet tags to the output
    # at the beginning of BODY
    $tidied_data = $JavaScript . $CSS . $tidied_data;
    $tidied_data =~ s/(&amp;amp;amp;lt;BODY[^&amp;amp;amp;gt;]*&amp;amp;amp;gt;\n?)/$JavaScript$CSS$1/si;

   }

   # strip everything before &amp;amp;amp;lt;BODY&amp;amp;amp;gt; and after &amp;amp;amp;lt;/BODY&amp;amp;amp;gt;
   $tidied_data =~ s/^.*&amp;amp;amp;lt;BODY[^&amp;amp;amp;gt;]*&amp;amp;amp;gt;\n?//si;
   $tidied_data =~ s\&amp;amp;amp;lt;/BODY[^&amp;amp;amp;gt;]*&amp;amp;amp;gt;.*&amp;amp;amp;lt;/HTML[^&amp;amp;amp;gt;]*&amp;amp;amp;gt;.*$\\si;
  }

  $tidied_data =~ s/&amp;amp;amp;lt;FONT[^&amp;amp;amp;gt;]*&amp;amp;amp;gt;&amp;amp;amp;lt;\/FONT[^&amp;amp;amp;gt;]*&amp;amp;amp;gt;\n?//gsi;
  while ($tidied_data =~ /\s*(BORDERCOLOR([a-z]*))\s*=\s*".*?"/i) {
   $tidied_data =~ s/\s*$1\s*=\s*".*?"//gsi;
  }

  # drop unknown empty attributes, ALT shall be allowed to be empty
  while ($tidied_data =~ /\s*([a-z]+)\s*=\s*""/i) {
   $myEmptyAttrib = $1;
   if ($myEmptyAttrib =~ /ALT/i) {
    # replace the recognized empty attribute with a placeholder
    $myEmptyAttribStr = "myEmpty" . $myEmptyAttrib . "Attrib";
    $tidied_data =~ s/(&amp;amp;amp;lt;[^&amp;amp;amp;gt;]*?)\s(?:[a-z]+)\s*=\s*""([^&amp;amp;amp;gt;]*&amp;amp;amp;gt/$1 $myEmptyAttribStr$2/gsi;
   } else {
    # drop the unrecognized empty attribute
    $tidied_data =~ s/(&amp;amp;amp;lt;[^&amp;amp;amp;gt;]*?)\s(?:[a-z]+)\s*=\s*""([^&amp;amp;amp;gt;]*&amp;amp;amp;gt/$1$2/gsi;
   }
  }
  # replace the placeholder with the appropriate empty attribute
  $tidied_data =~ s/myEmpty([a-z]+)Attrib/$1=""/gsi;

  return $tidied_data;

  # END of parent
 }
}&amp;amp;amp;lt;/pre&amp;amp;amp;gt;



I also compiled the latest version of libtidy/tidy from sourceforge &amp;amp;amp;lt;A HREF="http://tidy.sf.net/" TARGET='_blank'&amp;amp;amp;gt;tidy.sf.net&amp;amp;amp;lt;/A&amp;amp;amp;gt; and installed the binary in /usr/local (although you could put it anywhere). Obviously, this doesn't solve the issue of applying this to existing DCRs, but I figure this is useful code (and the config options I've listed in my perl module would help you in tidying old code anyway). Let me know if you want more information &amp;amp;amp;lt;/p&amp;amp;amp;gt;&amp;amp;amp;lt;/p&amp;amp;amp;gt;

gzevin

Good stuff.

just to add - if one's comfortable with PT methods, they could use TeamSite: Smiley Very Happy

CRnode if the processing is done outside of a TPL.

there could be other methos to tidy as well - regexes, HTML: Smiley Tongue

arser (like in John Cuiulli's s article), etc

Greg Zevin, Ph.D. Comp. Sc.
Independent Interwoven Consultant/Architect
Sydney, AU

samstephens

So you're wanting to clean legacy DCRs right. So you can just have a stand alone perl script that runs through all your DCRs, read in the content, fix it, and write it back out again.

Wouldn't the easiest way be to read each DCR using the TeamSite::XMLnode module, take each node that is a VisualFormat field, and run it through HTMLTidy. That way you can completely ignore encoding issues, XMLNode will just give you the unencoded text.

I've used TeamSite::XMLnode to handle a conversion from one DCR style to another, and found it very convenient for reading and writing DCRs.

Psuedocode something like:

  open SOURCEDCR, '<:utf8', "$dcrfilename" or die ("No such file");
  read( SOURCEDCR, $inxml, -s SOURCEDCR);
  close SOURCEDCR;
  my $rootnode = TeamSite::XMLnode->new($inxml);
  foreach my $node ($rootnode->get_node_list()) {
    if ($node->type() is a VisualFormat node) {
      $dirtyhtml = $node->value();
      # tidy_html will call HTMLTidy to clean the code
      $cleanhtml = tidy_html($dirtyhtml);
      $node->get_node('CDATA')->set_inner_xml($cleanhtml);
    }
  }
  $outxml = $rootnode->get_xml();
  my $header = '<?xml version="1.0" encoding="UTF-8"?>'."\n";
  $header = $header.'<!DOCTYPE your doctype SYSTEM "your DTD">'."\n";
  $outxml = $header.$outxml;
  open DESTDCR, '>:utf8', "$dcrfilename" or die ("Can't open output file $dcrfilename");
  print DESTDCR $outxml;
  close DESTDCR;

The example I've given is very simple (and possibly buggy as I haven't tested it). It only handles a "flat" DCR structure, with no nesting. But I think it's a good framework to build what you need on top of.

I had a wierd bug with how XMLNode handled the old style of DCR, describe by dcr4.5.dtd (before proper XML DCRs). I needed the line

$inxml =~ s!<item name="([^"]*)"/>!<item name="\1"><value></value></item>!g;

Before the call to TeamSite::XMLnode->new, in order for empty XML nodes to be picked up correctly.

Just noticed I'm posting to a dead thread... oops. Oh well, if someone else has issues like this, I think XMLnode is probably the easiest way to handle it.