Understanding wordcount.xsl

Mr. Fu · October 10th, 2007, 05:23 AM

Hi there,

I'm new to the forum, so please be gentle...

Maybe someone could help me understanding the wordcount.xsl example given in the XSLT 2.0 Programmers Reference. I'm not sure where to post it or nor, but since it is a short stylesheet, here we go:

<?xml version="1.0" encoding="iso-8859-1"?>
<xsl:stylesheet
   version="2.0"
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
<xsl:template match="/">
  <wordcount>
    <xsl:for-each-group group-by="." select="
          for $w in tokenize(string(.), '\W+') return lower-case($w)">
      <xsl:sort select="count(current-group())" order="descending"/>
      <word word="{current-grouping-key()}" frequency="{count(current-group())}"/>
    </xsl:for-each-group>
  </wordcount>
</xsl:template>
</xsl:stylesheet>

I tested the stylesheet with a hamlet.xml encoded by Jon Bosak (just google for it, I will definitely not post this one here...) on a Linux machine (Kubuntu 7.04, Java 1.6.0, Saxon-B 8.9.0.4J) and it worked perfect.

But then I realized I shouldn't. 'string(.)' returns the concatenation of all descending text nodes, in this case the whole text of the input document, ignoring the markup.
So something like

<SPEECH>
<SPEAKER>HAMLET</SPEAKER>
<LINE>Not so, my lord; I am too much i' the sun.</LINE>
</SPEECH>

would become

HAMLETNot so, my lord; I am too much i' the sun.

tokenizing (and lower-casing) this line would result in a 'hamletnot', amongst others, and not a 'hamlet' as the out put given by 'java -jar saxon8.jar hamlet.xml wordcount.xsl' would suggest. Saxon counts 475 occurrences of 'hamlet', which is perfectly correct, but theoretically it should only report 107, along with the occurrence with some esoteric words like 'hamletnot', 'hamletthy' and so on. Am I missing something?

In my case, using this stylesheet with the eXist open source XML database, I resolved the matter using a temporary tree in which the markup was removed in a more hamlet-friendly way, but I was still wondering about Saxon's behaviour in this case. Can anyone point me in the right direction?

Thanks,
Sven

samjudson · October 10th, 2007, 05:45 AM

The above xml snippet contains more text nodes than you are giving it credit for.
I've hightlighted them below by putting each text node in brackets.

Code:

<SPEECH>(
 )<SPEAKER>(HAMLET)</SPEAKER>(
 )<LINE>(Not so, my lord; I am too much i' the sun.)</LINE>(
)</SPEECH>

/- Sam Judson : Wrox Technical Editor -/

samjudson · October 10th, 2007, 05:56 AM

(It should be noted that some XSLT 2.0 processors strip out whitespace before it gets to the XSLT processor, e.g. Altova, so the example above does work exactly as you suspected).

/- Sam Judson : Wrox Technical Editor -/

Mr. Fu · October 10th, 2007, 07:48 AM

I understand (at last)... thank you for your quick help!

mhkay · October 10th, 2007, 08:05 AM

This is a good catch. The code as written is dependent on whitespace text nodes being preserved on their way in to the XSLT processor, and (especially as the example appears so early in the book) this should at least have been made clear. In an ideal world it would be possible to write XSLT code knowing that the whitespace will be there (unless you use xsl:strip-space to remove it). However, the world is not ideal, and neither you as the stylesheet author nor W3C as the author of the XSLT specification can legislate what happens to the document on its journey from a file on disk to a tree in memory before the XSLT processor gets to see it. Some XML parsers, notably those used by Microsoft and Altova, remove the whitespace by default, and in some environments (notably Altova) you don't even get any option to stop this happening. With Saxon there are command line options to request stripping of the whitespace, or conditional stripping based on the content model in any DTD or schema.

So, in the ideal situation where whitespace is preserved the code will work correctly, but in some environments it won't. To work correctly in all environments, the simplest fix is to replace the use of string(.) by normalize-space(string-join(descendant::text(), ' ')), which adds spaces between any two text nodes. Unfortunately this is a fairly expensive operation.

You've left me in a bit of a quandary about what to do with this example when I revise the chapter. The above is far too complicated to introduce at the time I present the example, but perhaps I can fix it by noting the problem briefly and coming back to it in Chapter 3 where whitespace issues are analyzed in sordid detail.

Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference