How to do Very Large String Concat?

EcceBozo · July 28th, 2008, 02:39 PM

I'm using XSLT to process datasets converted to XML, and hit upon an issue trying to concat a bunch of strings together. When the number of records and/or subsets gets very large (say over 50 subsets in a group of 100 records), the parser chokes (or at least I've never let it go beyond a few minutes, as that's unacceptable). Note that I'm stuck in 1.0 world with the .NET parser--the transform seems to work just fine in Altova XMLSpy. Any help is appreciated.

XML SAMPLE:
<?xml version="1.0" encoding="UTF-8"?>
<NewDataSet>
    <Table>
        <DataID>100</DataID>
        <SubsetID>201467</SubsetID>
        <SubsetSeq>1</SubsetSeq>
        <SubsetDate>1999-03-04T14:14:09.0000000-05:00</SubsetDate>
        <SubsetText>This is a Test.</SubsetText>
    </Table>
    <Table>
        <DataID>100</DataID>
        <SubsetID>201468</SubsetID>
        <SubsetSeq>1</SubsetSeq>
        <SubsetDate>1999-03-04T14:03:57.0000000-05:00</SubsetDate>
        <SubsetText>Also a test.</SubsetText>
    </Table>
    <Table>
        <DataID>100</DataID>
        <SubsetID>201469</SubsetID>
        <SubsetSeq>1</SubsetSeq>
        <SubsetDate>1999-03-08T20:54:47.0000000-05:00</SubsetDate>
        <SubsetText>Another test.</SubsetText>
    </Table>
    <Table>
        <DataID>100</DataID>
        <SubsetID>201470</SubsetID>
        <SubsetSeq>1</SubsetSeq>
        <SubsetDate>1999-03-08T20:55:23.0000000-05:00</SubsetDate>
        <SubsetText>Final test.</SubsetText>
    </Table>
    <Table>
        <DataID>99</DataID>
        <SubsetID>201460</SubsetID>
        <SubsetSeq>1</SubsetSeq>
        <SubsetDate>1999-03-04T13:32:28.0000000-05:00</SubsetDate>
        <SubsetText>Second dataset test 1.</SubsetText>
    </Table>
    <Table>
        <DataID>99</DataID>
        <SubsetID>201461</SubsetID>
        <SubsetSeq>1</SubsetSeq>
        <SubsetDate>1999-03-04T13:22:44.0000000-05:00</SubsetDate>
        <SubsetText>Second dataset test 2.</SubsetText>
    </Table>
</NewDataSet>

XSLT:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>

    <xsl:variable name="NewLine" select="'#x0d;#x0a;'"/>

    <xsl:template match="/">
        <DataSets>
            <xsl:for-each select="NewDataSet/Table[not (DataID = preceding-sibling::*/DataID)]">
                <xsl:variable name="CurrentID" select="DataID"/>
                <DataSet DataID="{$CurrentID}">
                    <xsl:apply-templates mode="BuildData" select="../Table[DataID = $CurrentID]"/>
                </DataSet>
            </xsl:for-each>
        </DataSets>
    </xsl:template>

    <xsl:template mode="BuildData" match="*">
        <xsl:if test="string-length(SubsetDate) > 0">
            <xsl:value-of select="concat(SubsetDate, ': ')"/>
        </xsl:if>
        <xsl:value-of select="concat(SubsetText, $NewLine)"/>
    </xsl:template>

</xsl:stylesheet>

mhkay · July 28th, 2008, 03:00 PM

If you've got a performance problem with a particular XSLT processor and you can't change processors then you need to go to the vendor for support.

But don't forget that Saxon works on .NET too.

Michael Kay
http://www.saxonica.com/
Author, XSLT 2.0 and XPath 2.0 Programmer's Reference

EcceBozo · July 29th, 2008, 03:58 PM

I suspect I'd be happier if it was an option to switch. However ...

Finally figured it out. First, using Microsoft's XPathDocument object instead of XMLDocument made it 10 times faster, but still slower than I wanted. So second and more importantly, found that Microsoft's implementation of preceding-sibling is awful. Replaced my external select with a key/generate-id, and now it flashes through in a second. Thanks for your thoughts!