how to produce single output from multiple sources?

ilyaz · December 13th, 2010, 01:41 AM

I have a set of source XML files FILE1, ..., FILEn that I am converting using XSLT to another set of XML files. Each resulting file has the following structure:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<data>
<!-- stuff corresponding to FILEi -->
</data>

I use Saxon. What I actually need is a single output XML file like this:

Code:

<?xml version="1.0" encoding="UTF-8"?>
<data>
<!-- stuff corresponding to FILE1 -->
<!-- stuff corresponding to FILE2 -->
...
<!-- stuff corresponding to FILEn -->
</data>

How can I achieve this? Can I use Saxon to do this many-to-one conversion at once, or do I need to run some type of postprocessing concatenation of the resulting XMLs into one? If latter, what tool should I use? I need to run this from command line.

Thanks!

samjudson · December 13th, 2010, 04:43 AM

You simply need to use the document() function to load each XMl file in turn.

[code]
<data>
<xsl:for-each select="1 to 10">
<xsl:apply-templates select="document(concat('FILE',.))/data/*"/>
</xsl:for-each>
</data>
[code]

Then you'd have templates to process the contents of the file. Or you could change the apply-template to a copy-of if you simply want to copy everything.

mhkay · December 13th, 2010, 05:31 AM

Or if the files are all in a directory, you can use

Code:

<xsl:template name="main">
  <data>
     <xsl:copy-of select="collection('file:///dir/?select=FILE*.xml')"/>
  </data>
</xsl:template>

ilyaz · December 13th, 2010, 11:30 AM

Quote:

Originally Posted by mhkay

Or if the files are all in a directory, you can use

Sorry for being slow, but I need this to be spelled out for me in more detail.

-----

Now I have an input folder "c:/data/XMLin" that contains a bunch of source XML files. I also have an XSL transform that creates a set of corresponding output files that have the form

Code:

<?xml version="1.0" encoding="UTF-8"?>
<data>
<!-- stuff corresponding to FILEi -->
</data>

These files are stored in a folder "c:/data/XMLout". What I need to do is to create a single XML file in the form

Code:

<?xml version="1.0" encoding="UTF-8"?>
<data>
<!-- stuff corresponding to FILE1 -->
<!-- stuff corresponding to FILE2 -->
...
<!-- stuff corresponding to FILEn -->
</data>

----------

So do I need to run my existing transform, generate the output files as I do now and then apply what you suggest to produce the single output? Or do I do it all in one step? What would be the command line options for Saxon to do this?

Thanks much!

mhkay · December 13th, 2010, 12:17 PM

Are you currently using the command line option to take a directory as input and produce a directory as output? It's not clear to me where you are starting from.

Do you want the single output file *as well as* the existing output files, or as a replacement for them?

I'd suggest doing a single transformation that uses collection() to read all the input files, and this single transformation produces multiple output files using xsl:result-document.

ilyaz · December 13th, 2010, 12:21 PM

Quote:

Originally Posted by mhkay

Are you currently using the command line option to take a directory as input and produce a directory as output?

Yes

Quote:

Originally Posted by mhkay

Do you want the single output file *as well as* the existing output files, or as a replacement for them?

Single output is enough. If I can avoid creating the individual output files I am creating now that would be great. If I have to do it in a two-step process, that's fine too.

mhkay · December 13th, 2010, 12:33 PM

OK. I guess "stuff corresponding to X" doesn't mean "a copy of X" as I supposed, but "the result of transforming X", using the same transformation as you are currently using to generate each output file individually.

So, take your existing transformation, add

Code:

<xsl:template name="main">
  <data>
     <xsl:apply-templates select="collection('file:///c:/data/XMLin/?select=FILE*.xml')"/>
  </data>
</xsl:template>

and then invoke it from the command line with no -s option, with -o specifying the name of the single output file, and with -it:main

ilyaz · December 13th, 2010, 01:46 PM

Quote:

Originally Posted by mhkay

OK. I guess "stuff corresponding to X" doesn't mean "a copy of X" as I supposed, but "the result of transforming X", using the same transformation as you are currently using to generate each output file individually.

and then invoke it from the command line with no -s option, with -o specifying the name of the single output file, and with -it:main

Michael,

When I ran it like this:

Code:

Transform.exe -xsl:myXform.xsl -it:main -o:"C:\Data\XMLout"

I got

Code:

Output is a directory, but input is not

So I reran it after adding

Code:

-s:"C:\Data\XMLin"

and it worked by producing N identical copies of output files containing output from all N input files. I would prefer to generate just one file but this works too. Thanks!

However, I just realized that I forgot about something when posting my original question. When processing each individual file, I pass inside the transform a parameter whose value is different for each source file. So before, when I was processing one source file at a time, I would do something like this:

Code:

Transform.exe -s:inFILE1.xml -xsl:myXform.xsl -o:outFILE1.xml param=100
Transform.exe -s:inFILE2.xml -xsl:myXform.xsl -o:outFILE2.xml param=200
...

Now I execute just one command, so how can I pass inside the whole array of param values? If I can't, then I must go back to producing one output XML for each input XML. IN this case I have to concatenate afterwards. How can I do that?Again, in this case each output XML will have its entire content in a single <data></data> element and I will have to pull out this content and dump it into a single XML output with a <data></data> element.

Thanks much.

mhkay · December 13th, 2010, 01:56 PM

As I said before, -o should be the name of the single output file - not a directory.

ilyaz · December 14th, 2010, 11:33 AM

At this point I have the following Xform:

Code:

<xsl:transform
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
 xmlns:xs="http://www.w3.org/2001/XMLSchema"
 version="2.0"
 exclude-result-prefixes="xs"
>
<xsl:output method="xml" indent="yes"/>
<xsl:strip-space elements="*"/>

xsl:template name="main">
<data>
     <xsl:apply-templates select="collection('file:///C:/Data/XMLout/?select=*.xml') "/> 
</data>	 
</xsl:template>  

<xsl:template match="data">
     <xsl:copy-of select="*|text()|comment()"/> 
</xsl:template>

I invoke Saxon as follows:

Code:

Transform.exe -xsl:merge.xsl -o:"C:\Data\merged.xml" -it:main

It partially works. What it does is that it appears to create an XML document with a single top-level <data> eleent, as I want. However, it cuts output midway and produces the following message:

Code:

Error on line 1409 column 30 of merged.xml:
  SXXP0003: Error reported by XML parser: XML document structures must start and end within the same entity.
Transformation failed: Run-time errors were reported

Line 1409 is the last one in the malformed output and the last three lines look like this:

Code:

	<container type="Entity" id="262" shortName="الفلسطينيين">
      <property name="Type">LOCATION</property>
      <property name="Value">

Note that the source is a UTF-8 XML that contains some Arabic text in it.

I checked the source file from which the output is being generated and it looks well-formed. What else can be going wrong here?

Thanks much