All you need for this is a single template in your xsl file:
Code:
<xsl:template match="@*|node()">
<xsl:copy>
<xsl:apply-templates select="@*|node()">
<xsl:sort data-type="number" select="loadorder"/>
</xsl:apply-templates>
</xsl:copy>
</xsl:template>
Basically, as the xslt processor goes through your xml document it will attempt to apply the sort to each node-set, but the only node that contains a "loadorder" child element is the library node, so the sort will effectively be ignored for all other nodes.
hth
Phil