Well depending on the platform or XSLT processor there might be an SGML parser you can simply plug into the chain of processing, for instance for .NET there was or might still be an SgmlReader class implementing the XmlReader API that is central to all other XML APIs in .NET so there you could use XslCompiledTransform for XSLT 1.0 or probably even Saxon 9 .NET for XSLT 2.0 and feed an SGML document to the XSLT processor, with a few lines of .NET code running the transformation.
Similar solutions might exist in the Java world, like there are ways to feed SGML based HTML 4 to XSLT with APIs like TagSoup.
But the primary input for XSLT is XML, not SGML. I think there is also a transformation language for SGML documents but I am not familiar with that.
Microsoft MVP (XML, Data Platform Development) 2005/04 - 2013/03