I think the adjective "well-formed" is rather unfortunate, because talking about "testing XML to see if it is well-formed" suggests that there can be XML that isn't well-formed. In fact all XML is well-formed, if it's not well-formed then it isn't XML.
The way to test whether an input file contains well-formed XML is to try parsing it with an XML parser, and if you get an error then it isn't.
XSLT and XPath can only process well-formed XML. If the input isn't well-formed you'll get an error from the XML parser long before the XSLT or XPath processor kicks in.
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference