A "node-set" in XSLT (you're using XSLT 1.0 terminology, but never mind) is a set of nodes, that is, it's the tree representation of some XML data. Your Java method is expecting a string in the form of unparsed lexical XML. You seem to be expecting that the system will work this out for itself, and "unparse" the data (that is, serialize it) to meet the expectations of your function. Well, it doesn't. There is no reason why the string should be well-formed XML, so it's not surprising that the XML parser throws it out.
Why are you trying to parse it anyway? You don't need to, it's already been parsed. For that matter, why are you using an extension function to do something that would be much better done in XSLT?
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference