XSLT can't produce binary output. Using 2.0 you can however produce base64 or hexBinary encoded output.
In Saxon 8.x, if an extension function returns byte[], Saxon translates this into an XPath 2.0 xs:integer*, that is, a sequence of integers. You can then convert this to a base64binary value using the extension function saxon:octetsToBase64Binary(), or to a hexBinary value using saxon:octetsToHexBinary(). You could then output these values to a result document and use some non-XSLT process to convert the result document into real binary.
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference