I don't think you can assign a complete document object to a window to have it rendered.
What you can try is serializing the transformation result and document.writing it to the new window:
Code:
var win = window.open('', 'windowName');
win.document.open();
win.document.write(new XMLSerializer().serializeToString(xsltProcessor.transformToDocument(xml)));
win.document.close();
That however is a pretty awful approach as the transformation result is first serialized as XML and then document.written and that way parsed as text/html.
What you should use is transformToFragment, passing in the document of an already existing window or frame as the second argument, and then you can appendChild the fragment returned by transformToFragment to a node (e.g. the body element) in the existing document.