I'm not sure why you're looking for distinct publishers, you don't have any duplicates in your data, and I'm not sure how you would want to handle them if there were any.
Using Muenchian grouping within a subtree, rather than within the full document, is always a little bit tricky. The usual answer is to define a key that makes the name globally unique, in this case concatenating the publisher name and the author name.
Where it's going wrong is here
<xsl:for-each select=".//author[count(.| key('authors',.)[1])=1]">
where you look for authors within the current publisher, and then test them to see if they are the first author with that name in the document. Try changing the key on author use="concat(ancestor::publisher/name, '#', .)", and changing the above to
select=".//author[count(.| key('authors',concat(current()/name, '#', .)[1])=1]">
I don't guarantee that's enough to make it work but it's the essence of the solution.
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference