Looks to me as if
<xsl:when test="following-sibling::*/@category=library/@category">
is wrong for four reasons:
(a) you only want to look at the immediately following sibling, which is following-sibling::*[1]/@category
(b) the path library/@category is wrong because (i) the library element doesn't have an @category attribute, and (ii) the context node in a book element which doesn't have a library child
(c) you don't want to do something different when the following sibling has the same value, you want to do something different when the preceding sibling doesn't have the same value.
So I think you want:
<xsl:if test="not(preceding-sibling::*[1]/@category=@category)">
<xsl:value-of select="@category"/>
</xsl:if>
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference