This code:
<xsl:variable name="isCA1" select="/*/Order/Header[State[.='CA'] and Category[not(.=preceding::Category)]]">
is selecting orders from California provided their category is different from all previous categories. I imagine that what you really want is orders from California provided that their category is different from the categories of all previous California orders. Which would be
select="/*/Order/Header[State[.='CA'] and Category[not(.=preceding::Category[../State='CA'])]]">
Not really an XSLT problem, just an error in your logic...
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference