It might be worth pointing out that you've used the word "tag" 6 times in your question, and each time you actually mean "element". An element has two tags, a start tag and an end tag. So the "second itemset tag" is actually the end tag of the first itemset element. A common mistake.
I'm not actually sure what you're trying to do here. This call is obviously wrong:
key('itemkey','@itemRef')
and my first instinct is to correct it to:
key('itemkey',@itemRef)
but that doesn't look right either, because the context node is an Itemset element, and Itemset elements don't have @itemRef attributes. In fact your key definition itself looks rather strange:
<xsl:key name="itemkey" match="pmml:ItemRef" use="parent::pmml:Itemset/@id"/>
Do you really want to index ItemRef elements on the id of their containing Itemset?
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference