The regex attribute of xsl:analyze-string is an attribute value template, so curly braces need to be doubled. Alternatively put the regex in a variable:
<xsl:variable name="regex">(\d{1,2}Y)(\d{1,2}M)(\d{1,2}D)</xsl:variable>
<xsl:analyze-string select="'4Y6M12D'" regex="{$regex}">
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference