You need to appreciate that ' is translated to ' by the XML parser, so this produces the text
concat('try', 'hello')
which is what the XPath processor then evaluates.
I find the simplest way to handle quotes and apostrophes is to declare
<xsl:variable name="apos">'</xsl:variable>
<xsl:variable name="quot">"</xsl:variable>
and then use things like concat($quot, 'hello', $quot)
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference