How about something like this:
<xsl:copy-of select="$msgop/a/msg[position() < 5]"/>
That will output the first 5 <msg> elements in the variable you have created above.
If you want to just use these first 5 <msg> elements to do some other processing then call <xsl:apply-templates> instead of <xsl:copy-of>.
To comment on your example XSLT specifically - your recursive template would a) never exit as it always calls itself again (use xsl:choose, xsl:when and xsl:otherwise to do this). b) you don't need the 'global' variable msgcnt, c) You would need to pass 0 in to the first call to xsl:with-param, and d) you've obviously not tried to run this as there are obvious spelling mistakes.