Not really sure what you're trying to achieve here, but start by doing it using template rules rather than xsl:for-each. Something like:
<xsl:template match="li">
<dt><a href="{@href}"><xsl:value-of select="@name"/></a></dt>
<xsl:if test="*">
<dd id="smenu1">[list]
<xsl:apply-templates select="li"/>
</ul></dd>
</xsl:if>
</xsl:template>
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference