Wouldn't you be better off using a tool such as JTidy? Cleaning up bad HTML is a fairly major project, and you seem to be tackling it in a fairly ad-hoc manner.
Having said that, if you want to group consecutive li elements that aren't children of a ul, you can do it like this:
<xsl:template match="p">
<xsl:for-each-group select="*" group-adjacent="boolean(self::li)">
<xsl:choose>
<xsl:when test="current-grouping-key()">
[list]<xsl:copy-of select="current-group()"/></ul>
</xsl:when>
<xsl:otherwise>
<xsl:copy-of select="current-group()"/>
</xsl:otherwise>
</xsl:for-each-group>
</xsl:template>
But that's only one of very many rules you will have to apply.
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference