Unfortunately you've only described your transformation by means of one or two examples; it's difficult to infer the general rules from that.
In many cases it's possible to apply regular expression processing to individual text nodes, in which case you need to process the markup down to leaf node level using xsl:apply-templates, and then use xsl:analyze-string only when you get to the text nodes.
In some cases you need to process a piece of content containing text and markup as a whole. There are two strategies for this, neither of them particularly easy.
One is to start by converting the markup to text, for example, convert H<sub>2</sub>O to H\2/O (using some suitable characters that won't confuse things, eg Unicode private use characters). You can then use regex processing on the whole string (using analyze-string), putting markup back where needed.
The other approach is to convert all the structural markers in the text to markup, for example change "[ABC1980]" to <cite>ABC1980</cite> which you can then process as a tree in the normal way.
