This kind of thing tends to be processor-dependent - you need to ask the question of a specific processor, it's not an XSLT language issue. Some processors may support relative URI addressing within a JAR file, others may not. (With Saxon, it depends which version you are using). In all cases, relative URI addressing will only work if the base URI is known, and that depends on how you identify the styesheet to the XSLT processor.
The jar: URL scheme is not particularly well documented - I think the best description I found was in the JDK Javadoc documentation for class java.net.JarURLConnection. A complication is that the syntax of jar: URLs isn't actually compatible with the RFCs for URIs, and isn't in fact supported by Sun's own java.net.URI class. This technically means that an XSLT processor that supports these URLs is non-conformant...
Michael Kay
http://www.saxonica.com/
Author, XSLT Programmer's Reference and XPath 2.0 Programmer's Reference