What the client does when it gets any file form any webserver is determined by the MIME types defined on the client, so in some cases, the file will be opened in the browser window, in others it will be downloaded as a normal file download.
The closest you can get is either to use a MIME Type that is usually opened - such as .txt / .html - or one that is usually downloaded - such as .zip.
This also means that you have no control over into where the file will open as it will always use the full browser window - if set to open in the window.
You may be better of using the MS Office ActiveX objects - see
http://www.greggriffiths.org/webdev/both/excel/ for an example using ActiveX and Excel.
Force a word download into a frame?
