This is a known issue.
I use lots of modal dialogs and have gotten around this problem by creating a helper page (aspx) that has a single iframe that fills the window. Inside of that goes the actual page that will be seen. This causes all the postbacks to happen inside the iframe window instead of in the modal dialog window.
I believe another way to fix it is to add the <base> tag with the target specified. But I'm not sure what you'd specify it too. Probably the name of the modal dialog window.