The non-standard functions showModalDialog and showModelessDialog are not fully supported. The browser-driven implementation of these functions is not fully compatible with the implementation of Internet Explorer. In particular javascript-heavy pages that are loaded in such a dialog may not function correctly.
Try to use a GUI-level testing approach in such situations.