You must put IFRAME in modal panel and execute javascript that will set src of that iframe when modal panel is shown ("onshow" attribute).

<rich:modalPanel onshow="loadIframe()">
     <iframe id="myIframe"></iframe>
</rich:modalPanel>
<script>
function myIframe() {
     $('myIframe').src='URLYouWant'; // assuming $ belongs Prototype
}</script>