Not supported in GFM
Embedding an <iframe> into GitHub-flavored Markdown (GFM) is not supported. Here’s the official stance on it from their specs:
6.11 Disallowed Raw HTML (extension)
GFM enables the tagfilter extension, where the following HTML tags will be filtered when rendering HTML output:
<title>
<textarea>
<style>
<xmp>
<iframe>
<noembed>
<noframes>
<script>
<plaintext>[…] These tags are chosen in particular as they change how HTML is interpreted in a way unique to them […], and this is usually undesireable in the context of other rendered Markdown content.
All other HTML tags are left untouched.
Possible Work-around
Similar to solutions mentioned for videos in other answers, you could embed a screenshot of your map, and make it a link to the URL of your OSM map section:
