How to limit the height of the modal?

Question

Use viewport units with calc. Like this:

.img-responsive {
    max-height: calc(100vh - 225px);
}

…where the 225px corresponds to the combined height of the top and bottom sections of the viewport which surround the dialog.

Also, in order to take care of the width of the modal we need to set a few more properties:

.modal {
    text-align:center;
}
.modal-dialog {
    display: inline-block;
    width: auto;
}

Updated Fiddle (Resize the viewport height to see the effect)

We can replace calc with a padding + negative margin technique like so:

.img-responsive {
    max-height: 100vh;
    margin: -113px 0;
    padding: 113px 0;
}

PS: browser support for viewport units is very good