I just had this problem too.
My problem was that i had a close button in my modal
<button class="close" data-dismiss="modal">×</button>
Pressing enter in the input field caused this button to be fired. I changed it to an anchor instead and it works as expected now (enter submits the form and does not close the modal).
<a class="close" data-dismiss="modal">×</a>
Without seeing your source, I can’t confirm that your cause is the same though.