Turn off enclosing tags in CKEditor 3.0

CKEDITOR.config.enterMode = CKEDITOR.ENTER_BR; – this works perfectly for me.
Have you tried clearing your browser cache – this is an issue sometimes.
You can also check it out with the jQuery adapter:

<script type="text/javascript" src="https://stackoverflow.com/js/ckeditor/ckeditor.js"></script>
<script type="text/javascript" src="/js/ckeditor/adapters/jquery.js"></script>
<script type="text/javascript">
$(function() {
    $('#your_textarea').ckeditor({
        toolbar: 'Full',
        enterMode : CKEDITOR.ENTER_BR,
        shiftEnterMode: CKEDITOR.ENTER_P
    });
});
</script>

UPDATE according to @Tomkay’s comment:

Since version 3.6 of CKEditor you can configure if you want inline content to be automatically wrapped with tags like <p></p>. This is the correct setting:

CKEDITOR.config.autoParagraph = false;

Source:
http://docs.cksource.com/ckeditor_api/symbols/CKEDITOR.config.html#.autoParagraph