Is it semantically incorrect to put a or inside of a ?

The current HTML5 draft says it is incorrect.

http://www.w3.org/TR/2012/WD-html5-20120329/the-button-element.html#the-button-element says that a <button> must contain only Phrasing content. Phrasing content is defined as including <span> but not <div>.