Ok so it turns out there is a bug in Twitter Bootstrap. It has been opened and closed a couple times.
There is currently a work around:
1) Do not use trigger: focus when initializing bootstrap popovers
2) Instead use data-trigger="focus" as an attribute to the items which will trigger the popover
3) Items which have a popover will need to have tabindex="-1" explicitly declared.
Cross-browser working example here: http://jsfiddle.net/v5L7m/3/
I have also confirmed this in a real-world use case scenario.
Edit: @xanderiel noted that tabindex=0 now appears to cause a browser native focus border which is what my original answer called for. They say a tabindex of -1 eliminates this border.