You can get the current transformation matrix that is applied to an element by using getComputedStyle(). You can use this to transform the current mouse position to its position in transformed space and see whether the click/drag events are within the element boundary and/or corners. Good resources for this:
The CSS3 matrix() Transform for the Mathematically Challenged
http://www.eleqtriq.com/2010/05/css-3d-matrix-transformations/
BTW, as you’re experiencing, this is non-trivial to code. We had to do it for Sencha Animator, and it was a beast.