The functional notation is on :not(), not :hover:
a:not(.active):hover
If you prefer to put :hover first, that’s fine:
a:hover:not(.active)
It doesn’t matter which pseudo-class comes first or last; either way, the selector works the same. It just happens to be my personal convention to put :hover last as I tend to place user-interaction pseudo-classes behind structural pseudo-classes.