OpenCV has quite good performance. It should be sufficient for most cases.
To improve performance, you can also use OpenCV together with Intel IPP, which is however a non-free commercial product. If OpenCV detects that IPP is installed it will use it where possible.
As a third option you can use IPP directly. IPP was designed with high performance (on Intel architectures) as a goal. It is optimized to use the intel SIMD instructions.