There can be a significant performance difference, of an order of magnitude for multiplications and multiple orders of magnitude for indexing a few random values.
I was actually wondering about the same thing and came across this interesting comparison:
http://penandpants.com/2014/09/05/performance-of-pandas-series-vs-numpy-arrays/