It’s not just the cost of passing a pointer (that’s essentially what a reference is), but also the de-referencing in the called method’s body to retrieve the underlying value.
That’s why passing an int by value will be virtually guaranteed to be faster (Also, the compiler can optimize and simply pass the int via processor registers, eliminating the need to push it onto the stack).