int has lots of bases.
*, 0*, 0x*, 0b*, 0o* and it can be long, it takes time to determine the base and other things
if the base is set, it saves a lot of time
python -m timeit "int('1',10)"
1000000 loops, best of 3: 0.252 usec per loop
python -m timeit "int('1')"
1000000 loops, best of 3: 0.594 usec per loop
as @Martijn Pieters metions the code the Object/intobject.c(int_new) and Object/floatobject.c(float_new)