Others have mentioned that certain constructs such as Collections require objects and that objects have more overhead than their primitive counterparts (memory & boxing).
Another consideration is:
It can be handy to initialize Objects to null or send null parameters into a method/constructor to indicate state or function. This can’t be done with primitives.
Many programmers initialize numbers to 0 (default) or -1 to signify this, but depending on the scenario, this may be incorrect or misleading.
This will also set the scene for a NullPointerException when something is being used incorrectly, which is much more programmer-friendly than some arbitrary bug down the line.