What you are missing here: the int values in your example go on the stack, not on the heap.
And it is much less overhead to deal with fixed size primitive values existing on the stack – compared to objects on the heap!
In other words: using a “pointer” means that you have to create a new object on the heap. All objects live on the heap; there is no stack for arrays! And objects becomes subject to garbage collection immediately after you stopped using them. Stacks on the other hand come and go as you invoke methods!
Beyond that: keep in mind that the abstractions that programming languages provide to us are created to help us writing code that is easy to read, understand and maintain. Your approach is basically to do some sort of fine tuning that leads to more complicated code. And that is not how Java solves such problems.
Meaning: with Java, the real “performance magic” happens at runtime, when the just-in-time compiler kicks in! You see, the JIT can inline calls to small methods when the method is invoked “often enough”. And then it becomes even more important to keep data “close” together. As in: when data lives on the heap, you might have to access memory to get a value. Whereas items living on the stack – might still be “close” (as in: in the processor cache). So your little idea to optimize memory usage could actually slow down program execution by orders of magnitude. Because even today, there are orders of magnitude between accessing the processor cache and reading main memory.
Long story short: avoid getting into such “micro-tuning” for either performance or memory usage: the JVM is optimized for the “normal, typical” use cases. Your attempts to introduce clever work-arounds can therefore easily result in “less good” results.
So – when you worry about performance: do what everybody else is doing. And if you one really care – then learn how the JVM works. As it turns out that even my knowledge is slightly outdated – as the comments imply that a JIT can inline objects on the stack. In that sense: focus on writing clean, elegant code that solves the problem in straight forward way!
Finally: this is subject to change at some point. There are ideas to introduce true value value objects to java. Which basically live on the stack, not the heap. But don’t expect that to happen before Java10. Or 11. Or … (I think this would be relevant here).