What happens when there’s insufficient memory to throw an OutOfMemoryError?

The JVM never really runs out of memory. It does memory computation of the heap stack in advance.

The Structure of the JVM, Chapter 3, section 3.5.2 states:

• If Java virtual machine stacks can be dynamically expanded, and expansion is attempted but insufficient memory can be made available
  to effect the expansion, or if insufficient memory can be made
  available to create the initial Java virtual machine stack for a new
  thread, the Java virtual machine throws an OutOfMemoryError.

For Heap, Section 3.5.3.

• If a computation requires more heap than can be made available by the automatic storage management system, the Java virtual machine
  throws an OutOfMemoryError.

So, it does a computation in advance before doing allocation of the object.

What happens is that the JVM tries to allocate memory for an object in the memory called Permanent Generation region (or PermSpace). If allocation fails (even after the JVM invokes the Garbage Collector to try & allocate free space), it throws an OutOfMemoryError. Even exceptions requires a memory space so the error will be thrown indefinitely.

Further reading.? Furthermore, OutOfMemoryError can occur in different JVM structure.