Can someone explain what exactly the comment warns me about (and the context where valueOf shouldn’t be used), and what could possibly go wrong.
The Integer class creates and maintains a cache of Integer objects representing small integer values; by default, values in the range -128 to 127 are covered (more discussion here, here, and many other places). Integer.valueOf() will return an object from this cache when its argument represents a number in the range. The comment is warning that parseInt() must not rely on valueOf() because the former may be invoked before that cache is populated.
The misbehavior that could be expected in that case is not specified, and conceivably might vary between Java versions, but plausible possibilities are that null would be returned or an exception (NullPointerException, IndexOutOfBoundsException, …) would be thrown.
In any case, this is an internal comment in the implementation, not a comment to users of class Integer. By the time any user code runs, the necessary cache initialization is complete, and Integer.valueOf() can be relied upon to behave fully as its API documentation describes.