Because final marks the reference, not the object. You can’t make that reference point to a different hash table. But you can do anything to that object, including adding and removing things.
Your example of an int is a primitive type, not a reference. Final means you cannot change the value of the variable. So, with an int you cannot change the value of the variable, e.g. make the value of the int different. With an object reference, you cannot change the value of the reference, i.e. which object it points to.