Guava’s CacheBuilder class allows you to do this easily.
CacheBuilder.newBuilder().weakKeys().build()
Note that this changes key equality semantics to be == instead of .equals() which will not matter in your case of using Class instances but is a potential pitfall.