From the Javadoc:
Note that this implementation is not synchronized. If multiple threads access a hash set concurrently, and at least one of the threads modifies the set, it must be synchronized externally
Reading doesn’t modify a set, therefore you’re fine.