Consider this:
Set<String> x = new HashSet<String>();
x.add("foo");
ImmutableSet<String> guava = ImmutableSet.copyOf(x);
Set<String> builtIn = Collections.unmodifiableSet(x);
x.add("bar");
System.out.println(guava.size()); // Prints 1
System.out.println(builtIn.size()); // Prints 2
In other words, ImmutableSet is immutable despite whatever collection it’s built from potentially changing – because it creates a copy. Collections.unmodifiableSet prevents the returned collection from being directly changed, but it’s still a view on a potentially-changing backing set.
Note that if you start changing the contents of the objects referred to by any set, all bets are off anyway. Don’t do that. Indeed, it’s rarely a good idea to create a set using a mutable element type in the first place. (Ditto maps using a mutable key type.)