You can use the .NET HashSet class that is found under System.Collections.Generic:
$set = New-Object System.Collections.Generic.HashSet[int]
The collection guarantees unique items and the Add, Remove, and Contains methods all operate with O(1) complexity on average.