The ordering if the segments change only at intersection points, and only for the segments that do intersect at the given point. This can be implemented by removing the intersecting segments and inserting them again.
The ordering function is by y coordinate, and when ys are equal, by the slope. The intersecting segments will then be inserted in the correct order. As the sweep progresses, the actual y coordinates of the segments’ intersections of the sweep line will change. It doesn’t matter as the order will remain the same (until we swap, that is, remove and insert again, intersecting segments). The actual y coordinate need not be stored anyway. It should be calculated dynamically for any given position of the sweep line, as we insert or remove segments.
The data structure in question should not be called Set, it’s a Map or, more precisely, an Ordered Map. The operation of finding neighbours of a given element is essential here.