Keep a reference to the sublayer added
Remove the sublayer from the super layer when not needed.
The code would be like:
Obj C:
[thesublayer removeFromSuperlayer]
Swift:
thesublayer.removeFromSuperlayer()
//thesublayer is the name of the layer you want to remove