masksToBounds
Any sublayers of the layer that extend outside its boundaries will be clipped to those boundaries. Think of the layer, in that case, as a window onto its sublayers; anything outside the edges of the window will not be visible. When masksToBounds is NO, no clipping occurs.
When the value of this property is true, Core Animation creates an implicit clipping mask that matches the bounds of the layer and includes any corner radius effects. If a value for the mask property is also specified, the two masks are multiplied to get the final mask value.
you can get the more information in API Reference.
clipToBounds
The use case for clipsToBounds is more for subviews which are partially outside the main view. For example, I have a (circular) subview on the edge of its parent (rectangular) UIView. If you set clipsToBounds to YES, only half the circle/subview will be shown. If set to NO, the whole circle will show up. Just encountered this so wanted to share
for more information sample link