I went through the developer docs again and found that:
On API 20 and before, CardView does not clip the bounds of the Card for the rounded corners. Instead, it adds padding to content so that it won’t overlap with the rounded corners.
So for pre-lollipop devices I have to call setPreventCornerOverlap (false); on the cardview.
Update: The same can be done through xml code by adding app:cardPreventCornerOverlap="false" in card view.