The BottomSheetBehavior does not work well with LayoutTransition (animateLayoutChanges="true") for now. I will work on a fix.
For now, you can use Transition instead. Something like this will fade the view inside and animate the size of the bottom sheet.
ViewGroup bottomSheet = ...;
View hidingView = ...;
TransitionManager.beginDelayedTransition(bottomSheet);
hidingView.setVisibility(View.GONE);
You can refer to Applying a Transition for more information including how to customize the animation.