Assuming AdvancedFormat is a bool, you need to declare and use a BooleanToVisibilityConverter:
<!-- In your resources section of the XAML -->
<BooleanToVisibilityConverter x:Key="BoolToVis" />
<!-- In your Button declaration -->
<Button
Height="50" Width="50"
Style="{StaticResource MyButtonStyle}"
Command="{Binding SmallDisp}" CommandParameter="{Binding}"
Cursor="Hand" Visibility="{Binding Path=AdvancedFormat, Converter={StaticResource BoolToVis}}"/>
Note the added Converter={StaticResource BoolToVis}.
This is a very common pattern when working with MVVM. In theory you could do the conversion yourself on the ViewModel property (i.e. just make the property itself of type Visibility) though I would prefer not to do that, since now you are messing with the separation of concerns. An item’s visbility should really be up to the View.