How to remove the border behind the button?

You have to use style="?android:attr/borderlessButtonStyle":

<ImageButton
    android:id="@+id/btn_photo_lib"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:onClick="startPhotoLibAction"
    android:src="https://stackoverflow.com/questions/14395603/@drawable/library_blau_2"
    style="?android:attr/borderlessButtonStyle"/>