How do I remove Flutter IconButton big padding?

Simply pass an empty BoxConstrains to the constraints property and a padding of zero.

IconButton(
    padding: EdgeInsets.zero,
    constraints: BoxConstraints(),
)

You have to pass the empty constrains because, by default, the IconButton widget assumes a minimum size of 48px.