If you take a look at the contents of Makefile you’ll see something as follows:
BUILDDIR = build
...
clean:
-rm -rf $(BUILDDIR)/*
This means that make clean just removes the build directory so, with regard to version control, ignoring the contents of the build directory should be enough as you already suspected.