I’m not sure what you want. First of all, of course each time you commit/push the directory is going to get a little larger, since it has to store each of those additional commits.
However, probably you want git gc which will “cleanup unnecessary files and optimize the local repository” (manual page).
Another possibly relevant command is git clean which will delete untracked files from your tree (manual page).