NuGet now has the ability for you to re-download the missing packages as a pre-build step, meaning that you only need to commit your packages.config file (and include nuget.exe in a tools folder).
Read Using NuGet Without Committing Packages to Source Control for more details.