You can create conditional references in the project file like this:
<Reference Include="32bit.dll" Condition="'$(Platform)'=='x86'"/>
<Reference Include="64bit.dll" Condition="'$(Platform)'=='x64'"/>
To use this inside VS, you have to create two solution platforms: one for the x86 target and one for the x64 target. Depending on the active platform one of the dlls will be selected, no need for re-referencing.
To automate this using msbuild, create a new project file that builds the other project file a number of times, each time for a different platform/configuration/…:
<Target Name="BuildAll">
<MSBuild Targets="myproject.proj" Properties="Platform=x86;Configuration=Debug"/>
<MSBuild Targets="myproject.proj" Properties="Platform=x64;Configuration=Debug"/>
<MSBuild Targets="myproject.proj" Properties="Platform=x64;Configuration=Release"/>
</Target>
Have a look at the MSBuild task reference for aditional options like building in parallel.