Some more detail for completeness in case it helps someone…
Note that the most common reason for this exception these days is attempting to load a 32 bit-specific (/platform:x86) DLL into a process that is 64 bit or vice versa (viz. load a 64 bit-specific (/platform:x64) DLL into a process that is 32 bit). If your platform is non-specific (/platform:AnyCpu), this won’t arise (assuming no referenced dependencies are of the wrong bitness).
In other words, running:
%windir%\Microsoft.NET\Framework\v2.0.50727\installutil.exe
or:
%windir%\Microsoft.NET\Framework64\v2.0.50727\installutil.exe
will not work (substitute in other framework versions: v1.1.4322 (32-bit only, so this issue doesn’t arise) and v4.0.30319 as desired in the above).
Obviously, as covered in the other answer, one will also need the .NET version number of the installutil you are running to be >= (preferably =) that of the EXE/DLL file you are running the installer of.
Finally, note that in Visual Studio 2010, the tooling will default to generating x86 binaries (rather than Any CPU as previously).
Complete details of System.BadImageFormatException (saying the only cause is mismatched bittedness is really a gross oversimplification!).
Another reason for a BadImageFormatException under an x64 installer is that in Visual Studio 2010, the default .vdproj Install Project type generates a 32-bit InstallUtilLib shim, even on an x64 system (Search for “64-bit managed custom actions throw a System.BadImageFormatException exception” on the page).