The IsolateApps modifier causes ASP.NET to generate a unique key for each application on your server. This is only applicable if you are getting ASP.NET to auto-generate keys at runtime.
If you are not getting ASP.NET to auto-generate keys, and are instead specifying the keys using decryptionKey="{hex-key value}", then the way to get ASP.NET to use a different key for each application is to simply specify a different key in each application’s Web.config.
The IIS config GUI allows you to create a Web.config with an explicit key together with the IsolateApps modifier, which is invalid, and in my opinion is a bug in the config GUI.