There’s a method called Markov Chains. Scan a few model files of both kinds and for each byte value from 0 to 255 gather stats (basically probability) of a subsequent value. This will give you a 64Kb (256×256) profile you can compare your runtime files against (within a % threshold).
Supposedly, this is how browsers’ Auto-Detect Encoding feature works.