Finally, I’ve found it! After thorough testing and experimentation my conclusions are:
-
The correct way is to calculate maximum possible difference between the two colors.
Formulas with any kind of estimated average/typical difference had room for discontinuities. -
I was unable to find a working formula that calculates the distance without blending RGBA colors with some backgrounds.
-
There is no need to take every possible background color into account. It can be simplified down to blending maximum and minimum separately for each of R/G/B channels:
- blend the channel in both colors with channel=0 as the background, measure squared difference
- blend the channel in both colors with channel=max as the background, measure squared difference
- take higher of the two.
Fortunately blending with “white” and “black” is trivial when you use premultiplied alpha.
The complete formula for premultiplied alpha color space is:
rgb *= a // colors must be premultiplied
max((r₁-r₂)², (r₁-r₂ - a₁+a₂)²) +
max((g₁-g₂)², (g₁-g₂ - a₁+a₂)²) +
max((b₁-b₂)², (b₁-b₂ - a₁+a₂)²)
C Source including SSE2 implementation.