Color matrix is the entity that defines new color component values for some pixel as linear functions of the current color components of the same pixel. I.e. the only input for color matrix conversion is the pixel, which colors should be transformed. There is no way to involve another pixels to such conversion. Regardless if they are pixels from another image or even neighbors of transformed pixel – it is impossible.