As a heuristic, you could construct a graph where each node represents a set of contiguous, same-colour squares, and each node is connected to those it touches. (Each edge weighted as 1). You could then use a path-finding algorithm to calculate the “distance” from the top left to all other nodes. Then, by looking the results of flood-filling using each of the other 5 colours, determine which one minimizes the distance to the “furthest” node, since that will likely be your bottleneck.
Add the result of that calculation to the number of fills done so far, and use that as your A* heuristic.