The basic idea of this algorithm is to use a pre-processing step to find all edges and then select the correct smoothing tile according to the shape of the edge.
The first step would be to find all edges. In the example below the edge tiles marked with an X are all green tiles with a tan tile as one or more of their eight neighbouring tiles. With different types of terrain this condition could translate to a tile being an edge tile if it has neighbours of lower terrain-number.
Once all edge tiles are detected the next thing to do is to select the right smoothing tile for each edge tile. Here is my representation of your smoothing tiles.
Note that there are actually not that many different types of tiles. We need the eight outer tiles from one of the 3×3 squares but only the four corner squares from the other since the straight-edge tiles are already found in the first square. This means that there in total are 12 different cases we must distinguish between.
Now, looking at one edge tile we can determine which way the boundary turns by looking at its four closest neighbour tiles. Marking an edge tile with X just as above we have the following six different cases.
These cases are used to determine the corresponding smoothing tile and we can number the smoothing tiles accordingly.
There is still a choice of a or b for each case. This depends on which side the grass is on. One way to determine this could be to keep track of the orientation of the boundary but probably the simplest way to do it is to pick one tile next to the edge and see what colour it has. The image below shows the two cases 5a) and 5b) which can be distinguished between by for example checking the colour of the top right tile.
The final enumeration for the original example would then look like this.
And after selecting the corresponding edge tile the border would look something like this.
As a final note I might say that this would work as long as the boundary is somewhat regular. More precisely, edge tiles that do not have exactly two edge tiles as their neighbours will have to be treated separately. This will occur for edge tiles on the edge of the map which will have a single edge neighbour and for very narrow pieces of terrain where the number of neighbouring edge tiles could be three or even four.