Implementing `make check` or `make test`

Your original approach, as stated in the question, is best. Each of your tests is in the form of a pair of expected inputs and outputs. Make is quite capable of iterating through these and running the tests; there is no need to use a shell for loop. In fact, by doing this you are losing the opportunity to run your tests in parallel, and are creating extra work for yourself in order to clean up temp files (which are not needed).

Here’s a solution (using bc as an example):

SHELL := /bin/bash

all-tests := $(addsuffix .test, $(basename $(wildcard *.test-in)))

.PHONY : test all %.test

BC := /usr/bin/bc

test : $(all-tests)

%.test : %.test-in %.test-cmp $(BC)
    @$(BC) <$< 2>&1 | diff -q $(word 2, $?) - >/dev/null || \
    (echo "Test $@ failed" && exit 1)

all : test 
    @echo "Success, all tests passed."

The solution directly addresses your original questions:

• The placeholders you’re looking for are $< and $(word 2, $?) corresponding to the prerequisites %.test-in and %.test-cmp respectively. Contrary to the @reinierpost comment temp files are not needed.
• The diff message is hidden and replaced using echo.
• The makefile should be invoked with make -k to run all the tests regardless of whether an individual test fails or succeeds.
• make -k all will only run if all the tests succeed.

We avoid enumerating each test manually when defining the all-tests variable by leveraging the file naming convention (*.test-in) and the GNU make functions for file names. As a bonus this means the solution scales to tens of thousands of tests out of the box, as the length of variables is unlimited in GNU make. This is better than the shell based solution which will fall over once you hit the operating system command line limit.