It is a seldom-used feature from C99 GNU extension (GCC documentation) that is used to forward-declare parameters used in VLA declarators.
matrix_* matrix_insert_values(int n; double a[][n], int m, int n);
Do you see how int n appears twice? The first int n; is just a forward declaration of the actual int n, which is at the end. It has to appear before double a[][n] because n is used in the declaration of a. If you were okay with rearranging parameters, you could just put n before a and then you wouldn’t need this feature
matrix_* matrix_insert_values_rearranged(int m, int n, double a[][n]);
Note about C++ compatibility
To be clear, the GNU extension is just the forward declaration of function parameters. The following prototype is standard C:
// standard C, but invalid C++
matrix_* matrix_insert_values_2(int m, int n, double a[][n]);
You cannot call this function from C++, because this code uses variable length arrays, which are not supported in C++. You would have to rewrite the function in order to be able to call it from C++.