What is the (…) called in C and C++?

It is one of the punctuators.

6.4.6  Punctuators
Syntax punctuator:
     one of  [    ]    (    )    {   }    .    ->
             ++   --   &    *    +   -    ~    !
             /    %    <<   >>   <   >    <=   >=    ==   !=   ^   |   &&   ||
             ?    :    ;    ...
             =    *=   /=   %=   +=  -=   <<=  >>=   &=   ^=   |=
             ,    #    ##
             <:   :>   <%   %>   %:   %:%:

In the function declaration it is called the ellipsis.

Ellipsis is also used by some compiler C language extensions.
Example – gcc switch/case range extension

const char *test(unsigned num)
{
    switch(num)
    {
        case 0 ... 9:
            return "the value is in the 0 to 9 range";
        case 10 ... 99:
            return "the value is in the 10 to 99 range";
        default:
            return "out of tested range";
    }
}

https://godbolt.org/z/YBLma-