The reason that you’re not finding a name for this is that it’s not an operating system — it’s the absence of an operating system. Often this is called something like “bare-metal” programming.
The general idea of bare-metal programming is that there is a small bit of general-purpose code — a “bootloader” — that sets up the memory controller and other hardware things on the board, and then that transfers control to your program. (Operating systems also have bootloaders, so in that sense your program is replacing the operating system.) Uboot is a fairly common open-source bootloader, so that might be a good place to start looking for information.
One of the tricky bits about bare-metal programming is that, since there isn’t an operating system in place to handle any of the hardware communication, you have to think about “what does a printf actually mean as far as what data goes to what peripheral?” and “how do I make it go there?” Again, some bootloaders provide support for this sort of thing, though it’s not always trivial to connect it all up. Again, Uboot is a good example.
Meanwhile, the C library itself is actually going to be provided by your compiler, rather than the bootloader.
(I should also add, as a name note: The company I work for makes a series of bare-metal and Linux compilers, known as Sourcery CodeBench. For CodeBench, the bare-metal versions are generally named after the ABI specification they use for linking programs, so the “ELF” or “EABI” versions are all bare-metal compilers, and I think that’s a pretty common way of referring to this sort of thing, so you’ll see that sort of name around as well.)