The following doesn’t answer your question directly but you might find it interesting; it says that IP packets can be disassembled/reassembled, and therefore bigger than limit on the underling media (e.g. 1500-byte Ethernet): Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC
- Re: UDP fragmentation says you should use ICMP instead of UDP to discover MTU
- Path MTU Discovery says that a TCP connection might include implicit MTU negotiation via ICMP
I don’t know about generating ICMP via an API on Windows: at one time such an API was proposed, and was controversial because people argued that would make it easy to write software that implements denial-of-service functionality by generating a flood of ICMP messages.
No, it looks like it is implemented: see for example Winsock Programmer’s FAQ Examples: Ping: Raw Sockets Method.
So, to discover MTU, generate ping packets with the ‘do not fragment’ flag.
Maybe there’s an easier API than this, I don’t know; but I hope I’ve given you to understand the underlying protocol[s].