I don’t see any UB here; your types are unsigned so signed-overflow UB is impossible, and there’s nothing weird. (And even if signed, it would have to produce correct outputs for inputs that don’t cause overflow UB, like rdi=1). It’s broken with GCC’s C++ front-end as well.
Also, GCC8.2 compiles it correctly for AArch64 and RISC-V (to an madd instruction after using movk to construct constants, or RISC-V mul and add after loading the constants). If it was UB that GCC was finding, we’d generally expect it to find it and break your code for other ISAs as well, at least ones that have similar type widths and register widths.
Clang also compiles it correctly.
This appears to be a regression from GCC 5 to 6; GCC5.4 compiles is correctly, 6.1 and later don’t. (Godbolt).
You can report this on GCC’s bugzilla using the MCVE from your question.
It really looks like it’s a bug in x86-64 System V struct-return handling, perhaps of structs containing padding. That would explain why it works when inlining, and when widening a to uint64_t (avoiding padding).