Update Sept 2022.
They now use VSCode builtin inlay function from 1.67
I setup mine as follow:
{
"editor.inlayHints.enabled": "offUnlessPressed"
}
Then you can toggle them with Ctrl+Alt pressed.
Old Answer
There is now a togglable command (Ctrl+Shift+P) : Rust Analyzer: Toggle inlay hints