Super-simple answer
If the terminal is not already displayed, I hit Ctrl+J.
If the terminal is displayed, I hit Ctrl+J twice.
Note: Ctrl is Command on Mac.
More complicated answer
Open the Command Palette and search for Preferences: Open Keyboard Shortcuts for details of current keyboard shortcuts.

Some people (and the official shortcuts reference) report that the following work out-of-the-box:
- Mac: Command + ` or
- Win/Linux: Ctrl + `
(that’s a backtick or grave, not an apostrophe, single-quote, or prime)
As far as I can see, by default as of 1.72.2 and 1.73.0-insider, this shortcut is not bound by default (on Windows, at least).

This could be because, on Windows, the backtick key on many (most?) PC keyboards is an OEM key (in my case, oem_8).
If you want to manually add the bindings to toggle between terminal and editor, do this:
-
Open keyboard shortcuts (Command Palette > Preferences: Open Keyboard Shortcuts)

-
Find Terminal: Focus Terminal, set your preferred keybinding, and set the When value to
!terminalFocus(right-click > Change When Expression) -
Find View: Focus Active Editor Group, set your preferred keybinding (can be same so it acts as a toggle), and set the When value to
terminalFocus.