The answers so far are correct for the Cuda side of things, but there’s also an issue on the ipython side of things.
When you have an error in a notebook environment, the ipython shell stores the traceback of the exception so you can access the error state with %debug. The issue is that this requires holding all variables that caused the error to be held in memory, and they aren’t reclaimed by methods like gc.collect(). Basically all your variables get stuck and the memory is leaked.
Usually, causing a new exception will free up the state of the old exception. So trying something like 1/0 may help. However things can get weird with Cuda variables and sometimes there’s no way to clear your GPU memory without restarting the kernel.
For more detail see these references:
https://github.com/ipython/ipython/pull/11572
How to save traceback / sys.exc_info() values in a variable?