This is a long-running missing feature in jupyter notebooks. I use a near-identical setup: my notebook runs inside a tmux session in a remote server, and I use it locally with ssh tunneling.
Before doing any work, I run the following snippet in the first cell:
import sys
import logging
nblog = open("nb.log", "a+")
sys.stdout.echo = nblog
sys.stderr.echo = nblog
get_ipython().log.handlers[0].stream = nblog
get_ipython().log.setLevel(logging.INFO)
%autosave 5
Now let’s say, I run a cell that will take a while to complete (like a training run). Something like:
import time
def train(num_epochs):
for epoch in range(num_epochs):
time.sleep(1)
print(f"Completed epoch {epoch}")
train(1000)
Now while train(1000) is running, after the first 10 seconds, I want to do something else and close the browser, and also disconnect from my remote connection.
(Note the modified short autosave duration; I added that as I often forget to save the notebook before closing the browser tab.)
After 500 seconds have passed, I can reconnect to the remote server and open the notebook in my browser. My logs of this cell will have stopped printing after “Completed epoch 9”, i.e. when I disconnected. However, the kernel will still actually be running train in the backend, and it will also show “busy”.
We can now just simply open up the file nb.log and we’ll find all the logs, including the ones after we closed the browser and connection. We can keep refreshing the nb.log file at our leisure and new logs will keep coming up, till the kernel finishes running train().
Now if we want to stop train() before it’s done, we can just press the Interrupt button in jupyter. The kernel will be freed and we can run other stuff (And a Keyboard Interrupt error message will also show up in your nb.log file). All our precomputed notebook variables and imported libraries are still there, as the kernel wasn’t actually disconnected.
Although this isn’t a very sophisticated solution, I find it quite easy to implement