Django long running asynchronous tasks with threads/processing

os.fork

A fork will clone the parent process, which in this case, is your Django stack. Since you’re merely wanting to run a separate python script, this seems like an unnecessary amount of bloat.

subprocess

Using subprocess is expected to be interactive. In other words, while you can use this to effectively spawn off a process, it’s expected that at some point you’ll terminate it when finished. It’s possible Python might clean up for you if you leave one running, but my guess would be that this will actually result in a memory leak.

threading

Threads are defined units of logic. They start when their run() method is called, and terminate when the run() method’s execution ends. This makes them well suited to creating a branch of logic that will run outside the current scope. However, as you mentioned, they are subject to the Global Interpreter Lock.

multiprocessing

This module allows you to spawn processes, and it has an API similar to that of threading. You could say is like threads on steroids. These processes are not subject to the Global Interpreter Lock, and they can take advantage of multi-core architectures. However, they are more complicated to work with as a result.

So, your choices really come down to threads or processes. If you can get by with a thread and it makes sense for your application, go with a thread. Otherwise, use processes.

Leave a Comment

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)