parallelism: not a very descriptive name. The description says it sets the maximum task instances for the airflow installation, which is a bit ambiguous — if I have two hosts running airflow workers, I’d have airflow installed on two hosts, so that should be two installations, but based on context ‘per installation’ here means ‘per … Read more
Here is how we manage it for our team. First in terms of naming convention, each of our DAG file name matches the DAG Id from the content of the DAG itself (including the DAG version). This is useful because ultimately it’s the DAG Id that you see in the Airflow UI so you will … Read more
In your airflow.cfg, you’ve these two configurations to control this behavior: # after how much time a new DAGs should be picked up from the filesystem min_file_process_interval = 0 dag_dir_list_interval = 60 You might have to reload the web-server, scheduler and workers for your new configuration to take effect.
In the UI: Go to the dag, and dag run of the run you want to change Click on GraphView Click on task A Click “Clear” This will let task A run again, and if it succeeds, task C should run. This works because when you clear a task’s status, the scheduler will treat it … Read more
I advice running airflow in a robust way, with auto-recovery with systemd so you can do: – to start systemctl start airflow – to stop systemctl stop airflow – to restart systemctl restart airflow For this you’ll need a systemd ‘unit’ file. As a (working) example you can use the following: put it in /lib/systemd/system/airflow.service … Read more
You can import the logging module into your code and write to logs that way import logging logging.info(‘Hello’) Here are some more options import logging logging.debug(‘This is a debug message’) logging.info(‘This is an info message’) logging.warning(‘This is a warning message’) logging.error(‘This is an error message’) logging.critical(‘This is a critical message’)
EDIT: This answer stores your secret key in plain text which can be a security risk and is not recommended. The best way is to put access key and secret key in the login/password fields, as mentioned in other answers below. END EDIT It’s hard to find references, but after digging a bit I was … Read more
Try the following: export AIRFLOW_GPL_UNIDECODE=yes OR export SLUGIFY_USES_TEXT_UNIDECODE=yes Using export makes the environment variable available to all the subprocesses. Also, make sure you are using pip install apache-airflow[postgres] and not pip install airflow[postgres] Which should you use: if using AIRFLOW_GPL_UNIDECODE, airflow will install a dependency that is under GPL license, which means you won’t be … Read more
In the DAGs screen you can see the running tasks: Example On ‘Recent Tasks’ press the running icon and Airflow will automatically run the search query with the filters for the Dag Id and State equal to ‘running’ and show the results on the Task Instances screen (you can find it manually on the tab … Read more