airflow

An on_failure_callback can be supplied to the DAG and/or individual tasks. In the first case (supplying to the DAG), there is no ‘exception’ in the context (the argument Airflow calls your on_failure_callback with). In the second case (supplying to a task), there is. The contained object should be a python Exception. It’s surprisingly non-intuitive to … Read more

Operator that does literally nothing. It can be used to group tasks in a DAG. https://airflow.apache.org/_api/airflow/operators/dummy_operator/index.html As far as I know, at least two use case: test purpose: in DAGs, the dummy operation is just between upstream and downstream, later, you can replace the true operator. Workflow purpose: BranchPythonOperator works with DummyOperator. If you want … Read more

Here are some solutions: 1. Override BashOperator to add some values to the context class NextExecutionDateAwareBashOperator(BashOperator): def render_template(self, attr, content, context): dag = context[‘dag’] execution_date = context[‘execution_date’] context[‘next_execution_date’] = dag.following_schedule(execution_date) return super().render_templates(attr, content, context) # or in python 2: # return super(NextExecutionDateAwareBashOperator, self).render_templates(attr, content, context) The good part with this approach: you can capture some … Read more

Operator that does literally nothing. It can be used to group tasks in a DAG. https://airflow.apache.org/_api/airflow/operators/dummy_operator/index.html as far as I know, at least to two case: test purpose. in dags, the dummy operation just between upstream and downstream, later, you can replace the true operator. Workflow purpose: BranchPythonOperator work with DummyOperator. If you want to … Read more

Like @tobi6 said, you can view the output from your DAG runs in your webserver or in your console depending on the environment. To do so in your webserver: Select the DAG you just ran and enter into the Graph View. Select the task in that DAG that you want to view the output of. … Read more

To run any DAGs, you need to make sure two processes are running: airflow webserver airflow scheduler If you only have airflow webserver running, the UI will show DAGs as running, but if you click on the DAG, none of it’s tasks are actually running or scheduled, but rather in a Null state. What this … Read more

My dag is being loaded but I had the name of the DAG wrong. I was expecting the dag to be named by the file but the name is determined by the first argument to the DAG constructor dag = DAG( ‘tutorial’, default_args=default_args, schedule_interval=timedelta(1))