You need to perform initialization after installation: $ export AIRFLOW_HOME=some/dir $ airflow db init # or `airflow initdb` for the legacy 1.X If AIRFLOW_HOME is unset, ~/airflow/ will be created and used. This is where the config and logs will be stored; if you want to reset the configuration, remove the dir stored in AIRFLOW_HOME … Read more
An on_failure_callback can be supplied to the DAG and/or individual tasks. In the first case (supplying to the DAG), there is no ‘exception’ in the context (the argument Airflow calls your on_failure_callback with). In the second case (supplying to a task), there is. The contained object should be a python Exception. It’s surprisingly non-intuitive to … Read more
Operator that does literally nothing. It can be used to group tasks in a DAG. https://airflow.apache.org/_api/airflow/operators/dummy_operator/index.html As far as I know, at least two use case: test purpose: in DAGs, the dummy operation is just between upstream and downstream, later, you can replace the true operator. Workflow purpose: BranchPythonOperator works with DummyOperator. If you want … Read more
Here are some solutions: 1. Override BashOperator to add some values to the context class NextExecutionDateAwareBashOperator(BashOperator): def render_template(self, attr, content, context): dag = context[‘dag’] execution_date = context[‘execution_date’] context[‘next_execution_date’] = dag.following_schedule(execution_date) return super().render_templates(attr, content, context) # or in python 2: # return super(NextExecutionDateAwareBashOperator, self).render_templates(attr, content, context) The good part with this approach: you can capture some … Read more
Adding the sys path again worked for me, import sys sys.path.insert(0,os.path.abspath(os.path.dirname(__file__)))
Operator that does literally nothing. It can be used to group tasks in a DAG. https://airflow.apache.org/_api/airflow/operators/dummy_operator/index.html as far as I know, at least to two case: test purpose. in dags, the dummy operation just between upstream and downstream, later, you can replace the true operator. Workflow purpose: BranchPythonOperator work with DummyOperator. If you want to … Read more
Since the existing answer here wasn’t totally clear, and this is the top hit for “airflow landing time” I went to the chat archives and found the original answer being referenced here: Maxime Beauchemin @mistercrunch Jun 09 2016 11:12 it’s the number of hours after the time the scheduling period ended take a schedule_interval=”@daily” run … Read more
Like @tobi6 said, you can view the output from your DAG runs in your webserver or in your console depending on the environment. To do so in your webserver: Select the DAG you just ran and enter into the Graph View. Select the task in that DAG that you want to view the output of. … Read more
The Airflow REST API Plugin would help you out here. Once you have followed the instructions for installing the plugin you would just need to hit the following url: http://{HOST}:{PORT}/admin/rest_api/api/v1.0/trigger_dag?dag_id={dag_id}&run_id={run_id}&conf={url_encoded_json_parameters}, replacing dag_id with the id of your dag, either omitting run_id or specify a unique id, and passing a url encoded json for conf (with … Read more