Den Roman’s link (Archived version) seems promising, but it seemed a little bit complicated to me, so I tried another approach. Here is a simple algorithm I used: Let say you want to compute LCA(x,y) with x and y two nodes. Each node must have a value color and count, resp. initialized to white and … Read more
You can first topologically sort the DAG (every DAG can be topologically sorted) in O(n+m). Once this is done, you know that edge go from lower index vertices to higher. This means that there exists a Hamiltonian path if and only if there are edge between consecutive vertices, e.g. (1,2), (2,3), …, (n-1,n). (This is … Read more
Label each node and make an edge list. That is, for each node store the nodes that it has edges to, for example: { “a”: [ “b”, “c”, “d” ], “b”: [ “d” ], “c”: [ “d” ], “d”: [ ] } You can store many kinds of graphs this way, not just DAGs, so … Read more
Run airflow dags list (or airflow list_dags for Airflow 1.x) to check, whether the dag file is located correctly. For some reason, I didn’t see my dag in the browser UI before I executed this. Must be issue with browser cache or something. If that doesn’t work, you should just restart the webserver with airflow … Read more
First, one obtains a list of commits (as with git rev-list), and parents of each commit. A “column reservation list” is kept in memory. For each commit then: If the commit has no column reserved for it, assign it to a free column. This is how the branch heads will start. Print the tree graphics … Read more
You should be able to use BashOperator. Keeping the rest of your code as is, import required class and system packages: from airflow.operators.bash_operator import BashOperator import os import sys set required paths: os.environ[‘SPARK_HOME’] = ‘/path/to/spark/root’ sys.path.append(os.path.join(os.environ[‘SPARK_HOME’], ‘bin’)) and add operator: spark_task = BashOperator( task_id=’spark_java’, bash_command=’spark-submit –class {{ params.class }} {{ params.jar }}’, params={‘class’: ‘MainClassName’, ‘jar’: … Read more
Even i have been looking in the web to learn about how spark computes the DAG from the RDD and subsequently executes the task. At high level, when any action is called on the RDD, Spark creates the DAG and submits it to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. … Read more
I would try to sort the graph topologically, and if you can’t, then it has cycles.
graph = structure consisting of nodes, that are connected to each other with edges directed = the connections between the nodes (edges) have a direction: A -> B is not the same as B -> A acyclic = “non-circular” = moving from node to node by following the edges, you will never encounter the same … Read more