Why does Spark fail with “Detected cartesian product for INNER join between logical plans”?

Question

You can triggers inner join after turning on the flag

spark.conf.set("spark.sql.crossJoin.enabled", "true")

You also could also use the cross join.

weights.crossJoin(input)

or set the Alias as

weights.join(input, input("sourceId")===weights("sourceId"), "cross")

You can find more about the issue SPARK-6459 which is said to be fixed in 2.1.1

As you have already used 2.1.1 the issue should have been fixed.