You get SyntaxError error exception because Python has no && operator. It has and and & where the latter one is the correct choice to create boolean expressions on Column (| for a logical disjunction and ~ for logical negation). Condition you created is also invalid because it doesn’t consider operator precedence. & in Python … Read more
For pyspark you can do the following: from functools import reduce from pyspark.sql import DataFrame dfs = [df1,df2,df3] df = reduce(DataFrame.unionAll, dfs) It’s also worth nothing that the order of the columns in the dataframes should be the same for this to work. This can silently give unexpected results if you don’t have the correct … Read more
Use wildcard, e.g. replace 2008 with *: df = sqlContext.read .format(“com.databricks.spark.csv”) .option(“header”, “true”) .load(“../Downloads/*.csv”) // <– note the star (*) Spark 2.0 // these lines are equivalent in Spark 2.0 spark.read.format(“csv”).option(“header”, “true”).load(“../Downloads/*.csv”) spark.read.option(“header”, “true”).csv(“../Downloads/*.csv”) Notes: Replace format(“com.databricks.spark.csv”) by using format(“csv”) or csv method instead. com.databricks.spark.csv format has been integrated to 2.0. Use spark not sqlContext
There are a few efficient ways to implement this. Let’s start with required imports: from pyspark.sql.functions import col, expr, when You can use Hive IF function inside expr: new_column_1 = expr( “””IF(fruit1 IS NULL OR fruit2 IS NULL, 3, IF(fruit1 = fruit2, 1, 0))””” ) or when + otherwise: new_column_2 = when( col(“fruit1”).isNull() | col(“fruit2”).isNull(), … Read more
Spark >= 3.4 In Spark 3.4 or later you can use built-in melt method (sdf .melt( ids=[‘A’], values=[‘B’, ‘C’], variableColumnName=”variable”, valueColumnName=”value”) .show()) +—+——–+—–+ | A|variable|value| +—+——–+—–+ | a| B| 1| | a| C| 2| | b| B| 3| | b| C| 4| | c| B| 5| | c| C| 6| +—+——–+—–+ This method is available … Read more
Your question is broad, thus my answer will also be broad. To get the data types of your DataFrame columns, you can use dtypes i.e : >>> df.dtypes [(‘age’, ‘int’), (‘name’, ‘string’)] This means your column age is of type int and name is of type string.
Spark >= 2.4 You can replace flatten udf with built-in flatten function import org.apache.spark.sql.functions.flatten leaving the rest as-is. Spark >= 2.0, < 2.4 It is possible but quite expensive. Using data you’ve provided: case class Record( visitorId: String, trackingIds: Array[String], emailIds: Array[String]) val df = Seq( Record(“a158”, Array(“666b”), Array(“12”)), Record(“7g21”, Array(“c0b5”), Array(“45”)), Record(“7g21”, Array(“c0b4”), Array(“87”)), … Read more
If all columns you want to pass to UDF have the same data type you can use array as input parameter, for example: >>> from pyspark.sql.types import IntegerType >>> from pyspark.sql.functions import udf, array >>> sum_cols = udf(lambda arr: sum(arr), IntegerType()) >>> spark.createDataFrame([(101, 1, 16)], [‘ID’, ‘A’, ‘B’]) \ … .withColumn(‘Result’, sum_cols(array(‘A’, ‘B’))).show() +—+—+—+——+ | … Read more
You can use input_file_name which: Creates a string column for the file name of the current Spark task. from pyspark.sql.functions import input_file_name df.withColumn(“filename”, input_file_name()) Same thing in Scala: import org.apache.spark.sql.functions.input_file_name df.withColumn(“filename”, input_file_name)
I believe you need to use window functions to attain the rank of each row based on user_id and score, and subsequently filter your results to only keep the first two values. from pyspark.sql.window import Window from pyspark.sql.functions import rank, col window = Window.partitionBy(df[‘user_id’]).orderBy(df[‘score’].desc()) df.select(‘*’, rank().over(window).alias(‘rank’)) .filter(col(‘rank’) <= 2) .show() #+——-+———+—–+—-+ #|user_id|object_id|score|rank| #+——-+———+—–+—-+ #| user_1| … Read more