apache-spark-sql

Dropping a nested column from Spark DataFrame

by

It is just a programming exercise but you can try something like this: import org.apache.spark.sql.{DataFrame, Column} import org.apache.spark.sql.types.{StructType, StructField} import org.apache.spark.sql.{functions => f} import scala.util.Try case class DFWithDropFrom(df: DataFrame) { def getSourceField(source: String): Try[StructField] = { Try(df.schema.fields.filter(_.name == source).head) } def getType(sourceField: StructField): Try[StructType] = { Try(sourceField.dataType.asInstanceOf[StructType]) } def genOutputCol(names: Array[String], source: String): Column = … Read more

PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

by

Mariusz answer didn’t really help me. So if you like me found this because it’s the only result on google and you’re new to pyspark (and spark in general), here’s what worked for me. In my case I was getting that error because I was trying to execute pyspark code before the pyspark environment had … Read more

The value of “spark.yarn.executor.memoryOverhead” setting?

by

spark.yarn.executor.memoryOverhead Is just the max value .The goal is to calculate OVERHEAD as a percentage of real executor memory, as used by RDDs and DataFrames –executor-memory/spark.executor.memory controls the executor heap size, but JVMs can also use some memory off heap, for example for interned Strings and direct byte buffers. The value of the spark.yarn.executor.memoryOverhead property … Read more

Access element of a vector in a Spark DataFrame (Logistic Regression probability vector) [duplicate]

by

Update: It seems like there is a bug in spark that prevents you from accessing individual elements in a dense vector during a select statement. Normally you should would be able to access them just like you would a numpy array, but when trying to run the code previously posted, you may get the error … Read more

Difference between createOrReplaceTempView and registerTempTable

by

registerTempTable is a part of the 1.x API and has been deprecated in Spark 2.0. createOrReplaceTempView and createTempView have been introduced in Spark 2.0, as a replacement for registerTempTable. Other than that registerTempTable and createOrReplaceTempView functionally equivalent and the former one calls the latter one.

Encoder for Row Type Spark Datasets

by

The answer is to use a RowEncoder and the schema of the dataset using StructType. Below is a working example of a flatmap operation with Datasets: StructType structType = new StructType(); structType = structType.add(“id1”, DataTypes.LongType, false); structType = structType.add(“id2”, DataTypes.LongType, false); ExpressionEncoder<Row> encoder = RowEncoder.apply(structType); Dataset<Row> output = join.flatMap(new FlatMapFunction<Row, Row>() { @Override public Iterator<Row> … Read more

How to create a udf in PySpark which returns an array of strings?

by

You need to initialize a StringType instance: label_udf = udf(my_udf, ArrayType(StringType())) # ^^ df.withColumn(‘subset’, label_udf(df.col1)).show() +————+——+ | col1|subset| +————+——+ | oculunt|[s, n]| |predistposed|[s, n]| | incredulous|[s, n]| +————+——+

pyspark join multiple conditions

by

Quoting from spark docs: (https://spark.apache.org/docs/1.5.2/api/python/pyspark.sql.html?highlight=dataframe%20join#pyspark.sql.DataFrame.join) join(other, on=None, how=None) Joins with another DataFrame, using the given join expression. The following performs a full outer join between df1 and df2. Parameters: other – Right side of the join on – a string for join column name, a list of column names, , a join expression (Column) or … Read more