From neeraj’s hint, it seems like the correct way to do this in pyspark is: expr = “Arizona.*hot” dk = dx.filter(dx[“keyword”].rlike(expr)) Note that dx.filter($”keyword” …) did not work since (my version) of pyspark didn’t seem to support the $ nomenclature out of the box.
The simplest way I can think of is to use collect_list import pyspark.sql.functions as f df.groupby(“col1”).agg(f.concat_ws(“, “, f.collect_list(df.col2)))
You need to cast the column low to class date and then you can use datediff() in combination with lit(). Using Spark 2.2: from pyspark.sql.functions import datediff, to_date, lit df.withColumn(“test”, datediff(to_date(lit(“2017-05-02”)), to_date(“low”,”yyyy/MM/dd”))).show() +———-+—-+——+—–+ | low|high|normal| test| +———-+—-+——+—–+ |1986/10/15| z| null|11157| |1986/10/15| z| null|11157| |1986/10/15| c| null|11157| |1986/10/15|null| null|11157| |1986/10/16|null| 4.0|11156| +———-+—-+——+—–+ Using < Spark 2.2, … Read more
2021-01-19 Updated There are many approaches here (setting ENV vars, adding to $SPARK_HOME/conf/spark-defaults.conf, etc…) other answers already cover these. I wanted to add an answer for those specifically wanting to do this from within a Python Script or Jupyter Notebook. When you create the Spark session you can add a .config() that pulls in the … Read more
Backticks seem to work just fine: scala> val df = sc.parallelize(Seq((“a”, 1))).toDF(“foo bar”, “x”) df: org.apache.spark.sql.DataFrame = [foo bar: string, x: int] scala> df.registerTempTable(“df”) scala> sqlContext.sql(“””SELECT `foo bar` FROM df”””).show foo bar a Same as DataFrame API: scala> df.select($”foo bar”).show foo bar a So it looks like it is supported, although I doubt it is … Read more
You can use also use a function from sql package. It will generate a unique id, however it will not be sequential as it depends on the number of partitions. I believe it is available in Spark 1.5 + from pyspark.sql.functions import monotonicallyIncreasingId # This will return a new DF with all the columns + … Read more
A little late but I found this while I was searching and it may help someone else… You might also try unpacking the argument list to spark.read.parquet() paths=[‘foo’,’bar’] df=spark.read.parquet(*paths) This is convenient if you want to pass a few blobs into the path argument: basePath=”s3://bucket/” paths=[‘s3://bucket/partition_value1=*/partition_value2=2017-04-*’, ‘s3://bucket/partition_value1=*/partition_value2=2017-05-*’ ] df=spark.read.option(“basePath”,basePath).parquet(*paths) This is cool cause you don’t … Read more
In practice performance impact will be almost the same as if you omitted partitionBy clause at all. All records will be shuffled to a single partition, sorted locally and iterated sequentially one by one. The difference is only in the number of partitions created in total. Let’s illustrate that with an example using simple dataset … Read more
I wouldn’t use Spark in the first place, but if you are really committed to the particular stack, you can combine a bunch of ml transformers to get best matches. You’ll need Tokenizer (or split): import org.apache.spark.ml.feature.RegexTokenizer val tokenizer = new RegexTokenizer().setPattern(“”).setInputCol(“text”).setMinTokenLength(1).setOutputCol(“tokens”) NGram (for example 3-gram) import org.apache.spark.ml.feature.NGram val ngram = new NGram().setN(3).setInputCol(“tokens”).setOutputCol(“ngrams”) Vectorizer (for … Read more
Yes it is possible although can be far from trivial. Typically you want a Java (friendly) wrapper so you don’t have to deal with Scala features which cannot be easily expressed using plain Java and as a result don’t play well with Py4J gateway. Assuming your class is int the package com.example and have Python … Read more