You can also sort the column by importing the spark sql functions import org.apache.spark.sql.functions._ df.orderBy(asc(“col1”)) Or import org.apache.spark.sql.functions._ df.sort(desc(“col1″)) importing sqlContext.implicits._ import sqlContext.implicits._ df.orderBy($”col1″.desc) Or import sqlContext.implicits._ df.sort($”col1”.desc)
It is strange that no one had suggested NIO.2 operations (available since Java 7): import java.nio.file.{Paths, Files} import java.nio.charset.StandardCharsets Files.write(Paths.get(“file.txt”), “file contents”.getBytes(StandardCharsets.UTF_8)) I think this is by far the simplest and easiest and most idiomatic way, and it does not need any dependencies sans Java itself.
Since Scala 2.9 you can use sys.env for the same effect: scala> sys.env(“HOME”) res0: String = /home/paradigmatic I think is nice to use the Scala API instead of Java. There are currently several project to compile Scala to other platforms than JVM (.NET, javascript, native, etc.) Reducing the dependencies on Java API, will make your … Read more
implicitly is avaliable in Scala 2.8 and is defined in Predef as: def implicitly[T](implicit e: T): T = e It is commonly used to check if an implicit value of type T is available and return it if such is the case. Simple example from retronym’s presentation: scala> implicit val a = “test” // define … Read more
You can use c map (t => t.getP -> t) toMap but be aware that this needs 2 traversals.
Edit: Newest newest version Since spark 2.x you should use dataset api instead when using Scala [1]. Check docs here: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Dataset.html#withColumn(colName:String,col:org.apache.spark.sql.Column):org.apache.spark.sql.DataFrame If working with python, even though easier, I leave the link here as it’s a very highly voted question: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.withColumn.html >>> df.withColumn(‘age2’, df.age + 2).collect() [Row(age=2, name=”Alice”, age2=4), Row(age=5, name=”Bob”, age2=7)] [1] https://spark.apache.org/docs/latest/sql-programming-guide.html: In … Read more
From the official documentation: http://www.playframework.com/documentation/2.1.1/SBTDependencies Getting the right Scala version with %% If you use groupID %% artifactID % revision instead of groupID % artifactID % revision (the difference is the double %% after the groupID), SBT will add your project’s Scala version to the artifact name. This is just a shortcut. You could write … Read more
This question is answered in duplicate by Travis here. There are links to the issue in the tracker and to Eugene’s discussion (in the comments and mailing list). In the famous “Skylla and Charybdis” section of the type checker, our hero decides what shall escape dark anonymity and see the light as a member of … Read more
The shortest answer I know of that uses only the standard library is map1 ++ map2.map{ case (k,v) => k -> (v + map1.getOrElse(k,0)) }
Most RDD operations are lazy. Think of an RDD as a description of a series of operations. An RDD is not data. So this line: val textFile = sc.textFile(“/user/emp.txt”) It does nothing. It creates an RDD that says “we will need to load this file”. The file is not loaded at this point. RDD operations … Read more