Use coalesce before write operation dataFrame.coalesce(1).write.format(“parquet”).mode(“append”).save(“temp.parquet”) EDIT-1 Upon a closer look, the docs do warn about coalesce However, if you’re doing a drastic coalesce, e.g. to numPartitions = 1, this may result in your computation taking place on fewer nodes than you like (e.g. one node in the case of numPartitions = 1) Therefore as … Read more
First of all, new List() won’t work, since the List class is abstract. The other two options are defined as follows in the List object: override def empty[A]: List[A] = Nil override def apply[A](xs: A*): List[A] = xs.toList I.e., they’re essentially equivalent, so it’s mostly a matter of style. I prefer to use empty because … Read more
isInstanceOf looks if there is a corresponding entry in the inheritance chain. The chain of A with T includes A, B and T, so a.isInstanceOf[B] must be true. edit: Actually the generated byte code calls javas instanceof, so it would be a instanceof B in java. A little more complex call like a.isInstanceOf[A with T] … Read more
Yes, you should probably start off with pattern matching instead of the visitor pattern. See this interview with Martin Odersky (my emphasis): So the right tool for the job really depends on which direction you want to extend. If you want to extend with new data, you pick the classical object-oriented approach with virtual methods. … Read more
Set.empty is that set; although you can’t get at it directly, it turns out that it is just a private object in the Set companion object (called, obviously enough, EmptySet). All that Set.empty does is return that set with a cast to the correct type. It is done this way, instead of with Nil, because … Read more
Edit 2017-09-18: As of Scala 2.12, there is a transform method that takes a Try[T] => Try[S]. So you can write val future = … // Future[T] val mapped = future.transform { case Success(_) => Success(“OK”) case Failure(_) => Success(“KO”) } For 2.11.x, the below still applies: AFAIK, you can’t do this directly with a … Read more
Try invalidating Intellij IDEA cache via File -> Invalidate Caches / Restart It takes like 5 minutes but did the trick for me.
Portability: Linux and Windows at least I hope. What about mobile phones, did anyone succeed in getting it to run there? Yes. There is quite some movement about Scala on Android. As for J2ME, I saw something in that respect, but not much. There is some code pertaining to J2ME on the source code repository. … Read more
You want unzip scala> List((1,”a”), (3, “b”), (4, “d”)).unzip res1: (List[Int], List[String]) = (List(1, 3, 4),List(a, b, d)) Similarly there is an unzip3 for List[Tuple3[A, B, C]], though anything of higher arity you’d have to implement yourself. scala> List((1,”a”, true), (3, “b”, false), (4, “d”, true)).unzip3 res2: (List[Int], List[String], List[Boolean]) = (List(1, 3, 4),List(a, b, … Read more
You’re on the right track using JavaConversions, but the method you need for this particular conversion is seqAsJavaList: java.util.List<String> convert(scala.collection.Seq<String> seq) { return scala.collection.JavaConversions.seqAsJavaList(seq); } Update: JavaConversions is deprecated, but the same function can be found in JavaConverters. java.util.List<String> convert(scala.collection.Seq<String> seq) { return scala.collection.JavaConverters.seqAsJavaList(seq); }