Yeah now I know the meaning of that cryptic exception, the executor got killed because it exceeds the container memory threshold. There are couple of reasons that could happen but the first culprit is to check your job (e.g. repartition) or try adding more nodes/executors to your cluster.
The second option is best, it keeps everything asynchronous. but… you can do one better and abstract the solution into a reusable pattern: implicit class RichListenableFuture[T](lf: ListenableFuture[T]) { def asScala: Future[T] = { val p = Promise[T]() Futures.addCallback(lf, new FutureCallback[T] { def onFailure(t: Throwable): Unit = p failure t def onSuccess(result: T): Unit = p … Read more
In Scala 2.8: Welcome to Scala version 2.8.0.r20327-b20091230020149 (Java HotSpot(TM) Client VM, Java 1.6. Type in expressions to have them evaluated. Type :help for more information. scala> val seq: Seq[(String,Object)] = (“a”,”A”)::(“b”,”B”)::Nil seq: Seq[(String, java.lang.Object)] = List((a,A), (b,B)) scala> val map = Map(seq: _*) map: scala.collection.immutable.Map[String,java.lang.Object] = Map(a -> A, b -> B) scala> val … Read more
Java makes an assumption that you want to automatically convert the Array args to varargs, but this can be problematic with methods that accept varargs of type Object. Scala requires that you be explicit, by ascribing the argument with : _*. scala> def bar(args:String*) = println(“count=”+args.length) bar: (args: String*)Unit scala> def foo(args:String*) = bar(args: _*) … Read more
case x creates a variable named x, which would match everything and since a variable with the same name already exists you shadow it by using the same name. case `x` uses the value of the variable x which was declared before and would only match inputs with same values. PS You can leave the … Read more
As you go explicitly from sequential to parallel collection via .par, you go back to sequential via .seq. Since sets and maps have parallel implementations, toMap and toSet calls leave the collection in the current domain. The example of reduce works because it, well, reduces the collection (the outer ParSeq disappears, leaving you with the … Read more
For almost all of these, there are Scala alternatives that cover some but not all of the use cases for these patterns. All of this is IMO, of course, but: Creational Patterns Builder Scala can do this more elegantly with generic types than can Java, but the general idea is the same. In Scala, the … Read more
When you do x.foldLeft(1)(_ * _.toInt), the result type will be inference to an Int, but 9415087488 is too large for an Int to store it. So you need to tell Scala using Long to store it. scala> val x = “Hello” x: java.lang.String = Hello scala> x.foldLeft(1L)(_ * _.toInt) res1: Long = 9415087488 scala> … Read more
The easy way to do this in 1.8 is to add a constructor that takes a dummy argument. ScalaTest (and sbt) won’t discover the class if it doesn’t have a public, no-arg constructor: class MySuite(ignore: String) extends FunSuite { // … } In 2.0 you’ll be able to just write @Ignore on the class: @Ignore … Read more
It is a general rule how 64-bit floating point numbers are compared according to IEEE 754 (not Scala or even Java related, see NaN): double n1 = Double.NaN; double n2 = Double.NaN; System.out.println(n1 == n2); //false The idea is that NaN is a marker value for unknown or indeterminate. Comparing two unknown values should always … Read more