This is a fundamental feature of object-oriented programming that doesn’t get as much attention as it deserves. Suppose you have a collection C[+T]. What the +T means is that if U <: T, then C[U] <: C[T]. Fair enough. But what does it mean to be a subclass? It means that every method should work … Read more
The problem is when using sbt:assembly the default merge strategy excludes all the reference.conf files as per If multiple files share the same relative path (e.g. a resource named application.conf in multiple dependency JARs), the default strategy is to verify that all candidates have the same contents and error out otherwise. The solution is to … Read more
All of the option parameters are passed in the option() function as below: val segments = sqlContext.read.format(“com.databricks.spark.csv”) .option(“delimiter”, “\t”) .load(“s3n://michaeldiscenza/data/test_segments”)
Add the following code into the log4j.properties file inside the src/test/resources dir, create the file/dir if not exist # Change this to set Spark log level log4j.logger.org.apache.spark=WARN # Silence akka remoting log4j.logger.Remoting=WARN # Ignore messages below warning level from Jetty, because it’s a bit verbose log4j.logger.org.eclipse.jetty=WARN When I run my unit tests (I’m using JUnit … Read more
Maybe this will help def matchContainer[A: Manifest](c: Container[A]) = c match { case c: Container[String] if manifest <:< manifest[String] => println(c.value.toUpperCase) case c: Container[Double] if manifest <:< manifest[Double] => println(math.sqrt(c.value)) case c: Container[_] => println(“other”) } Edit: As Impredicative pointed out, Manifest is deprecated. Instead you could do the following: import reflect.runtime.universe._ def matchContainer[A: TypeTag](c: … Read more
On Scala Collections there is usually :+ and +:. Both add an element to the collection. :+ appends +: prepends. A good reminder is, : is where the Collection goes. There is as well colA ++: colB to concat collections, where the : side collection determines the resulting type. If a :++ exists, it is … Read more
Maybe something like this: somethingReturningAnOpt match { case Some(actualThingIWant) => doSomethingWith(actualThingIWant) case None => } or as pst suggests: somethingReturningAnOpt.foreach { actualThingIWant => doSomethingWith(actualThingIWant) } // or… for (actualThingIWant <- somethingReturningAnOpt) { doSomethingWith(actualThingIWant) }
This code works for Spark 2.1, where .codec is not available. df.write .format(“com.databricks.spark.csv”) .option(“codec”, “org.apache.hadoop.io.compress.GzipCodec”) .save(my_directory) For Spark 2.2, you can use the df.write.csv(…,codec=”gzip”) option described here: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=codec
I’ll first explain what type Unit is, just in case. Even if you know already, other people having the same kind of problem might well not know it. Type Unit is similar to what is known in C or Java as void. In those languages, that means “this does not return anything”. However, every method … Read more
If there were such an option, one might have seen it in use here: https://github.com/scala/scala/pull/3669 But the tool of choice seems to be grep: scalac -Xlog-implicits test/files/neg/t2031.scala 2>&1 | grep diverg Since that area is under active development, now might be the time to submit a request for enhancement.