a: A is type ascription; see What is the purpose of type ascriptions in Scala? : _* is a special instance of type ascription which tells the compiler to treat a single argument of a sequence type as a variable argument sequence, i.e. varargs. It is completely valid to create a Queue using Queue.apply that … Read more
Robert’s answer covers the techinal details of Context Bounds. I’ll give you my interpretation of their meaning. In Scala a View Bound (A <% B) captures the concept of ‘can be seen as’ (whereas an upper bound <: captures the concept of ‘is a’). A context bound (A : C) says ‘has a’ about a … Read more
Try explicitly specify sc.textFile(“file:///path to the file/”). The error occurs when Hadoop environment is set. SparkContext.textFile internally calls org.apache.hadoop.mapred.FileInputFormat.getSplits, which in turn uses org.apache.hadoop.fs.getDefaultUri if schema is absent. This method reads “fs.defaultFS” parameter of Hadoop conf. If you set HADOOP_CONF_DIR environment variable, the parameter is usually set as “hdfs://…”; otherwise “file://”.
When inferring types, the compiler often needs to calculate the Least Upper Bound (LUB) of a list of types. For example, the type of if (cond) e1 else e1 is the LUB of the types of e1 and e1. These types can get quite large, for example try this in a REPL: :type Map(1 -> … Read more
ListBuffer is a mutable list which has constant-time append, and constant-time conversion into a List. List is immutable and has constant-time prepend and linear-time append. How you construct your list depends on the algorithm you’ll use the list for and the order in which you get the elements to create it. For instance, if you … Read more
There is a case where private[this] is required to make code compile. This has to do with an interaction of variance notation and mutable variables. Consider the following (useless) class: class Holder[+T] (initialValue: Option[T]) { // without [this] it will not compile private[this] var value = initialValue def getValue = value def makeEmpty { value … Read more
the Enumeration trait has a type member Value representing the individual elements of the enumeration (it’s actually an inner class, but the difference doesn’t matter here). Thus object WeekDay inherits that type member. The line type WeekDay = Value is just a type alias. It is useful, because after you import it elsewhere with import … Read more
scala> val arr = Array(“Hello”,”World”) arr: Array[java.lang.String] = Array(Hello, World)
The TLDR: .config(“spark.master”, “local”) a list of the options for spark.master in spark 2.2.1 I ended up on this page after trying to run a simple Spark SQL java program in local mode. To do this, I found that I could set spark.master using: SparkSession spark = SparkSession .builder() .appName(“Java Spark SQL basic example”) .config(“spark.master”, … Read more
The trick is to first make sure that none of the futures has failed. .recover is your friend here, you can combine it with map to convert all the Future[T] results to Future[Try[T]]] instances, all of which are certain to be successful futures. note: You can use Option or Either as well here, but Try … Read more