A better way to test the value of an Option?
How about if (opt == Some(“lakes”)) This expresses the intent clearly and is straight forward.
scala
When would I use the “Mark directory as …” option in IntelliJ IDEA?
Lets go through the options: Sources Root – this is where your actual project code goes – typically in src/main/scala (though you can change that, or add extra source directories in your build). Directories nested below this level should be packages. Intellij needs to know these are sources so it can highlight them, check them … Read more
How to access values in array column?
Since Spark 2.4.0, there is a new function element_at($array_column, $index). See Spark docs
How do I access scala documentation from the repl?
Scaladocs are generated as HTML, so you don’t want them appearing in the REPL window. You might want to load docs in a browser from the REPL, however. You can do that by creating your own method like so (this one takes an instance; you could have it take an instance of Class[A] instead, if … Read more
Forward References – why does this code compile?
The body of a class or an object is the primary constructor. A constructor, like a method, is a sequence of statements that are executed in order — to do anything else, it would have to be a very different language. I’m pretty sure you wouldn’t like for Scala to execute the statements of your … Read more
Spark / Scala: forward fill with last observation
Initial answer (a single time series assumption): First of all try avoid window functions if you cannot provide PARTITION BY clause. It moves data to a single partition so most of the time it is simply not feasible. What you can do is to fill gaps on RDD using mapPartitionsWithIndex. Since you didn’t provide an … Read more
What does dollar sign do in scala
That isn’t special Scala syntax, it’s a method name. In Scala $ is a legal identifier. The method is inherited from the org.apache.spark.ml.param.Param trait. See the source.
Spark final task takes 100x times longer than first 199, how to improve
Spark >= 3.0 Since 3.0 Spark provides built-in optimizations for handling skewed joins – which can be enabled using spark.sql.adaptive.optimizeSkewedJoin.enabled property. See SPARK-29544 for details. Spark < 3.0 You clearly have a problem with a huge right data skew. Lets take a look a the statistics you’ve provided: df1 = [mean=4.989209978967438, stddev=2255.654165352454, count=2400088] df2 = … Read more
What reflection capabilities can we expect from Scala 2.10?
update 2012-07-04: Daniel SOBRAL (also on SO) details in his blog post “JSON serialization with reflection in Scala! Part 1 – So you want to do reflection?” some of the features coming with reflection: To recapitulate, Scala 2.10 will come with a Scala reflection library. That library is used by the compiler itself, but divided … Read more
Integrating native system libraries with SBT
From the research I’ve done in the past, there are only two ways to get native libraries loaded: modifying java.library.path and using System.loadLibrary (I feel like most people do this), or using System.load with an absolute path. As you’ve alluded to, messing with java.library.path can be annoying in terms of configuring SBT and Eclipse, and … Read more