Warnings while building Scala/Spark project with SBT

Is there a better way to structure build.sbt (add other resolvers e.g.?), so that I can get rid off the warnings?

One way is to manually tell sbt what dependencies you prefer, for your case:

dependencyOverrides ++= Set(
  "io.netty" % "netty" % "3.9.9.Final",
  "commons-net" % "commons-net" % "2.2",
  "com.google.guava" % "guava" % "11.0.2"
)

I also recommend reading about conflict management in sbt.

Should I care about warnings at all?

In your case – no, since your conflicts stem from using only spark-related artifacts released under same version. Spark is a project with big userbase and possibility of jar hell introduced due to transitive dependencies is rather low (although, technically not guaranteed).

In general case – maybe. Typically it is ok in most of the cases, but there is a small possibility of a problem that may need careful manual dependency resolution (if this is possible at all). In these cases it is really hard to tell whether there is a problem before you run your app and bump into some problem like missing class, method, mismatching method signature or some reflection-related problem.

Leave a Comment

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)