Shading dependencies with sbt-assembly (in particular, shapeless in Spark 2.1.0)

A few weeks ago I needed to parse configuration files in Scala for a Spark project and decided to use PureConfig. It is incredibly lean to use, needing minimal boilerplate. I recommend you check it out (give also a look at CaseClassy, which I haven’t had time to test yet).

Everything seemed straightforward enough, and I got it working pretty quickly (as in, it compiled properly). The surprise? spark-submit failed with a conflict with Shapeless (lacking a witness). This is due to Spark 2.1.0 needing Breeze, which in turn needs Shapeless 2.0.0 (which is pretty old). Problem is, Spark’s required library prevented PureConfig from pulling the correct version. D’oh!

There is an easy fix, though, if you are using sbt-assembly you can shade dependencies, by adding something like the following to your assembly.sbt file:

assemblyShadeRules in assembly := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll)

• Apache Spark
• scala