Skip to content

Commit

Permalink
[gradle] exclude scala-reflect (#13894)
Browse files Browse the repository at this point in the history
CHANGELOG: Fixes #13837 in which Hail could break a Spark installation
if the Hail JAR appears on the classpath before the Scala JARs.

We and several dependencies of ours are exposing an old version of
scala-reflect (rude of us and them). If our JAR appears on the classpath
before the scala JARs and the sub-versions of Scala differ, Spark is
likely to have incompatible bytecode.

See #13837 .

In this PR, this command,

```
make shadowJar && \
    jar tf build/libs/hail-all-spark.jar | grep scala/reflect | wc -l && \
    jar tf build/libs/hail-all-spark.jar | grep 'MutableSettings'
```

prints:

```
    0

```

On main (`5d7dc2cab7`) it prints:

```
    1417
scala/reflect/internal/settings/MutableSettings$.class
scala/reflect/internal/settings/MutableSettings$SettingValue.class
scala/reflect/internal/settings/MutableSettings.class
```

---

A bit more details follow for the curious.

`./gradlew dependencies` shows these packages as depending on
`scala-reflect`:

```
+--- org.scalanlp:breeze-natives_2.12:1.1
|    +--- org.scala-lang:scala-library:2.12.10 -> 2.12.17
|    +--- org.scalanlp:breeze_2.12:1.1 -> 1.2
|    |    +--- org.scala-lang:scala-library:2.12.10 -> 2.12.17
|    |    +--- org.scalanlp:breeze-macros_2.12:1.2
|    |    |    +--- org.scala-lang:scala-library:2.12.10 -> 2.12.17
|    |    |    \--- org.scala-lang:scala-reflect:2.12.10 -> 2.12.15
...
+--- org.elasticsearch:elasticsearch-spark-30_2.12:8.4.3
|    +--- org.scala-lang:scala-library:2.12.8 -> 2.12.17
|    +--- org.scala-lang:scala-reflect:2.12.8 -> 2.12.15 (*)
|    \--- org.apache.spark:spark-core_2.12:3.2.1 -> 3.3.0
...
|         +--- org.scala-lang:scala-reflect:2.12.15 (*)
...
+--- org.scala-lang:scala-reflect:2.12.15 (*)
...
+--- org.apache.spark:spark-sql_2.12:3.3.0
...
|    +--- org.apache.spark:spark-catalyst_2.12:3.3.0
|    |    +--- org.scala-lang:scala-reflect:2.12.15 (*)
```

This root (the fourth to last package above):

```
+--- org.scala-lang:scala-reflect:2.12.15 (*)
```

is coming from this line in build.gradle:

```gradle
    shadow 'org.scala-lang:scala-reflect:' + scalaVersion
```

I think `shadow` means "needed to compile but do not include in shadow
JAR", so I do not know if we can trust `./gradlew dependencies` to tell
us why any particular class file is in our shadow JAR.
  • Loading branch information
danking authored Oct 25, 2023
1 parent eabff3d commit 0402aad
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions hail/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ configurations {
hailJar.extendsFrom implementation
hailJar {
exclude group: 'org.scala-lang', module: 'scala-library'
exclude group: 'org.scala-lang', module: 'scala-reflect'
exclude group: 'org.apache.spark'
}

Expand Down

0 comments on commit 0402aad

Please sign in to comment.