JVM 22+ support #876

sgammon · 2025-01-06T04:39:43Z

Summary

This PR updates dependencies and build process for Pkl to bring GraalVM and Truffle support up to their latest versions; today, Pkl builds against Java 17+, and requires Java 17+ to run.

After merging this PR, Pkl would build against Java 23, test against Java 21, and imply no change to bytecode (i.e. still only requiring Java 17+ to run or transitively compile against Pkl).

Java 21 adds a lot of features, and JPMS enforcement (and related deprecations) are a concern. This PR takes care of those but does not modularize Pkl itself.

Test downstream here, via Github Actions. On that fork PR, the change set is expressed in full, with fixups.

Rationale

JVM 21 support: Needed for modern Java projects to embed Pkl; needed for modern GraalVM native-image binaries built with embedded Pkl.
GVM 23 toolchain: Support for the latest release version of GraalVM and Truffle brings many improvements to the compiler and polyglot implementation
JVM 21-pinned test toolchain: Tests run under JVM 21. It's annoying to have two JDKs here, but JDK 23 is best, and JDK 21 is needed for JVM execution of Truffle implementation tests, due to a dependency on sun.misc.Unsafe.ensureInitialized, which was removed in JDK 22.
Gradle 8.12: Needed for Gradle to be able to understand class targets as high as 65, which corresponds to JVM 23.
Shadow 8.3.5: Necessitated by a plugin ID change (com.github.johnrengelman.shadow → com.gradleup.shadow), to enable support for JVM 21+
Truffle SVM Dependency: Certain superclasses used by Pkl (notably, AbstractTruffleException and TruffleFeature) have moved to the new org.graalvm.nativeimage:truffle-runtime-svm coordinate.
Non-relocations for Truffle/GraalVM classes. After the Truffle Unchained transition, Truffle languages and runtime artifacts must be present on the Java module path. By definition, this precludes use in "relocated" form.

Known Issues

Pre-merge Checklist

Drop GHA workflow scaffolding
Closely review Gradle lockfiles
Review decisions for Uber JAR class inclusion

Changelog

feat: support for jvm21+/gvm jdk 23, latest truffle
feat: flag support for gvm, native arch, use new -Os optimization mode
feat: initial transitive native image flag support
fix: support up to graalvm/jdk 23 (latest)
fix: don't use gu tool for modern graalvm versions
fix: coordinate change for shadow plugin (com.gradleup.shadow)
fix: build with --add-modules=jdk.unsupported where needed
fix: use jdk21 to run the tests (needed for Unsafe.ensureInitialized)
fix: truffle svm dependency is required after graalvm 24.0.0
fix: warnings for gvm flag usage, renamed truffle svm macro
chore: bump graalvm → 24.1.0
chore: bump shadow → 8.3.5
chore: update gradle → 8.12
chore: update lockfiles

sgammon · 2025-01-06T05:05:54Z

buildSrc/src/main/kotlin/BuildInfo.kt

@@ -80,6 +80,10 @@ open class BuildInfo(project: Project) {

  val isReleaseBuild: Boolean by lazy { java.lang.Boolean.getBoolean("releaseBuild") }

+  val isNativeArch: Boolean by lazy { java.lang.Boolean.getBoolean("nativeArch") }


pass gradlew ... -DnativeArch=true to effectively pass -march=native to native-image

sgammon · 2025-01-06T05:06:56Z

buildSrc/src/main/kotlin/BuildInfo.kt

@@ -80,6 +80,10 @@ open class BuildInfo(project: Project) {

  val isReleaseBuild: Boolean by lazy { java.lang.Boolean.getBoolean("releaseBuild") }

+  val isNativeArch: Boolean by lazy { java.lang.Boolean.getBoolean("nativeArch") }
+
+  val isEnableOracleGraalvm: Boolean by lazy { java.lang.Boolean.getBoolean("oracleGraalvm") }


pass gradlew ... -DoracleGraalvm=true to enable SBOM and G1 GC; intended to guard for features which are only supported within Oracle GraalVM and not GraalVM Community Edition

sgammon · 2025-01-06T05:07:28Z

buildSrc/src/main/kotlin/InstallGraalVm.kt

-      execOperations.exec {
-        val executableName = if (os.isWindows) "gu.cmd" else "gu"
-        executable = distroBinDir.resolve(executableName).toString()
-        args("install", "--no-progress", "native-image")
+      val gvmVersionMajor =
+        requireNotNull(graalVm.get().version.split(".").first().toIntOrNull()) {
+          "Invalid GraalVM JDK version: ${graalVm.get().graalVmJdkVersion}"
+        }
+      if (gvmVersionMajor < 24) {
+        execOperations.exec {
+          val executableName = if (os.isWindows) "gu.cmd" else "gu"
+          executable = distroBinDir.resolve(executableName).toString()
+          args("install", "--no-progress", "native-image")
+        }


gu is no longer applicable when using newer versions of GraalVM; in Pkl's case, no action is needed because native-image now ships with GraalVM by default

sgammon · 2025-01-06T05:08:20Z

gradlew

-APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s
-' "$PWD" ) || exit
+APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit


applied after gradlew wrapper ...

sgammon · 2025-01-06T05:08:57Z

pkl-cli/pkl-cli.gradle.kts

-        add("-H:Name=${outputFile.get().asFile.name}")
+        add("-o")
+        add(outputFile.get().asFile.name)


prefers stable options

sgammon · 2025-01-06T05:10:32Z

pkl-cli/pkl-cli.gradle.kts

        add("-H:IncludeResources=org/pkl/core/stdlib/.*\\.pkl")
        add("-H:IncludeResources=org/jline/utils/.*")
        add("-H:IncludeResourceBundles=org.pkl.core.errorMessages")
        add("-H:IncludeResources=org/pkl/commons/cli/PklCARoots.pem")
-        add("--macro:truffle")


(1) renamed version of this is now --macro:truffle-svm
(2) it is now applied automatically for builds with Truffle on the modulepath (see here)

sgammon · 2025-01-06T05:11:37Z

pkl-cli/pkl-cli.gradle.kts

-  val exclusions = listOf(libs.truffleApi, libs.graalSdk).map { it.get().module.name }
+  val exclusions = listOf(libs.graalSdk).map { it.get().module.name }


the SDK is a virtual artifact anyway and is ignored by native-image; truffleApi must be on the modulepath due to the operating mechanism behind Truffle Unchained. otherwise, build errors surface

this is also why non-relocations must be added for truffle types

sgammon · 2025-01-06T05:13:32Z

pkl-cli/pkl-cli.gradle.kts

        if (!buildInfo.isReleaseBuild) {
          add("-Ob")
+        } else {
+          add("-Os")
+        }
+        if (buildInfo.isNativeArch) {
+          add("-march=native")
+        } else {
+          add("-march=compatibility")
+        }
+        if (buildInfo.isEnableOracleGraalvm) {
+          add("--gc=G1")
+          add("--enable-sbom=cyclonedx")
+        } else {
+          add("--gc=serial")
        }
-        add("-march=compatibility")


changes here:

when building a release...

-Os is passed, which optimizes for size (this is a new feature supported by native-image)

under oracle graalvm...

--gc=G1 --enable-sbom=cyclonedx are passed; the SBOM is embedded within the final binary

when building with -DnativeArch=true...

-march=native is passed

otherwise, -march=compatibility is passed (no change)

when building under graalvm CE...

--gc=serial is passed; this is the only feasible option aside from epsilon (no-op)

when not building a release...

-Ob is passed to optimize build time (no change)

sgammon · 2025-01-06T05:18:09Z

pkl-core/src/test/files/LanguageSnippetTests/input/errors/analyzeInvalidHttpModule.inert

this test is broken for unclear reasons. the test output is:

> Task :pkl-core:test errors > analyzeInvalidHttpModule.pkl FAILED Changed content at line 2: expecting: ["HTTP/1.1 header parser received no bytes"] but was: ["parsing HTTP/1.1 status line, receiving [P], parser state [STATUS_LINE]"]

sgammon · 2025-01-06T05:18:31Z

pkl-executor/pkl-executor.gradle.kts

+val testToolchain =
+  javaToolchains.launcherFor {
+    languageVersion = JavaLanguageVersion.of(21)
+    vendor = JvmVendorSpec.GRAAL_VM
+  }
+
+tasks.test {
+  javaLauncher = testToolchain
+  dependsOn(prepareTest)
+  useJUnitPlatform()
+  jvmArgumentProviders.add(CommandLineArgumentProvider { listOf("--add-modules=jdk.unsupported") })
+}


note to self: clean up launcher configuration

bioball · 2025-01-06T17:21:27Z

This is great, thanks, Sam!

I will do a review of this by next week.

FYI: Pkl today supports Java 21. The range enabled here is Java 22+ (updated your PR title).

settings.gradle.kts

sgammon · 2025-01-06T20:15:44Z

@bioball Thanks! Yes, I'll drop the toolchain stuff. #262 seems too out of date to use here; if toolchain issues arise I can help with manual setup or configuration under Gradle (that was added to account for GHA scaffolding; I am not sure if Circle needs additional JDKs without a CI run)

StefMa · 2025-01-06T20:20:52Z

@sgammon feel free to extract the toolchain support into another PR. Indeed, my PR is quited dated. I will close it...

sgammon · 2025-01-06T20:31:18Z

@StefMa roger, no worries :) I am a well known offender in that regard

- feat: support for jvm21+/gvm jdk 23, latest truffle - feat: flag support for gvm, native arch, use new `-Os` optimization mode - feat: initial transitive native image flag support - fix: support up to graalvm/jdk 23 (latest) - fix: don't use `gu` tool for modern graalvm versions - fix: coordinate change for `shadow` plugin (`com.gradleup.shadow`) - fix: build with `--add-modules=jdk.unsupported` where needed - fix: use jdk21 to run the tests (needed for `Unsafe.ensureInitialized`) - fix: truffle svm dependency is required after graalvm `24.0.0` - fix: warnings for gvm flag usage, renamed truffle svm macro - chore: bump graalvm → `24.1.0` - chore: bump shadow → `8.3.5` - chore: update gradle → `8.12` Signed-off-by: Sam Gammon <[email protected]>

Signed-off-by: Sam Gammon <[email protected]>

sgammon · 2025-01-06T22:06:10Z

@bioball I dropped toolchain provisioning, but toolchains are still used to force JVM 21 for testing. I can centralize that code within buildSrc to keep things clean, but as expressed the build will fail if it cannot find both GVM 21 and GVM 23 installations at build time.

This is a tricky one. GVM 23 is needed to use the latest Truffle API, or version checks must be disabled otherwise (since Truffle and GraalVM's versions need to line up). This is the support we need downstream to use Pkl in embedded form.

But, GVM 21 is needed specifically because of deprecations in JDK 22, which are still in use by Truffle; so, JVM executions involving Truffle can't exceed JDK 21 (and this would include test runs).

Thus, at least two toolchains are needed. It is easiest to declare these requirements using Gradle's toolchains feature; however, where those toolchains come from (dynamic provisioning or a static expectation) is another question.

sgammon · 2025-01-06T22:11:14Z

Wait, I guess it would make no difference to consolidate on GVM 21:

The bytecode level is not changing anyway
They do issue a JDK 21 and JDK 23 release at latest GraalVM
That release would still have the right Truffle API version, but would not disrupt things with JDK22+ support

So, maybe it makes sense to update to the latest GVM/Truffle API, but only up to JDK 21. What do you think @bioball? Then there would be only one toolchain which simplifies things a lot.

Ultimately our goal is just to get Pkl working in embedded form at latest Truffle API. JVM build toolchain is immaterial to the final product anyway since bytecode is pinned at JVM 17.

sgammon force-pushed the feat/jvm21plus-upstream branch from 8c2a05f to dc45756 Compare January 6, 2025 04:52

sgammon mentioned this pull request Jan 6, 2025

[WIP] feat: java/graalvm 21+ support elide-dev/pkl#3

Draft

13 tasks

sgammon changed the title ~~feat: jvm 21+ support~~ JVM 21+ support Jan 6, 2025

sgammon mentioned this pull request Jan 6, 2025

Upstream: Build Upgrades #204

Closed

sgammon commented Jan 6, 2025

View reviewed changes

bioball changed the title ~~JVM 21+ support~~ JVM 22+ support Jan 6, 2025

bioball reviewed Jan 6, 2025

View reviewed changes

settings.gradle.kts Outdated Show resolved Hide resolved

StefMa mentioned this pull request Jan 6, 2025

Add Gradle toolchain support #262

Closed

sgammon added 2 commits January 6, 2025 14:00

chore: update lockfiles

1982880

Signed-off-by: Sam Gammon <[email protected]>

sgammon force-pushed the feat/jvm21plus-upstream branch from dc45756 to 1982880 Compare January 6, 2025 22:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JVM 22+ support #876

JVM 22+ support #876

sgammon commented Jan 6, 2025 •

edited

Loading

sgammon Jan 6, 2025

sgammon Jan 6, 2025

sgammon Jan 6, 2025

sgammon Jan 6, 2025

sgammon Jan 6, 2025

sgammon Jan 6, 2025

sgammon Jan 6, 2025

sgammon Jan 6, 2025

sgammon Jan 6, 2025

sgammon Jan 6, 2025

bioball commented Jan 6, 2025

sgammon commented Jan 6, 2025

StefMa commented Jan 6, 2025

sgammon commented Jan 6, 2025

sgammon commented Jan 6, 2025 •

edited

Loading

sgammon commented Jan 6, 2025

		@@ -80,6 +80,10 @@ open class BuildInfo(project: Project) {

		val isReleaseBuild: Boolean by lazy { java.lang.Boolean.getBoolean("releaseBuild") }

		val isNativeArch: Boolean by lazy { java.lang.Boolean.getBoolean("nativeArch") }

		val exclusions = listOf(libs.truffleApi, libs.graalSdk).map { it.get().module.name }
		val exclusions = listOf(libs.graalSdk).map { it.get().module.name }

JVM 22+ support #876

Are you sure you want to change the base?

JVM 22+ support #876

Conversation

sgammon commented Jan 6, 2025 • edited Loading

Summary

Rationale

Known Issues

Pre-merge Checklist

Changelog

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bioball commented Jan 6, 2025

sgammon commented Jan 6, 2025

StefMa commented Jan 6, 2025

sgammon commented Jan 6, 2025

sgammon commented Jan 6, 2025 • edited Loading

sgammon commented Jan 6, 2025

sgammon commented Jan 6, 2025 •

edited

Loading

sgammon commented Jan 6, 2025 •

edited

Loading