Add experimental Kotlin incremental compilation #1316
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erikkerber
force-pushed
the
ek/incremental
branch
from
0e33330 to
9fe2ada
Compare
restingbull
reviewed
May 29, 2025
| errStream: java.io.PrintStream, | ||
| vararg args: String, | ||
| ): ExitCode { | ||
| System.setProperty("zip.handler.uses.crc.instead.of.timestamp", "true") |
There was a problem hiding this comment.
Do you need to set this, or is this pulled from the K2JVM impl?
There was a problem hiding this comment.
Pulled from BazelK2JVMCompiler. It hits the same internals as the existing implementation on the Kotlin side, and figured if it was still needed there, it may be needed here.
restingbull
reviewed
May 29, 2025
| // The incremental directory is roughly analogous to a Gradle transform cache directory. Must be unique per compilation target. | ||
| val incrementalDirectory = Paths.get("").toAbsolutePath().parent.resolve("_kotlin_incremental/$label/$incrementalId").toFile() | ||
|
|
||
| val kotlinService = CompilationService.loadImplementation(this.javaClass.classLoader!!) |
There was a problem hiding this comment.
Is it possible/safe to keep the CompilationService implementation around?
There was a problem hiding this comment.
Do you mean simply not loading it every compile task? I'm sure that's safe, yeah.
erikkerber
force-pushed
the
ek/incremental
branch
6 times, most recently
from
b05d750 to
bbd1f67
Compare
Replace commented out snapshot generation code Add new jar dependencies needed for incremental compilation Temporarily force persistent working directory for incremental compilation
erikkerber
force-pushed
the
ek/incremental
branch
from
bbd1f67 to
eec887d
Compare
Invalidate snapshot when underlying (abi) jar changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
This is presented as a POC, but nowhere near merge-ready. In its current state, it may compile your apps and you may see significant build time improvements.
This adds an experimental implementation of incremental compilation via the new "Build Tools API" (BTAPI). The BTAPI itself can replace the direct usage of
K2JVMCompiler, but is also required to leverage incremental compilation.Initial Design
In short:
BazelK2JVMCompilernamedBuildToolsApiCompiler. This invokes the BTAPI.<output base>/execroot/_kotlin_incremental/)build_tools_apito enable building with the new BTAPI (only)incremental_compilationto then add incremental compilationThe rest is fairly trivial usage of the BTAPI, a little of which requires some tinkering and tribal knowledge to know how exactly it is supposed to be used. I did my best to document my knowledge, with brevity.
Performance
The main driver of performance gains are incremental builds that only build a subset of Kotlin files in a target. The incremental data passed in to subsequent builds gives an additional level of work avoidance beyond the action-based caching in Bazel.
I hope to get more comprehensive numbers, but our main test case was a sample incremental build that was ~4x faster in Gradle due to only needing to recompile very small slices of a chain of Kotlin modules. With incremental, Bazel matches what we see with Gradle.
4x isn't a fixed number, it could be greater or smaller depending on how many Kotlin files are in how many dependent targets, and how many files can skip compilation based on the affected classpaths in a build.
TODO
CompilationTask. It works, but not idea.