The projects of a build mostly depend on the same libraries, and each project's
report materializes its own copies of every coordinate it names, so one value
exists once per project that mentions it: on a 302-module monorepo the cached
reports hold 949,492 ModuleReports for 2,036 distinct values.
UpdateReportInterner adds weak pools for ConfigRef, InclExclRule, File,
Artifact, ModuleID, Caller and ModuleReport itself. Pooling the report is worth
more than pooling its parts alone, because sharing it also shares its licenses
vector, extraAttributes map, homepage string and artifacts vector. Reports
carrying a publicationDate are canonicalized but never pooled, since that
java.util.Calendar is mutable; everything else reachable from a ModuleReport is
immutable, so sharing is semantically invisible.
The pools are weak, so a value lives exactly as long as some report references
it and nothing accumulates across a reload.
Two sites cover a freshly resolved report. SbtUpdateReport interns each report
as it builds it rather than sweeping the finished one, so only the module under
construction is ever un-interned. That alone would not survive, though:
coursier memoizes moduleReport on a key that includes the dependees, so each
project builds its own instance of a shared coordinate, and transformDetails
then rebuilt every report to drop the callers -- discarding the sharing and
leaving a copy per module per configuration. Dropping the callers is what makes
those reports value-equal in the first place, so transformDetails now re-interns
what it rebuilds, and only rebuilds when there is something to drop. A scripted
test pins it: two projects resolving one coordinate must end up holding one
ModuleReport instance, on the fresh path and from the cache.
Co-Authored-By: Claude Opus 5 <[email protected]>
WeakInterner is a ConcurrentHashMap keyed by a WeakReference subclass that
hashes and compares by its referent, with dead entries expunged on each
operation. Ported from zinc's sbt.internal.inc.WeakInterner, so lm-core gains
no interning dependency.
internWith derives the instance to pool only when the value is not pooled yet.
Equality is structural, so a value finds its pooled twin whatever the identity
of the values inside it, which lets a caller whose canonical form is itself
expensive to build skip building it on a hit. intern is internWith with
identity.
Co-Authored-By: Claude Opus 5 <[email protected]>
sjsonnew serializes a File as a (uri, Long) pair whose Long is a SHA-256 of the
file's contents, and the read direction discards it. A report names each
artifact once per configuration it resolved in, and the projects of a build
largely share their dependencies, so writing the caches re-reads the whole
downloaded classpath many times over: on a 302-module monorepo, 755 GB of jars
and about 11 minutes of CPU for bytes no reader looks at.
Staleness comes from LibraryManagement.fileUptodate instead, which checks
File.exists and the modification time against UpdateReport.stamps.
UpdateReportPersistence.CacheCodec extends the LibraryManagementCodec trait and overrides
the inherited fileStringLongIso so the pair carries 0. That member is virtual,
so the override also reaches the Vector[(Artifact, File)] nested inside the
generated ModuleReportFormat, which a locally-scoped JsonFormat[File] could
not. The inputs store keeps the stock codec, so Tracked.inputChanged still
hashes contents for invalidation.
The JSON shape is unchanged, so caches stay readable in both directions.
Co-authored-by: Claude Opus 5 <[email protected]>
sbt.bat fails to start in client/server mode on Windows, falling back to running the full JVM in the foreground.
sbtn uses CreateProcess to spawn the server process. The original %%20 path encoding doesn't work in batch delayed expansion context, and the default install path C:\Program Files (x86) contains spaces that CreateProcess splits at.
**Problem**
There's a race condition between per-byte readSystemIn notification
and one-byte-read thread lifecycle.
Note: One-byte-read thread was introduced as a solution to the problem
that switching the terminal between raw and canonical mode cannot happen
if it's blocked by read.
**Solution**
This eliminates the thread lifecycle issue by keeping the thread alive
throughout the lifecycle of sbtn itself.
read is still called on demand by the server readSystemIn notification.
**Problem**
subproject build.sbt like a/build.sbt leaks to siblings.
**Solution**
Don't forward freshly computed common settings unless
the build.sbt is at root.
forked run used the project's baseDirectory as the working directory, while non-forked execution inherits sbt's own working directory — so toggling fork silently changed how relative paths resolved.
forked run (and forked console) now inherit sbt's working directory, consistent with non-forked execution and `sbtn` expectations.
Running reboot in the sbt shell dropped to the OS shell instead of rebooting.
The break was in teardown: Server.shutdown opened with
log.info, and during a client-initiated reboot the terminal in scope is that
client's already-closed virtual terminal, so the log write throws
ClosedChannelException through the terminal proxy. That aborted teardown
before the portfile was deleted and the server socket closed, and the
exception was swallowed by the shutdown hook (whose own error print goes to
the same dead terminal).
Server.shutdown now completes its state cleanup (portfile, tokenfile, running
flag, server socket) before logging, and CommandExchange.shutdown wraps each
channel shutdown and the server shutdown individually so one failing step
cannot skip the rest.
Fixes#9095
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
**Problem**
Checksums are still generated for asc file.
1. localStaging is a file repo, which was not handled
2. It was checking Artifact name, not the file name
**Solution**
This fixes both.
putBlobsIfNeeded reads each blob's hash and size once, up front, and
returns only plain HashedVirtualFileRef values, so serializing an ActionResult
(disk, in-memory, or remote store) performs no file I/O and nothing re-stats a
blob after its CAS entry is written: a file vanishing once stored no longer
prevents the write, and I/O errors on an output file surface upfront at blob
storage time rather than mid-serialization.
Fixes#9349
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
The cached-task macro allocated one mutable slot per syntactic
Def.declareOutput / Def.declareOutputDirectory call site and snapshotted the
slots into the task's outputs after the body ran. A call inside a loop or .map
over a runtime-determined list is a single syntactic site executed many times,
so each iteration overwrote the same slot and only the last file was cached and
restored on a cache hit. There was also no way for a conditional call site that
did not execute to stay out of the outputs: its slot remained null.
Declared outputs now accumulate in a per-task ListBuffer: the macro emits one
buffer at the top of the cached body and rewrites each call site to
ActionCache.registerOutput(vf, buffer), which appends and returns the value.
Every execution registers, an unexecuted site contributes nothing, and the
static multi-site shape is unchanged.
Refs #9462 (the declareOutput-in-a-loop half)
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: eugene yokota <[email protected]>
A directory declared via Def.declareOutputDirectory is packaged as a sibling
<dir>.sbtdir.zip, so deleting the directory leaves the zip behind. On a cache
hit, syncFile's up-to-date short-circuit saw the zip in sync (same digest,
already a CAS symlink) and returned without the unpack side effect, which only
ran from the file-write path: the directory was never restored. For sbt's own
compile, whose classes directory is declared this way, rm -rf of the classes
directory with a warm cache meant run failed with ClassNotFoundException and no
recompile; only deleting the zip as well (or the whole cache) recovered.
The up-to-date branch now re-extracts when the extracted directory itself is
missing: a single stat on the warm path, per review preference over a
manifest-based per-file check. Partial deletions inside a still-existing
directory are not repaired, consistent with treating target/ contents as
sbt-managed.
Refs #9462 (the directory-restoration half; the declareOutput-in-a-loop half is
a separate macro-layer issue)
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: eugene yokota <[email protected]>
Three independent, pre-existing retention bugs kept the classloader of a
finished in-process test run -- and the open jar handles it holds -- alive
for the rest of the server session.
- JUnitXmlTestsListener: testSuite is an InheritableThreadLocal, so
threads spawned during a run (e.g. async-framework pool workers) inherit
a copy of the suite reference that remove() cannot reach.
- TestRecap.collect: the recap is stashed on State.attributes and outlives
the command, so it must not retain live throwables.
- ClassLoaderCache: loaders evicted from delegate by clearExpiredLoaders
are unreachable from the map and never enqueued on the ReferenceQueue, so
neither clear()/close() nor the cleanup thread could ever close them.
SuiteResult now documents the retention hazard on throwables. Adds
deterministic, cross-platform tests for all three severed chains.
A genuinely broken restore now degrades to the onsite task instead of a
silent cache hit with incomplete outputs. Related to #9349.
Co-authored-by: sshevchenko <[email protected]>
Failure caching assumes a CompileFailed is a function of
the sources, which is true for source errors. But zinc also surfaces I/O write
failures ("error writing X.class") as compiler problems, so an environmental
failure (a concurrent target/ deletion, a permission blip) was cached under the
same mechanism and replayed from the global action cache on every later build,
even after the cause was gone. When the poisoned task is the metabuild compile
this is self-sustaining and unrecoverable from inside sbt: project loading
fails, so no task -- including clean -- can run, and only deleting the global
cache by hand recovers. The replayed diagnostics also name files/permissions
that no longer exist.
Refs #9455
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
**Problem**
sbt 2.x currently constructs the test classloader using symlinks
to the JARs in CAS (content-addressable storage).
This causes issues because JVM apparently caches opened JAR files by path,
and since the symlink itself doesn't change it could end up
serving stale resource files.
**Solution**
We can workaround this issue by resolving the symlinks to the real path.
The ivyless-publish-http and ivyless-publish-http-plugin scripted tests
defined publishToHttp as three unordered `.value` dependencies
(startPublishServer, publish, stopPublishServer). sbt evaluates task
dependencies before the task body in an unspecified order, so publish
could run before the HTTP server was listening and fail intermittently
with "Connection refused". It is reproducible under CPU contention (a
2-vCPU runner), which is why it began flaking on the JDK-25 shard and
turned develop red.
Use Def.sequential to guarantee start -> publish -> stop ordering, the
way the sibling ivyless-publish-maven-http test already sequences via
separate scripted commands.
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
The thin client (sbtn) parsed the launcher value flags (-java-home, -mem,
-jvm-debug, -sbt-dir, ...) but then dropped them. The space form was consumed
and discarded; the flag=value form fell through to the residual arguments and
was forwarded to the server verbatim, where --java-home=/path was rejected as a
command (`Not a valid command: --`).
When the client had to start a server (none running, no --server), the consumed
-java-home never reached the forked sbt launcher, so the server came up under
the default JVM and, in CI where the intended JDK is only reachable via
-java-home, failed to connect.
parseArgs now captures the consumed launcher value flags (both `flag value` and
`flag=value`) into Arguments.launcherValueArgs, and the cold-start fork re-passes
them to the sbt launcher so the server runs under the requested JVM. The client
tokenizes arguments by splitting on whitespace, which would otherwise fragment a
value that contains spaces (a Windows path like C:\Program Files\Java); parseArgs
tracks those split boundaries and rejoins a value flag's value. An empty flag=
value and a dangling flag with no value are consumed but not propagated, since
the launcher's require_arg would otherwise fail the fork.
The fork command construction is extracted into a pure, package-visible
serverCommand so a test can assert the propagated flag reaches the started
server. The sbt-launch-jar path is unchanged: it invokes java directly, with no
launcher to interpret the flag.
Fixes#9418
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Problem:
When useIvy := false, ivyless publishing dropped the optional
scala_[scalaVersion]/ and sbt_[sbtVersion]/ path segments from Ivy-style
plugin publish patterns. This caused sbt 1 plugins published from sbt 2 builds
to land under non-plugin Ivy paths, so consumers looking under
scala_2.12/sbt_1.0/ could not resolve them.
Solution:
Read the plugin cross-version attributes from the CsrProject module and use
them when constructing ivyless Ivy-layout publish paths. Apply the same
substitution for local/file Ivy publishing and remote Ivy-style URL publishing,
while continuing to omit those optional segments when the attributes are absent.
Add scripted regressions for local and HTTP Ivy-style publishing of an sbt 1
plugin with useIvy := false.
Mai Huy HoàngandClaude Opus 5
Mai Huy HoàngandClaude Opus 5
Fabrizio Colonna
eugene yokota
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
kenji yoshida
Jozef Koval
BrianHotoppandClaude Opus 4.8
Kevin Lee
Stas Shevchenkoandsshevchenko
Anatolii Kmetiuk