With usePipelining enabled, earlyOutputPing was completed only as a side
effect of zinc reporting progress mid-compile (CompileProgress.afterEarlyOutput
via writeEarlyOut / notifyNoEarlyOut in zinc-core). On an action-cache hit zinc
never runs, and on a failed compile it never reports, so the promise stayed
unfulfilled and every task waiting on it - compileEarly and makePickleProducts
across downstream pipelined projects - parked forever. A warm compile;compile
on a multi-module build hung indefinitely (#9486).
The compile task now completes the ping on every resolution path. Zinc's own
completion still wins when zinc ran (tryComplete is atomic, so it can never
pre-empt it); a cache hit completes it from the pickle jar's presence on disk
(absent jar means downstream falls back to a full compile, which is the correct
degradation); a failed compile completes it false so waiters take the full
compile path and propagate the failure instead of hanging.
Generated-by: kimi-code/k3 (Oh My Pi)
forked run used the project's baseDirectory as the working directory, while non-forked execution inherits sbt's own working directory — so toggling fork silently changed how relative paths resolved.
forked run (and forked console) now inherit sbt's working directory, consistent with non-forked execution and `sbtn` expectations.
Running reboot in the sbt shell dropped to the OS shell instead of rebooting.
The break was in teardown: Server.shutdown opened with
log.info, and during a client-initiated reboot the terminal in scope is that
client's already-closed virtual terminal, so the log write throws
ClosedChannelException through the terminal proxy. That aborted teardown
before the portfile was deleted and the server socket closed, and the
exception was swallowed by the shutdown hook (whose own error print goes to
the same dead terminal).
Server.shutdown now completes its state cleanup (portfile, tokenfile, running
flag, server socket) before logging, and CommandExchange.shutdown wraps each
channel shutdown and the server shutdown individually so one failing step
cannot skip the rest.
Fixes#9095
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
putBlobsIfNeeded reads each blob's hash and size once, up front, and
returns only plain HashedVirtualFileRef values, so serializing an ActionResult
(disk, in-memory, or remote store) performs no file I/O and nothing re-stats a
blob after its CAS entry is written: a file vanishing once stored no longer
prevents the write, and I/O errors on an output file surface upfront at blob
storage time rather than mid-serialization.
Fixes#9349
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
The cached-task macro allocated one mutable slot per syntactic
Def.declareOutput / Def.declareOutputDirectory call site and snapshotted the
slots into the task's outputs after the body ran. A call inside a loop or .map
over a runtime-determined list is a single syntactic site executed many times,
so each iteration overwrote the same slot and only the last file was cached and
restored on a cache hit. There was also no way for a conditional call site that
did not execute to stay out of the outputs: its slot remained null.
Declared outputs now accumulate in a per-task ListBuffer: the macro emits one
buffer at the top of the cached body and rewrites each call site to
ActionCache.registerOutput(vf, buffer), which appends and returns the value.
Every execution registers, an unexecuted site contributes nothing, and the
static multi-site shape is unchanged.
Refs #9462 (the declareOutput-in-a-loop half)
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: eugene yokota <[email protected]>
A directory declared via Def.declareOutputDirectory is packaged as a sibling
<dir>.sbtdir.zip, so deleting the directory leaves the zip behind. On a cache
hit, syncFile's up-to-date short-circuit saw the zip in sync (same digest,
already a CAS symlink) and returned without the unpack side effect, which only
ran from the file-write path: the directory was never restored. For sbt's own
compile, whose classes directory is declared this way, rm -rf of the classes
directory with a warm cache meant run failed with ClassNotFoundException and no
recompile; only deleting the zip as well (or the whole cache) recovered.
The up-to-date branch now re-extracts when the extracted directory itself is
missing: a single stat on the warm path, per review preference over a
manifest-based per-file check. Partial deletions inside a still-existing
directory are not repaired, consistent with treating target/ contents as
sbt-managed.
Refs #9462 (the directory-restoration half; the declareOutput-in-a-loop half is
a separate macro-layer issue)
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: eugene yokota <[email protected]>
The thin client (sbtn) parsed the launcher value flags (-java-home, -mem,
-jvm-debug, -sbt-dir, ...) but then dropped them. The space form was consumed
and discarded; the flag=value form fell through to the residual arguments and
was forwarded to the server verbatim, where --java-home=/path was rejected as a
command (`Not a valid command: --`).
When the client had to start a server (none running, no --server), the consumed
-java-home never reached the forked sbt launcher, so the server came up under
the default JVM and, in CI where the intended JDK is only reachable via
-java-home, failed to connect.
parseArgs now captures the consumed launcher value flags (both `flag value` and
`flag=value`) into Arguments.launcherValueArgs, and the cold-start fork re-passes
them to the sbt launcher so the server runs under the requested JVM. The client
tokenizes arguments by splitting on whitespace, which would otherwise fragment a
value that contains spaces (a Windows path like C:\Program Files\Java); parseArgs
tracks those split boundaries and rejoins a value flag's value. An empty flag=
value and a dangling flag with no value are consumed but not propagated, since
the launcher's require_arg would otherwise fail the fork.
The fork command construction is extracted into a pure, package-visible
serverCommand so a test can assert the propagated flag reaches the started
server. The sbt-launch-jar path is unchanged: it invokes java directly, with no
launcher to interpret the flag.
Fixes#9418
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Failure caching assumes a CompileFailed is a function of
the sources, which is true for source errors. But zinc also surfaces I/O write
failures ("error writing X.class") as compiler problems, so an environmental
failure (a concurrent target/ deletion, a permission blip) was cached under the
same mechanism and replayed from the global action cache on every later build,
even after the cause was gone. When the poisoned task is the metabuild compile
this is self-sustaining and unrecoverable from inside sbt: project loading
fails, so no task -- including clean -- can run, and only deleting the global
cache by hand recovers. The replayed diagnostics also name files/permissions
that no longer exist.
Refs #9455
Co-authored-by: BrianHotopp <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
A task that floods stdout could not be stopped promptly: cancelling it
(Ctrl+C from the thin client) stopped the task, but the server kept
draining the already-queued output to the client for several seconds,
because onCancellationRequest never discarded the pending backlog.
onCancellationRequest now sets a per-channel isCanceled flag and clears the
queued frames and buffered stdout (discardPending), and jsonRpcNotify
drops the cancelled task's systemOut/systemErr while the flag is set, so
output stops promptly.
Co-authored-by: Claude Fable 5 <[email protected]>
NetworkChannel batches writes to the client at most once per 20ms to
reduce terminal flicker and byte volume. forceFlush(), used to order
buffered stdout ahead of a control-plane message, called
flushExecutor.shutdownNow() -- the only place the executor was ever
shut down. Once it ran during active output it tore the executor
down (and, if a coalesced flush was pending, cancelled its future
without resetting flushFuture, leaving the slot stuck), so the 20ms
coalescing was gone for the rest of the connection: stdout was then
flushed per write via the inline fallback instead of batched. Output
still reaches the client, so the effect is extra flushes rather than
lost output, but shutting the shared executor down on the first
forceFlush is clearly unintended.
Extract the flush state machine into CoalescingFlusher. forceFlush now
drains immediately and leaves the timer live (a pending coalesced
drain harmlessly drains the remainder when it fires); the executor is
shut down once, at channel teardown, so it no longer leaks. doFlush
now holds a lock across both the drain and the publish so an inline
forceFlush and the timer's drain can't deliver two stdout batches out
of order.
Fixes#9415
Co-authored-by: BrianHotopp <[email protected]>
Co-authored-by: Claude Fable 5 <[email protected]>
When a file referenced by a task's inputs/outputs (e.g. `Compile / resources +=
file("nope.txt")`) does not exist, hashing the task's cache key threw a
NoSuchFileException deep inside sjsonnew serialization. It surfaced as an opaque
sjsonnew.SerializationException that dumped the entire input list, with the real
cause buried several `Caused by:` levels down, so users routinely mistook it for
a corrupt cache and reached for `clean`.
ActionCache.mkInput now catches the hashing failure, detects a NoSuchFileException
anywhere in the cause chain (ActionCache.findMissingFile), and throws a
MessageOnlyException naming the file:
[error] file referenced by the build does not exist: nope.txt
util-cache gains a dependency on util-control (a leaf module, no cycle) for
MessageOnlyException.
Fixes#9217.
Co-authored-by: Brian Hotopp <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
`update` held a process-global lock around coursier resolution and artifact
fetching whenever a logger was set OR coursier was not in fallback mode. That
lock exists only to serialize coursier's interactive progress bar, which is
rendered solely when no custom logger is supplied and coursier is not in
fallback mode. The `loggerOpt.nonEmpty` clause therefore over-serialized the
common non-interactive case (IntelliJ re-imports, CI, any non-TTY run, where
sbt supplies a quiet debug-only logger), making `update` scale with the number
of modules rather than the number of distinct artifacts.
Narrow the predicate to `!hasCustomLogger && !fallbackMode`
(Lock.progressBarActive), so resolution runs in parallel unless coursier is
actually drawing its progress bar. Interactive terminals stay serialized;
COURSIER_PROGRESS=false opts into parallelism there, and #5627 tracks fully
parallel resolution with live progress bars.
Safe: SbtCoursierCache is ConcurrentHashMap-backed, a CacheLogger is already
invoked concurrently within a single resolution (parallel downloads), and the
COURSIER_PROGRESS=false parallel path has been used in production for years.
Fixes#5508.
Co-authored-by: Brian Hotopp <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
* [2.x] feat: Add cacheVersion setting for global cache invalidation
**Problem**
There was no escape hatch to invalidate all task caches when needed.
**Solution**
Add `Global / cacheVersion` setting that incorporates into the cache key
hash. Changing it invalidates all caches. Defaults to reading system
property `sbt.cacheversion`, or else 0L. When 0L, the hash is identical
to the previous behavior (backward compatible).
Fixes#8992
* [2.x] refactor: Simplify BuildWideCacheConfiguration and add cacheVersion test
- Replace auxiliary constructors with default parameter values
- Add unit test verifying cacheVersion invalidates the cache
* [2.x] fix: Restore auxiliary constructors for binary compatibility
* [2.x] test: Improve cacheVersion scripted test and add release note
- Scripted test now verifies cache invalidation via a counter
that increments only when the task body actually executes
- Add release note documenting the cacheVersion setting
For the details about this PR, please see the blog post https://eed3si9n.com/sbt-remote-cache/.
* Add cache basics
* Refactor Attributed to use StringAttributeMap, which is Map[StringAttributeKey, String]
* Implement disk cache
* Rename Package to Pkg
* Virtualize packageBin
* Use HashedVirtualFileRef for packageBin
* Virtualize compile task
Normally scripted tests are forked using the JVM that is running sbt.
If set `scripted / javaHome`, forked using it.
```
scripted / javaHome := Some(file("/path/to/jdk-x.y.z"))
```
Or use `java++` command before scripted.
```
sbt> java++ 11!
sbt> scripted
```
Add `testReportsDirectory` setting to allow output directory for
JUnitXmlTestsListener to be configured.
Add `testReportSettings` which provides defaults values:
- by default this uses the build configuration name as a prefix so
`target/test-reports` for `Test` config, but `target/it-reports`
for `IntegrationTest` (previously this was hardcoded to always
use `target/test-reports`). To override this set e.g.
`Test / testReportsDirectory := target.value / "my-custom-dir"`
- the `JunitXmlTestsListener` is now only attached to the `Test`
and `IntegrationTest` configs by default (previously it was added
to the global configuration object). Any configs which inherit
from one of these will continue to have the listener attached;
but completely custom configurations will need to re-add with:
`project.settings(testReportSettings)`
Fixes#2853
Fixes https://github.com/sbt/sbt/issues/5047
When setting swoval.tmpdir via globalBase, changed to set globalBase as absolute path.
`com.swoval.runtime.NativeLoader.loadPackaged` uses `java.lang.System.load`.
It requires absolute path, so we should set `swoval.tmpdir` with absolute path.
There's also a special case for aliases that will try to resolve
the target of the alias to a task key if possible and display the
output of that key if found.
see https://github.com/sbt/sbt/issues/2881
Fixes https://github.com/sbt/sbt/issues/1502
This adds `--addPluginSbtFile=<file>` command, which adds the given .sbt file to the plugin build.
Using this mechanism editors or IDEs can start a build with required plugin.
```
$ cat /tmp/extra.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.7")
$ sbt --addPluginSbtFile=/tmp/extra.sbt
...
sbt:helloworld> plugins
In file:/xxxx/hellotest/
...
sbtassembly.AssemblyPlugin: enabled in root
```
BrianHotopp
Jozef Koval
Dream
Josh Soref
xuwei-k
Krzysztof Pado
Rui Gonçalves
kijuky
kxbmap
Ashley Mercer
Yusuke Yamada
Josh Soref
Yusuke Izawa
Nepomuk Seiler
Kazuhiro Sera
Jason Pickens
Guillaume Poirier
alodavi
Liu Fengyun
Alistair Johnson
Dale Wijnand
Maksym Fedorov
exoego
Jonas Fonseca