Reuse the sibling dirzip when zinc reports it wrote nothing. The output
is still declared, so the stored ActionResult stays complete and a later hit
restores the class directory exactly as before -- it is the same
HashedVirtualFileRef the packaging path would have produced, read off disk
rather than rebuilt.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
syncFile treated an output as up to date only when its digest matched and the
path was already a symlink. packageDirectory installs the archive with
Files.move, so on the write path it is always a regular file and the second
conjunct never held. Every compile therefore fell through to IO.delete +
writeFileAndNotify -> afterFileWrite -> unpackageDirZip, which inflated the
archive it had just written into a temp directory, digest-compared every file in
the output tree, found all of them unchanged, and deleted the temp directory.
The cost is proportional to output size and is paid on every compile of every
module. Measured at ~450 ms for a 2557-file, 33 MB classes directory (11 MB
archive), against a 4.2 s warm single-file edit.
A matching digest already means the local content is the blob's, whether or not
the path is a symlink, so take the up-to-date path in that case. Where the path
is not yet a symlink, still relink it to the CAS to keep deduplication, but
reach afterFileUpToDate rather than afterFileWrite. The symlink-creation
fallback is factored into linkOrCopy and shared with writeFileAndNotify.
For non-dirzip outputs both hooks are no-ops, and that branch already did a
delete-and-relink, so their behaviour is unchanged. The behaviour given up is
that a file inside the extracted directory which diverged from the manifest is
no longer silently restored while the archive digest still matches; on a cache
hit with a differing digest the full unpackage runs as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**Problem**
In theory, we shouldn't have to care about the timestamps,
but for local optimization it's still a useful key for caching.
Currently local Analysis cache gets invalidated after
syncBlob replaces the Analysis file with a symlink.
**Solution**
This preserves the original timestamp when copying files to
the local diskcache.
putBlobsIfNeeded reads each blob's hash and size once, up front, and
returns only plain HashedVirtualFileRef values, so serializing an ActionResult
(disk, in-memory, or remote store) performs no file I/O and nothing re-stats a
blob after its CAS entry is written: a file vanishing once stored no longer
prevents the write, and I/O errors on an output file surface upfront at blob
storage time rather than mid-serialization.
Fixes#9349
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cached-task macro allocated one mutable slot per syntactic
Def.declareOutput / Def.declareOutputDirectory call site and snapshotted the
slots into the task's outputs after the body ran. A call inside a loop or .map
over a runtime-determined list is a single syntactic site executed many times,
so each iteration overwrote the same slot and only the last file was cached and
restored on a cache hit. There was also no way for a conditional call site that
did not execute to stay out of the outputs: its slot remained null.
Declared outputs now accumulate in a per-task ListBuffer: the macro emits one
buffer at the top of the cached body and rewrites each call site to
ActionCache.registerOutput(vf, buffer), which appends and returns the value.
Every execution registers, an unexecuted site contributes nothing, and the
static multi-site shape is unchanged.
Refs #9462 (the declareOutput-in-a-loop half)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: eugene yokota <eed3si9n@gmail.com>
A directory declared via Def.declareOutputDirectory is packaged as a sibling
<dir>.sbtdir.zip, so deleting the directory leaves the zip behind. On a cache
hit, syncFile's up-to-date short-circuit saw the zip in sync (same digest,
already a CAS symlink) and returned without the unpack side effect, which only
ran from the file-write path: the directory was never restored. For sbt's own
compile, whose classes directory is declared this way, rm -rf of the classes
directory with a warm cache meant run failed with ClassNotFoundException and no
recompile; only deleting the zip as well (or the whole cache) recovered.
The up-to-date branch now re-extracts when the extracted directory itself is
missing: a single stat on the warm path, per review preference over a
manifest-based per-file check. Partial deletions inside a still-existing
directory are not repaired, consistent with treating target/ contents as
sbt-managed.
Refs #9462 (the directory-restoration half; the declareOutput-in-a-loop half is
a separate macro-layer issue)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: eugene yokota <eed3si9n@gmail.com>
A genuinely broken restore now degrades to the onsite task instead of a
silent cache hit with incomplete outputs. Related to #9349.
Co-authored-by: sshevchenko <sshevchenko@evolution.com>
Failure caching assumes a CompileFailed is a function of
the sources, which is true for source errors. But zinc also surfaces I/O write
failures ("error writing X.class") as compiler problems, so an environmental
failure (a concurrent target/ deletion, a permission blip) was cached under the
same mechanism and replayed from the global action cache on every later build,
even after the cause was gone. When the poisoned task is the metabuild compile
this is self-sustaining and unrecoverable from inside sbt: project loading
fails, so no task -- including clean -- can run, and only deleting the global
cache by hand recovers. The replayed diagnostics also name files/permissions
that no longer exist.
Refs #9455
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
**Problem**
BatchUpdateBlobsRequest suffers from gRPC's message size limitation.
**Solution**
For larger files, we switch to using the ByteStream API, chunked to 1MB at a time.
When a file referenced by a task's inputs/outputs (e.g. Compile / resources +=
file("nope.txt")) does not exist, hashing the task's cache key threw a
NoSuchFileException deep inside sjsonnew serialization. It surfaced as an opaque
sjsonnew.SerializationException that dumped the entire input list, with the real
cause buried several `Caused by:` levels down, so users routinely mistook it for
a corrupt cache and reached for `clean`.
ActionCache.mkInput now catches the hashing failure, detects a NoSuchFileException
anywhere in the cause chain (ActionCache.findMissingFile), and throws a
MessageOnlyException naming the file:
[error] file referenced by the build does not exist: nope.txt
util-cache gains a dependency on util-control (a leaf module, no cycle) for
MessageOnlyException.
Fixes#9217.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
**Problem**
We need some tooling to debug caching issues.
**Solution**
This adds an exeprimental execution log support,
which shows input and output of cached tasks.
Bumps io to released 1.12.0 which contains the atomic-write changes
to IO.copyFile, IO.transfer(InputStream, File), and IO.jar/zip.
ActionCacheStore.putBlob now uses IO.transfer instead of duplicating
staging logic.
**Problem**
Intermittent java.nio.file.FileAlreadyExistsException when publishing
classes.sbtdir.zip during action-cache packaging under parallel tasks
(#9043). Copying from a temp directory straight into the final path races
on the fixed destination name.
**Solution**
Stage the built zip next to the destination with a unique temp file, then
replace the final path via Files.move with REPLACE_EXISTING and ATOMIC_MOVE,
falling back to a non-atomic move when needed. Add a concurrent
packageDirectory test.
Closes#9043
Generated-by: Cursor (AI-assisted)
**Problem**
When a file referenced during cache serialization is deleted between task
execution and cache storage, sbt crashes with an uncaught NoSuchFileException.
The IOException catch added in #8699 only wraps store.put(), but
Converter.toJsonUnsafe(result) and mkInput() also call
hashedVirtualFileRefToStr which calls Files.readAttributes — and these are
outside the try-catch.
**Solution**
Move the try/catch IOException to wrap the entire post-action cache storage
section, so NoSuchFileException from result serialization or cache key
computation is caught and handled gracefully (skip caching, return result).
Fixes#9044
Co-authored-by: bittoby <218712309+bittoby@users.noreply.github.co>
* [2.x] feat: Add cacheVersion setting for global cache invalidation
**Problem**
There was no escape hatch to invalidate all task caches when needed.
**Solution**
Add `Global / cacheVersion` setting that incorporates into the cache key
hash. Changing it invalidates all caches. Defaults to reading system
property `sbt.cacheversion`, or else 0L. When 0L, the hash is identical
to the previous behavior (backward compatible).
Fixes#8992
* [2.x] refactor: Simplify BuildWideCacheConfiguration and add cacheVersion test
- Replace auxiliary constructors with default parameter values
- Add unit test verifying cacheVersion invalidates the cache
* [2.x] fix: Restore auxiliary constructors for binary compatibility
* [2.x] test: Improve cacheVersion scripted test and add release note
- Scripted test now verifies cache invalidation via a counter
that increments only when the task body actually executes
- Add release note documenting the cacheVersion setting
**Problem**
When the remote cache server (e.g. bazel-remote using S3 for storage) reports an AC (Action Cache)
hit but the underlying CAS (Content Addressable Storage) blob is missing or
corrupt, ActionCache.cache propagates the resulting exception (typically
java.io.FileNotFoundException) directly to the SBT task engine process with no interception of the propogated error.
This causes a build failure instead of a graceful cache miss.
The three unguarded call sites are:
1. organicTask - syncBlobs after a successful put only caught NoSuchFileException,
missing FileNotFoundException and other IO errors.
2. getWithFailure / readFromSymlink fast-path - syncBlobs inside flatMap with no
exception handling.
3. getWithFailure main branch - both syncBlobs calls and the subsequent IO.read
were completely unguarded.
**Solution**
Guard all three call sites with NonFatal catches:
- Cache read failures (getWithFailure) return Left(None) which the caller
interprets as a cache miss, triggering organic recomputation.
- Cache write failures (organicTask) are demoted to a debug-level log; the task
result that was already computed is returned successfully.
Two regression tests are added to ActionCacheTest:
1. Tests the main getWithFailure branch using the default relative-path converter.
2. Tests the readFromSymlink fast-path using an absolute-path converter so the
symlink created on the first run is found by Files.isSymbolicLink on the second.
Catch parse errors when reading cached values and return Left(None)
so the task is re-run instead of failing with IncompleteParseException.
Fixes plugins/dotty-sandwich scripted test flake.
Fixes#8627
- Use ${OUT} placeholder and remove StringVirtualFile1 special-case handling
- Make dirZipExt check generic (use vf instead of svf: StringVirtualFile1)
- Let syncBlobs handle StringVirtualFile1 transparently via fileConverter.toPath()
Fixes#8631
**Changes:**
- Add `useIvy` setting key (defaults to `true`)
- Add `ivylessPublishLocalImpl` helper that publishes without Ivy
- Modify `publishLocal` to use ivyless publisher when `useIvy := false`
- Generate ivy.xml via `lmcoursier.IvyXml`
- Generate MD5/SHA-1 checksums for all files
- Add scripted test `dependency-management/ivyless-publish-local`
* test: Migrate FileInfoSpec.scala to verify.BasicTestSuite (#8542)
Migrate FileInfoSpec.scala from ScalaTest's AnyFlatSpec to
verify.BasicTestSuite, following the pattern established by other
test files in the util-cache module.
Changes:
- Replace AnyFlatSpec class with BasicTestSuite object
- Convert 'it should ... in' syntax to 'test(...)' syntax
- Use Scala 3 syntax with colon indentation
- Change === to == for assertions (BasicTestSuite style)
- Add explicit Unit return types for consistency
- Add 'end FileInfoSpec' marker
Fixes#8542
Migrate SingletonCacheSpec.scala from ScalaTest's AnyFlatSpec to
verify.BasicTestSuite, following the pattern established by other
test files in the util-cache module.
Changes:
- Replace AnyFlatSpec class with BasicTestSuite object
- Convert 'should ... in' syntax to 'test(...)' syntax
- Use Scala 3 syntax with colon indentation
- Change === to == for assertions (BasicTestSuite style)
- Replace intercept[Exception] with scala.util.Try pattern
- Add 'end SingletonCacheSpec' and 'end ComplexType' markers
Related to the ongoing test migration effort.
Co-authored-by: GlobalStar117 <GlobalStar117@users.noreply.github.com>
* always create symlinks to the cache in the target locations, even if the digest matches #8445
* create a test (currently failing even on #develop) that fails because if `zipPath` in `sbt.util.ActionCache.packageDirectory` is a symlink to the CAS, in later calls, this path in the CAS gets overridden by the new sources.
- in this test, after "run 1" in line 15, the produced file "target/out/jvm/scala-3.7.4/a/classes.sbtdir.zip" is a symlink to the CAS, let's call it SH1.
- when "run 3" is executed, `IO.zip` saves the new value to `zipPath`, which is "target/out/jvm/scala-3.7.4/a/classes.sbtdir.zip -> SH1", so SH1 gets overridden.
- when the last "run 1" is executed, the cache retrieves SH1, but it contains the data from "run 3" (the test fails with "actual A.x is 3").
* when packaging a directory into a zip, use a temp directory to avoid overwriting the cache #8461
When a compilation fails with CompileFailed, the failure is now cached
so that subsequent builds with the same inputs don't re-run the failed
compilation. This significantly improves the experience when using BSP
clients like Metals that may trigger many compilations in a row.
The implementation:
- Adds CachedCompileFailure, CachedProblem, and CachedPosition types
to serialize compilation failures
- Modifies ActionCache.cache to catch CompileFailed exceptions and
store them in the cache with exitCode=1
- On cache lookup, checks for cached failures first and re-throws
the cached exception if found
- Fixes DiskActionCacheStore.put to preserve exitCode from request
- Adds unit test to verify cached failure behavior
Fixes#7662
- Ensure output files are synced even when reading from symlink
- Call findActionResult to get ActionResult for side-effect files
- Maintains performance benefit while ensuring correctness
- Addresses @eed3si9n's concern about tasks generating files on the side
- Replace try-catch with Exception.nonFatalCatch.opt for cleaner code
- Follows Scala best practices for non-fatal exception handling
- More functional and idiomatic approach
- Avoids catching fatal exceptions like VirtualMachineError
- Use config.fileConverter.toPath() instead of string manipulation
- Avoid hardcoded '/' prefix removal
- More robust and maintainable approach
- Fix IO.read() to include UTF-8 charset parameter
- Check for symlinked value files before reading AC JSON
- When symlink exists and is valid, read directly from it
- Fallback to original AC file reading if symlink read fails
- Improves performance by avoiding unnecessary AC file reads
- All existing tests pass
**Problem**
sha256 is currently a bottleneck for no-op compilation.
**Solution**
This adds a local, in-memory cache of sha256 hashes of binary files
using their timestamp and file size.
The size of the digest cache can be configured using localDigestCacheByteSize,
which is set to 1MB by default.
Mai Huy Hoàng
eugene yokota
kenji yoshida
BrianHotopp
Stas Shevchenko
Anatolii Kmetiuk
bitloi
BitToby
Dream
Idan Ben-Zvi
bitloi
calm
NeedmeFordev
dive2tech
E.G
azdrojowa123
MkDev11
john0030710
Matt Dziuban