compileOrder and usePipelining were missing from CompileInputs2, so two builds differing only in those settings shared one cache entry and replayed each other's results. Adds both to the key, with a unit test on the hash and a scripted test that fails without the change.
LogExchange.unbindLoggerAppenders calls clearAppenders, which drops and closes every appender on the logger. So if you bind an appender to the global logger and then unbind it, sbt's own console and backing appenders go with it and nothing prints for the rest of the session.
LoggerContext now has removeAppender(loggerName, appender), which removes that one instance by identity and closes it. LogExchange gets an unbindLoggerAppenders(name, appenders) overload on top of it, so unbinding mirrors binding. The existing single-arg version behaves as before, it just says in the scaladoc what it does now.
Task-local filters used in sources.all(filter).value currently produce generic Scala macro scope errors. Report one actionable diagnostic at the offending reference, explaining how to move the definition outside the task or use Def.taskDyn.
**Problem**
When a key does not say which project it belongs to, Extracted.runAggregated filled in the global scope instead of the current project.
**Solution**
Fill in the current project instead, the way runInputTask always has. The only difference between the two scopes is the project, so a key that already names a project behaves exactly as before.
**Problem**
If a client cannot reach the server, it deletes the portfile. Then it
starts a second server. The first server is displaced: its socket path
now belongs to the second server.
A displaced server keeps running. A dropIfIdle notification cannot
reach it, because the proc file it registered names the socket that
the second server now owns.
**Solution**
The portfile holds the serverId of whichever server wrote it. Watch
that file, and exit when the id is not this server's. Leave the
portfile to the server that owns it.
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
+cleanFull failed with Not a valid key: Full. The cross command re-parses its argument with the key parser, and clean is an accepting prefix, so the rest was taken as task args. Args now have to be preceded by whitespace.
**Problem**
Currently bad resources are silently dropped.
**Solution**
1. Fix a bad resource generator in the build.
2. Fail the build when bad resources are found.
The overload that takes axisValues and settings passes an empty
scalaVersions into the fold that builds the rows, so a call with
autoScalaLibrary = true returns the matrix unchanged. The assertion
records that, so the fix can move it.
Now the call adds one row from the axes it was given, so a build can name a partial or a full version instead.
* [2.x] feat: testForkedWorker setting
**Problem**
A major bottleneck of the default setting is that concurrentRestriction
allows exactly one forked test in flight at a time,
and partly due to the restriction all test suites from a subproject runs
within the forked process.
**Solution**
This adds a new setting called testForkedWorker, which lets the build users
tune the number of concurrent worker process in flight.
I've set the default to CPU count / 3.
Within a single worker, the default parallelism is reduced to 2.
The default test grouping now splits the test suites into
testForkedWorker count.
* Use Global scope for overall worker count
Subproject scoping can be used for single vs split
* Adds math.max(..., 1)
* Avoid running setup/cleanup on an empty selection
* Fix the tag application order
* Rename key to workerMaxInstances
* Implement TestTopology
* fix: Fixes exclusive test
Note due to the limitation of the tag system, setup-clean/cleanup
may overlap.
* Check that workerMaxInstances >= 1
**Problem**
The accept loop did not catch an exception from onIncomingSocket, so
its thread ended. Nothing closed the server socket, so the path stayed
bound: the server took no further client, and the process kept running.
The socket of the client that failed stayed open too.
**Solution**
Log that client, close its socket, and take the next one. An exception
from accept itself keeps the handling it had, so a socket that is
really broken still ends the loop.
The callback now takes a holder rather than the socket, and calls
AtomicCloseable.release to keep it. The loop closes whatever the
callback left behind.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
**Problem**
Nothing covers what the accept loop does when it cannot serve a client.
The next commit changes it.
**Solution**
Assert what the server does now. An exception from onIncomingSocket
ends the accept loop, and the server serves no client after that.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
* fix: restart the sbt server when -D options change
* fix: don't restart on completions, wait for the socket
* fix: only trust -D options recorded for this build
* fix: confirm the server left before reporting it gone
* Update protocol/src/main/contraband/portfile.contra
Co-authored-by: eugene yokota <[email protected]>
* fix: record the server -D options as a list
* fix: record the -D options without their values
The connection file keeps the name of each option and a salted digest of it
rather than the option itself, and the comparison follows the JVM in taking the
last definition of a name. A client that is not allowed to restart the server
says which options it cannot pick up instead of staying quiet about them.
* fix: compare the -D options given before a command
A trailing -D option used to switch the whole comparison off, which dropped the
options written before it, and a shutdown request that never reached the server
waited the full timeout before saying so.
* fix: leave a server no client started alone
The connection file says whether its recorded options are the whole story, so a
server an editor keeps is warned about instead of shut down, while one the client
started with no options is still replaced. A restart that cannot reach the server
warns and carries on rather than failing the invocation.
* build: filter NetworkClient clinit in MiMa
---------
Co-authored-by: eugene yokota <[email protected]>
CommandExchange asked twice whether an exec came from the closing
channel, and cancelled an exec in two places. Each has a name now, as
does the read of shuttingDown and the Try whose failure is ignored.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Problem
-------
SemanticdbPlugin.compileIncAndCacheSemanticdbTargetRootTask is a cached task
that reads semanticdbTargetRoot as a bare File, so its absolute path is hashed
into the action cache key:
((true,${OUT}/.../classes,${OUT}/.../classes.sbtdir.zip),/tmp/repro/target/out/.../meta)
Every other component of that key is virtualized. This one names one checkout
on one machine, so the entry can never be served to a second checkout, to a
colleague, or to CI. A monorepo we measured had 118 such keys, none of them
reachable across machines.
Fixes#9709
Solution
--------
Read the target root through semanticdbTargetRootVF, following the convention
sourcesVF and resourcesVF already set: an uncached task holding the virtualized
form of a File-typed key, so that a cached task hashes ${OUT}/.../meta rather
than the path behind it. A cached helper would key on the same bare File and
reintroduce the problem one level up.
The scripted test asserts the property the bug breaks: relocating
rootOutputDirectory must leave every key alone, so the second compile is a pure
cache hit. It pins Compile / semanticdbOptions, because -semanticdb-target
carries an absolute path into scalacOptions as well, which is a separate issue
and would otherwise mask this one.
Generated-by: Claude Opus 5
* [2.x] test: Pin how a server writes its files
**Problem**
Nothing covers how the server writes the portfile and the token file.
The next commit changes both.
**Solution**
Assert what the server does now. The tests read the token file only.
The server rewrites it on every authentication, so a test can read it
while the server writes. The server writes the portfile once, as it
starts.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
* [2.x] fix: Write the portfile and zip by rename
**Problem**
A client reads the portfile as soon as it appears. The server wrote it
with IO.write, in place, so a client could read part of one and fail to
parse it. ActionCache staged and renamed its zip by hand.
**Solution**
The portfile calls IO.writeFileAtomically, and the cache zip calls
IO.copyFile, which stages and renames on its own. The token file needs
its staging file kept to the owner, which neither can do yet, so it
still writes its own way.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
* [2.x] sbt-io: upgrade to v1.13.1
* [2.x] fix: Write the token file by rename
**Problem**
writeTokenfile would first remove the existing token file, and then go
through a non-atomic sequence of steps to write a new one. If a client
attempts to read the file during that process, it is likely to find it
either missing or half-written.
**Solution**
Use newly released IO.writeFileAtomically with ownerOnly flag set.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
* [2.x] test: Pin how a client meets an unreadable portfile
**Problem**
Nothing covers what the client does when it cannot read the portfile.
The next commit changes it.
**Solution**
Assert what the client does now. The client reads the portfile once, so
a whole one that arrives a moment later comes too late.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
* [2.x] fix: Retry a read before discarding a server
**Problem**
One failure was enough for the client to delete the portfile and start
a second server, which took the socket from a server that was still
running. The client already tried ten times when a Windows pipe was
busy. It tried once when the portfile would not parse, and once when
the connection was refused.
**Solution**
Give those two failures the same ten attempts as the busy pipe. After
ten the client still deletes the portfile, which a portfile left by a
dead server needs.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Follow-up to #9643. The verification added there re-read every downloaded blob right after writing it, and it sat on top of an older cost: DiskActionCacheStore fully re-hashes CAS files on every findBlobs/syncBlobs/getBlobs call, even though the file name is the digest. On a cache hit that meant hashing a whole classpath worth of blobs just to conclude nothing changed.
- Remote blobs (grpc downloads and inline data) are hashed while the stream is written to a staged temp file, then moved into the CAS, so verification costs no extra I/O and tampered bytes never land under the CAS name. Local puts are copied without hashing and are not stamped, so a mismatched local file is still caught by the lookup hash.
- Verified entries are remembered per process by (digest, size, mtime, fileKey), so repeated lookups are a stat instead of a full hash. Any external change to a CAS file invalidates the stamp and forces a re-hash. This is the same trade-off CacheImplicits already makes for file stamps.
The cache now writes each value-distinct ModuleReport once into a "modules"
table and gives each configuration's details a list of indices into it. That is
~5x smaller and hands the reader instance sharing structurally, with nothing to
probe to rediscover it.
CacheStoreFactory.makeCompressed writes gzip-framed JSON and reads either
framing, sniffing the magic bytes so a cache written before a store was
switched to compression still loads. It defaults to make, so existing
CacheStoreFactory implementations are unaffected.
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
**Problem**
Four fields hold a closeable that a later caller replaces. Each one
reads the field, closes what it finds and writes the new value in its
own way.
**Solution**
AtomicCloseable holds such a value. It closes the value it replaces,
and closes the value that loses a race to fill an empty field.
Two things change for a caller. The client replaced its session
without closing the one it dropped, and now closes it. A close that
throws no longer escapes: each of these sites is discarding the value
it closes, and whatever led there matters more than the close.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
This is essentially applying the same fix as in #8795 but for the Windows sbt.bat script.
We now only set -Dsbt.global.base when explicitly asked for with the --sbt-dir option.
**Problem**
1. On Windows, sbt.bat re-parses its own already-received arguments a second time internally, via call/goto with an unquoted variable splice (call :run !SBT_ARGS!, and similarly for -D/-XX flags, several _SBT_OPTS-sourced flags, and the native-client dispatch path). That extra re-tokenization could let &, |, (, or ) inside an argument escape their quoting and run as separate shell commands.
2. The -D/-XX/-- argument handling spliced the raw argument value into a for /F ... in ("%g%") do ( ... ) construct nested up to four levels deep inside multi-line if blocks. Apparently, that confuses cmd to miscount the parentheses.
**Solution**
1. Call :run without the argument.
2. Avoid for /F
A cache-restored tree is a farm of symlinks into the shared CAS. Writing it is
cheap, but walking it is not: resolving each link reads an inode out of a
directory far too large to stay in the OS metadata cache. Measured against a
57 GB / 952k-blob CAS, a cold stat of a CAS blob costs 165 us against 3.6 us for
a regular file, which makes a cold classpath walk of a restored tree roughly 38x
slower than one of real files. On an 81-module build with exportJars, a no-op
compile drops from 14.75s to 5.96s once the tree is real files.
DiskActionCacheStore now seeds the existing symlinkSupported latch from the
filesystem holding the CAS, and materializes entries with copyFile on APFS.
Files.copy reaches clonefile(2) there, so each entry gets its own inode and walks
at full speed while its data blocks stay shared with the CAS: the restored tree
measured 11 MB against 137 MB for the symlinked one. Copies also cannot corrupt
a blob when a task overwrites its output, which a symlink into the CAS can.
On every other filesystem the behavior is unchanged.
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Stas Shevchenko
Albert Meltzer
eugene yokotaandkenji yoshida
Stas Shevchenko
Jozef Koval
Anatolii Kmetiuk
Christian Harrington
Mai Huy HoàngandClaude Opus 5
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Viktor Rudebeck