I noticed that sometimes this test hangs in ci. I think it may be
because sometimes the event for A got handled while the cache entry for
B.scala was invalidated. Because the test runs ~compile, the next run of
compile would then reset the cache entry for B.scala. The next run of
watchOnIteration invalidtes B.scala again, but this time to the same
hash, so it doesn't actually trigger a file input event. By changing the
content of B.scala every time, we ensure that it should still cause a
file event.
I haven't confirmed that this will fix the sporadic hangs because I
can't reproduce on my machine, but this change won't hurt.
I had previously not though there was much reason to support commands in
continuous builds. This was primarily because there were a number of
questions regarding semantics. Commands cannot have fileInputs
specifically assigned to them because they don't have an associated
scope. They also can arbitrarily modify state so what is the expectation
when running ~foo where foo is a command that, for example, replaces
itself. I settled on the following semantics:
1) Commands run in a continuous build cannot modify the sbt execution
state which is to say that the state that is returned by continuous
is the same that was passed in (unless a reload occurred or we exited
the build with an exception)
2) Any global watchTriggers or fileInputs apply to a watched command.
They automatically inherit any fileInputs that are queried when
running tasks in a command. So, for example, ~+compile does what
you'd expect.
The implementation is fairly straightforward. If we can successfully
parse a command, but we cannot parse a scopedKey from it, we assign it a
private ScopedKey. When computing the watch settings for that key, we
will select the global settings through delegation. This is how it picks
up the global watchTriggers.
To run the command, I had to rework the task evaluation portion because
a command may return a state with additional commands to run. The cross
build command works this way. We recursively run all of the commands
starting with the original until we run out of commands to run. As part
of this work, I was able to remove the three argument version of
Command.processCommand that I'd previously added to support my old
approach to evaluating commands. This was a nice bonus.
I added scripted tests that check that global watchTriggers are picked
up and that commands that delegate to a command that uses fileInputs
automatically pick up those inputs during the watch. I also added a test
that approximates the ~+compile use case and ensures that the failure
semantics are what we expect and that the task runs for all defined
scala versions.
I'm debugging some test failures on travis in ServerSpec and I noticed
that very often noisy stack traces were printed because the process was
not given time to exit. There also was a confusing error message that
said that it would wait 30 seconds for the process to start, but it
would actually wait 90 seconds. I rewrote waitForPortfile to use
durations instead of integers to make it more clear. I changed
process from type Future[Process] to Process. There was no reason to
make it a future. I also made waitForString a blocking function because
it also didn't need to be implemented with a future.
I noticed that sbt 1.3.0 was using more cpu when idling (either at the
shell or while waiting for file events) than 1.2.8. This was because I'd
reduced a number of timeouts to 2 milliseconds which was causing a
thread to keep waking up every 2 milliseconds to poll a queue. I thought
that this was cheaper than it actually is and drove the cpu utilization
to O(10%) of a cpu on my mac.
To address this, I consolidated a number of queues into a single queue
in CommandExchange and Continuous. In the CommandExchange case, I
reworked CommandChannel to have a register method that passes in a Queue
of CommandChannels. Whenever it appends an exec, it adds itself to the
queue. CommandExchange can then poll that queue directly and poll the
returned CommandChannel for the actual exec. Since the main thread is
blocking on this queue, it does not need to frequently wake up and can
just poll more or less indefinitely until a message is received. This
also reduces average latency compared to older versions of sbt since
messages will be processed almost as soon as they are received.
The continuous case is slightly more complicated because we are polling
from two sources, stdin and FileEventMonitor. In my ideal world, I'd
have a reactive api for both of those sources and they would just write
events to a shared queue that we could block on. That is nontrivial to
implement, so instead I consolidated the FileEventMonitor instances into
a single FileEventMonitor. Since there is now only one FileEventMonitor
queue, we can block on that queue for 30 milliseconds and the poll
stdin. This reduces cpu utilization to O(2%) on my machine while still
having reasonably low latency for key input events (the latency of file
events should be close to zero since we are usually polling the
FileEventMonitor queue when waiting).
I actually had a TODO about the FileEventMonitor change that this
resolves.
I noticed that ~run in the Play plugin relied on the presence of the
ContinuousEventMonitor key. Rather than completely break that feature, I
re-added the ContinuousEventMonitor attribute to the state in a
continuous build. That being said, the play team does need to update
their plugin because reading from the console no longer works in 1.3.0 so the
user has to Ctrl-C to exit the watch. I think the best way for them to
fix this is to override the '~' command in their plugin and if the input
is 'run', then they do their custom thing, otherwise they delegate to
the default '~' command.
The intent was to bring prominence to the desires expressed in the
contributing guidelines. But nowadays GitHub has ways in which it
informs users (e.g. "the contributing guidelines have changed since you
last contributed"). So I think we can drop this now.
Not caching scala reflect is extremely painful if the build uses
scalatest. It adds O(1second) to my watch performance benchmarks. It
actually made sbt 1.3.0 much slower than 0.13.17
I was benchmarking sbt with turbo mode on and found that tests weren't
running. This was because we were inadvertently excluding all of the
dependency jars from the dynamic classpath. I have no idea why the
scripted tests didn't catch this.
The scalatest scripted test didn't catch this because 'test' just
automaticaly succeeds if no test frameworks are found. To guard against
regression, I had to ensure that 'test' failed for every strategy if a
bad test file was present.
We want to check the build sources before any command runs, not just
tasks. To achieve this, I moved the logic for checking for build source
changes to CommandProcess.processCommand. Also, @smarter had noticed
that if a user modified a build file and then ran reload, a warning
would be displayed about changed build sources even though they had just
ran reload. This was because running reload didn't update the previous
cache for checkBuildSources / fileInputStamps. I fixed that bug by
running 'checkBuildSources / changedInputFiles' instead of
'checkBuildSources' when the user runs reload.
I verified that after this change:
- If I changed a build file and ran 'show version' a warning was printed
before it displayed the version. If I also set
global / onChangedBuildSource := ReloadOnSourceChanges, it
automatically reloaded before displaying the version.
- If I changed a build source and ran 'reload', followed by
'show version', no warnings were ever displayed.
As an implementation detail, I had to add the Aggregation.suppressShow
attribute key. We set this key to true before checking the build
sources. Without this, log.success is called whenever we check the build
sources which is both confusing and noisy.
Because we are sharing the scala library classloader with test and run,
it is possible that sbt will be competing with for resources with the
test and run tasks when trying to get threads from the global execution
context. Also, by using our own execution context, we can shut it down
when sbt exits.
The motivation for this change is that I was looking at the active jvm
threads of an idle sbt process and noticed a bunch of global execution
context threads.
@olegych reported in #4721 that play projects would get stuck in a
strange loop where modifying any source file would cause that source
file to always be recompiled every time a build was triggered regardless
of whether or not it was modified. This was because the play project
sets custom watchSources (using the legacy api) that overlap with the
fileInputs.
There were two parts to this fix:
1) When detecting an event, find if any of the dynamic inputs that cover
the glob use a hash. If so, these are file inputs so we want to
update the hash for the path, not the last modified time.
2) Only write hashes into the persistent file stamp cache. Computing the
last modified time is much cheaper than the hash so it makes sense to
avoid ever caching last modified times.
I wrote a scripted test that fails if Continuous writes a last modified
time into the file stamp cache instead of a hash. I also verified
manually that a sample play project no longer exhibits the weird
recompilation behavior.
Fixes#4721
This allows the user to do, for example,
watchTriggeredMessage := { (count, path, commands) =>
println(Watched.clearScreen)
watchTriggeredMessage.value(count, path, commands)
}
Also, there was a bug where I accidentally inadvertently used the
deprecated watch message setting where I meant to use the deprecated
trigger message setting.
Fixes#4696
@olegych reported in https://github.com/sbt/sbt/issues/4722 that
sometimes even when a build was triggered during watch that no
recompilation would occur. The cause of this was that we never
invalidated the file stamp cache for managed sources or output files.
The optimization of persisting the source file stamps between task
evaluations in a continuous build only really makes sense for unmanaged
sources. We make the implicit assumption that unmanaged sources are
infrequently updated and generally one at a time. That assumption does
not hold for unmanaged source or output files.
To fix this, I split the fileStampCache into two caches: one for
unmanaged sources and one for everything else. We only persist the
unmanagedFileStampCache during continuous builds. The
managedFileStampCache gets invalidated every time.
I added a scripted test that simulates changing a generated source file.
Prior to this change, the test would fail because the file stamp was not
invalidated for the new source file content.
Fixes#4722
eugene yokota
Dale Wijnand
Ethan Atkins
Ethan Atkins