Commit Graph

2218 Commits

Author SHA1 Message Date
Ethan Atkins fcd24ba7cd Reduce command polling period
The only thing this function does is poll a queue and check a deadline.
There is no need to put such a large sleep duration in.
2019-02-14 21:13:07 -08:00
Ethan Atkins c37f2607f1 Lint CommandExchange.scala
Clears out intellij warnings.
2019-02-14 21:09:44 -08:00
Ethan Atkins faf6348a16 Run gc when idle
I often find that when I run a command it takes a long time to start up
because sbt triggers a full gc. To improve the ux, I update the command
exchange to run full gc only once while it's waiting for a command to
run and only after the user has been idle for at least one minute.

Bonus: optimize imports
2019-02-14 21:09:17 -08:00
Guillaume Martres 85f91e76ca Remove Dotty handling in scalaInstance
sbt-dotty 0.3.0 (https://github.com/lampepfl/dotty/pull/5835) sets
`managedScalaInstance := false` and does everything by itself, so this
isn't needed anymore.
2019-02-12 23:30:45 +01:00
eugene yokota fbadfabfe2
Merge pull request #4538 from eatkins/compiler-cache
Clear compiler cache in clearCaches
2019-02-04 10:27:14 -05:00
eugene yokota e73d1fb7c6
Merge pull request #4537 from eatkins/allow-watch-loops
Allow watch loops
2019-02-04 10:25:11 -05:00
Ethan Atkins e8af828c73 Add FileCacheEntry
Previously, we were leaking the internal details of incremental
compilation to users by defining FileTree(DataView|Repository)[Stamp].
To avoid this, I introduce the new class FileCacheEntry that is quite
similar to Stamp except defined using scala Options rather than java
Optionals. The implementation class just delegates to an actual Stamp
and I provided a private[sbt] ops class that adds a
method `stamp` to FileCacheEntry. This will usually just extract the
stamp from the implementation class. This allows us to use
FileCacheEntry almost interchangeably with Stamp while still avoiding
exposing users to Stamp.
2019-02-02 16:03:59 -08:00
Ethan Atkins ba0494df14 Stop Stamped from inheriting File and TypedPath
In the FileTreeDataView use case, we were previously working with
FileTreeDataView[Stamped], which actually contained a lot of redundant
information because FileTreeDataView.Entry[_] has a toTypedPath method
that could be used to read the path related fields in Stamped. Instead,
we can just return the Stamp itself in FileTreeDataView.list* methods
and convert to Stamped.File where needed (i.e. in ExternalHooks).

Also move BasicKeys.globalFileTreeView to Keys since it isn't actually
used in the main-command project.
2019-02-02 12:22:57 -08:00
Ethan Atkins 0bdc30b60b Rename StampedFile to Stamped 2019-02-02 12:22:57 -08:00
Ethan Atkins f44e2a2e74 Clear compiler cache in clearCaches
Resident compilation actually works pretty well most of the time*,
but if there ever is an issue with the cached compilation, we should be
able to easily clear the cache.

* I've only had issues when package objects are involved
2019-02-02 12:20:32 -08:00
Ethan Atkins 116d2ae815 Use ? instead of extracting state
I was unaware of the '?' method when I wrote this.
2019-02-01 18:01:56 -08:00
Ethan Atkins 2ef97768a0 Don't attempt to prevent certain watch loops
If the managedSources task writes into an unmanaged source directory,
that would cause an infinite loop. I don't think it's worth doing out of
band task execution to try and prevent this.
2019-02-01 18:01:56 -08:00
Ethan Atkins 92807869ea Fix community build
The community build was broken for some projects because I broke builds
that relied on the unscoped definition of `runner`. To preserve legacy
behavior, I restore the old unscoped behavior and append the new scoped
runners that used the layered classloaders. This makes more sense
because the layered classloaders were specifically designed for the
Runtime and Test configurations and may not make sense in other
contexts.
2019-01-30 20:54:08 -08:00
Ethan Atkins ba3ff8198d Remove unneeded parens in Definition.scala 2019-01-30 10:46:00 -08:00
Ethan Atkins 06a153278f Cleanup cache when CommandExchange shuts down
I noticed that sometimes when running scripted tests that I'd run out of
metaspace. I believe that this may be due to the caffeine cache leaking
classloaders. Regardless, it's a good idea to clear the cache whenever
we shutdown the command exchange or reload the state.
2019-01-30 10:45:22 -08:00
Ethan Atkins 0a4fbc9f5a Set classLoaderLayeringStrategy in relevant configs
Previously, the ClassLoaderLayeringStrategy was set globally. This
didn't really make sense because the Runtime and Test configs had
different strategies available (Test being a superset of Runtime).
Instead, we now set the layering strategy in the Runtime and Test
configurations directly. In doing this, we can eliminate the Default
ClassLoaderLayeringStrategy. Previously this had existed so that we
could set the layering strategy globally and have it do the right thing
in both test and runtime.

To implement this, I factored out the logic for generating the layered
classloader in the test task and shared it with the runtime task. I did
this because I realized that Test / run is a thing. Previously I had
been operating under the assumption that the runner would never include
the test dependencies. Once I realized this, it made sense to combine
the logic in both tasks.

As a bonus, I only allow the layering strategies that explicitly make
sense to be set in each configuration. If the user sets an invalid
strategy, an error will be thrown that specifies the valid strategies
for the task.

I also added ScalaInstance as an option for the runtime layer. It was an
oversight that this was left out.
2019-01-30 08:55:22 -08:00
Ethan Atkins 3a211710f9 Rename some layering related fields/classes
In code review, @eed3si9n suggested that I switch to a more verbose and
descriptive naming scheme. In addition to trying to make layers more
descriptive, I also made the various layer case objects extend the
relevant layers so it's more clear what the layer should look like.
2019-01-30 08:55:22 -08:00
Ethan Atkins 1ca6b97de2 JDK 11 fixes
In JDK 11, String.lines returns java.util.stream.Stream[String]. Switch
to linesIterator instead.
2019-01-30 08:55:22 -08:00
Ethan Atkins 1d467bd966 Fix FileRepository implementation for scripted
I was seeing spurious travis failures and I finally tracked it down to
the fact that in some cases, the project metabuild would sometimes use a
caching file tree repository instead of a polling repository. This
caused problems because the caching repository can take a few milliseconds
to detect changes in a directory. Because scripted copies the project
sources to the temporary test directory, it was possible for the project
meta build compilation to be initiated before the cache was aware of all
of the files.

The reason this happened was because scripted would create a state where
the remaining commands looked like:
List(sbtPopOnFailure, resumeFromFailure, notifyUsersAboutShell, iflast shell, ~compile, < 41684)
The ~compile command was causing the continuous flag to get set to true
which caused the default file tree repository task to return the caching
version. The reason for the continuous flag was so that when sbt is
started in a non-interactive mode where the command is to be repeated,
then we use the caching file tree repository. To support this use case,
we just need to check that the last command begins with `~`, not that
_any_ command begins with `~`.
2019-01-30 08:55:22 -08:00
Ethan Atkins 5fc5846737 Add TaskRepository to manage ClassLoaderCache
We want the user to be able to invalidate the classloader cache in the
event that it somehow gets in a bad state. The cache is, however,
defined in multiple configurations, so there are in fact many
ClassLoaderCache instances that are managed by sbt. To make this sane, I
add a global cache that is keyed by a TaskKey[_] and can return
arbitrary data back. Invalidating all of the ClassLoaderCache instances
is then as straightforward as just replacing the TaskRepository
instance.

I also went ahead and unified the management of the global file tree
repository. Instead of having to specifically clear the file tree
repository or the classloader cache, the user can now invalidate both
with the new clearCaches command.
2019-01-30 08:55:22 -08:00
Ethan Atkins a06f5435c6 Use layered ClassLoaders in run and test tasks
Using the data structures that I added in the previous commits, it is
now possible to rework the run and test task to use (configurable)
layered class loaders. The layering strategy is globally set to
LayeringStrategy.Default. The default strategy leads to what is
effectively a three layered ClassLoader for the both the test and run
tasks. The first layer contains the scala instance (and test framework
loader in the test task). The second layer contains all of the
dependencies for the configuration while the third layer contains the
project artifacts.

The layering strategy is very easily changed both at the Global or
Configuration level, e.g. adding
Test / layeringStrategy := LayeringStrategy.Flat
to the project build.sbt will make the test task not even use the scala
instance and instead a create a single layer containing the full
classpath of the test task.

I also tried to ensure that all of the ClassLoaders have good toString
overrides so that it's easy to see how the ClassLoader is constructed
with, e.g. `show testLoader`, in the sbt console.

In this commit, the ClassLoaderCache instances are settings. In the next
commit, I make them tasks so that we can easily clear out the caches
with a command.
2019-01-30 08:55:22 -08:00
Ethan Atkins f118a5e410 Add LayeringStrategy
This introduces a new trait LayeringStrategy that is used to configure
how sbt constructs the ClassLoaders used by the run and test tasks. In
addition to defining the various options, I try to give a good high
level overview of the problem that the LayeringStrategy is intended to
address in its scaladoc.
2019-01-30 08:55:22 -08:00
Ethan Atkins ef08290ecc Add ClassLoaderCache
In order to speed up the start up time of the test and run tasks, I'm
introducing a ClassLoaderCache that can be used to avoid reloading the
classes in the project dependencies (which includes the scala library).
I made the api as minimal as possible so that we can iterate on the
implementation without breaking binary compatibility. This feature will
be gated on a feature flag, so I'm not concerned with the cache class
loaders being useable in every user configuration. Over time, I hope
that the CachedClassLoaders will be a drop in replacement for the
existing one-off class loaders*.

The LayeredClassLoader was adapted from the NativeCopyLoader. The main
difference is that the NativeCopyLoader extracts named shared libraries
into the task temp directory to ensure that the ephemeral libraries are
deleted after each task run. This is a problem if we are caching the
ClassLoader so for LayeredClassLoader I track the native libraries that
are extracted by the loader and I delete them either when the loader is
explicitly closed or in a shutdown hook.

* This of course means that we both must layer the class loaders
appropriately so that the project code is in a layer above the cached
loaders and we must correctly invalidate the cache when the project, or
its dependencies are updated.
2019-01-30 08:55:22 -08:00
Ethan Atkins 5bbda9cf69 Add Repository interface
I am going to be introducing multiple caches throughout sbt and I am
going to build these features out using this simple Repository
interface. The idea is that we access data by some key through the
Repository. This allows us to use the strategy pattern to easily switch
the runtime implementation of how to get the data.
2019-01-30 08:55:22 -08:00
Ethan Atkins ec22d6c0da Add LRUCache to sbt.internal
I am going to add a classloader cache to improve the startup performance
of the run and test tasks. To prevent the classloader cache from having
unbounded size, I'm adding a simple LRUCache implementation to sbt. An
important characteristic of the implementation of the cache is that when
entries are evicted, we run a callback to cleanup the entry. This allows
us to automatically cleanup any resources created by the entry.

This is a pretty naive implementation that uses an array of entries that
it manipulates as elements are removed/accessed. In general, I expect
these caches to be quite small <= 4 elements, so the storage overhead /
performance of the simple implementation should be quite good. If
performance ever becomes an issue, we can specialzed LRUCache.apply to
use a different implementation for caches with large limits.
2019-01-30 08:55:22 -08:00
Ethan Atkins 161dfd77f2 Update TypedPath apis
I renamed getPath to toPath in io so we need to update sbt to using the
correct api.
2019-01-16 13:22:56 -08:00
Ethan Atkins 541d4047e6 Bump scala version 2.12.8
I ran a zsh one liner to generate this:
git grep "2.12.7" | cut -d ':' -f1 | sort | uniq | xargs perl -p -i -e "s/2.12.7/2.12.8/"
2019-01-08 11:11:23 -08:00
Thomas Droxler fff6b044bc Use `ListBuffer` instead of `List` in `LanguageServerReporter`
In order to have a better data structure for appending
2019-01-04 13:31:04 +01:00
Thomas Droxler a5388fed60 Preserve errors order when publishing diagnostics
The `sbt-server` was prepending a new probem and not appending.

The result was a `textDocument/publishDiagnostics` notification
containing a inverted list of problems compare to what was show in the
sbt console.
2019-01-04 13:31:04 +01:00
eugene yokota 1e7564c408
Merge pull request #4499 from dwijnand/ParseKey-rare-failure
Avoid ParseKey failing due to homonymous axes
2018-12-22 14:29:59 -05:00
Dale Wijnand 30dca6b818
Cleanup ConfigIndex 2018-12-22 17:05:51 +00:00
Dale Wijnand 3d924978c2
Add some matcher messaging to ParserSpec 2018-12-22 17:02:48 +00:00
Dale Wijnand 6988b2289a
Cleanup some formatting/wrapping 2018-12-22 17:02:47 +00:00
Dale Wijnand e533bc93e8
Avoid ParseKey failing due to homonymous axes 2018-12-22 12:51:47 +00:00
Dale Wijnand 645f484184
Switch ParseKey to propertyWithSeed 2018-12-22 12:51:47 +00:00
Dale Wijnand 1f14a2c340
Split out ParseKey properties 2018-12-22 12:51:47 +00:00
Dale Wijnand a254341b6f
Remove dead guessedConfigIdents code 2018-12-22 12:37:42 +00:00
Ethan Atkins bfffb4f489 Rename TestBuild.Keys to TestBuild.TestKeys
It drives me crazy that in intellij when I do the go to class task that
TestBuild.Keys comes up before Keys. Given how central Keys is to sbt,
it doesn't seem like a good idea to alias that particular class name.
2018-12-14 13:55:19 -08:00
Ethan Atkins de7c67ea45 Optimize imports in Defaults, Main and Keys
It was becoming a pain to work on these files in intellij because the
auto-import feature would implicitly optimize all of the imports in
these files, leading to a large diff. I'd then have to go and manually
add the import that I care about. This change does add some wildcard
imports, which I don't always love, but these files are so unwieldy
already that I think it's worth it to have the imports follow the format
preferred by intellij.
2018-12-03 13:28:10 -08:00
eugene yokota 34ef5ab500
Merge pull request #4454 from eed3si9n/wip/prompt-fix
Fix prompt for task progress
2018-11-29 10:47:30 -05:00
Eugene Yokota 9858bc29fb more tweaking on task progress
move the cursor up after the user enters command.
2018-11-29 04:40:40 -05:00
Ólafur Páll Geirsson fefb83f8fd Update semanticdb-scalac to v4.1.0
This release supports more Scala versions, works on Java 11 and
is ~8mb smaller than v4.0.0.
2018-11-28 08:56:18 +01:00
Arnout Engelen ae932f7390 Discover Oracle JDK installed in /opt 2018-11-22 12:50:13 +01:00
Ethan Atkins c00cc37953 Do not require leading semicolon for multi command
It has long been a frustration of mine that it is necessary to prepend
multiple commands with a ';'. In this commit, I relax that restriction.
I had to reorder the command definitions so that multi comes before act.
This was because if the multi command did not have a leading semicolon,
then it would be handled by the action parser before the multi command
parser had a shot at it. Sadness ensued.
2018-11-19 10:42:51 -08:00
Eugene Yokota b00dcb1b9d Fix task progress blank line behavior
Run readyLog lazily.
2018-11-16 23:06:36 -08:00
Eugene Yokota d58b302a97 Fix prompt for task progress 2018-11-16 23:05:51 -08:00
Ethan Atkins 7b5cf84fe1 Fix '~' for dependent projects with a broken parent
In #4446, @japgolly reported that in some projects, if a parent project
was broken, then '~' would immediately exit upon startup. I tracked it
down to this managed sources filter. The idea of this filter is to avoid
getting stuck in a build loop if managedSources writes into an unmanaged
source directory. If the (managedSources in ThisScope).value line
failed, however, it would cause the watchSources, and by delegation,
watchTransitiveSources task to fail. The fix is to only create this
filter if the managedSources task succeeds.

I'm not 100% sure if we shouldn't just get rid of this filter entirely
and just document that '~' will probably loop if a build writes the
result of managedSources into an unmanaged source directory.
2018-11-11 09:35:46 -08:00
Eugene Yokota c51c00bb3f Use app classpath for metabuild
Fixes #4437

Until now, sbt was resolved twice once by the launcher, and the second time by the metabuild.
This excludes sbt from the metabuild graph, and instead uses app classpath from the launcher.
2018-11-09 04:32:38 -05:00
Eugene Yokota c91371b7f8 add isMetaBuild setting
Fixes #3436

This implements isMetaBuild setting that is explicitly for meta build only,
unlike sbtPlugin setting which can be used for both meta build and plugin development purpose.
2018-11-09 03:08:47 -05:00
Ethan Atkins 7ec65f174d Close the current global file tree view on unload
I noticed that when using the latest nightly, triggered execution would
fail to work if I switched projects with, e.g. ++2.10.7. This was
because the background thread that filled the file cache was incorrectly shutdown.
To fix this, we just need to close whatever view is cached in the
globalFileTreeView attribute in the exit hook rather than the view
created by the method.

After making this change and publishing a local SNAPSHOT build, I was
able to switch projects with ++ and have triggeredExecution continue to
work.
2018-10-29 14:21:36 -07:00
Eugene Yokota f70ce9bab0 Set withMetadataDirectory by default
This partly fixes cached resolution.

Ref https://github.com/sbt/sbt/issues/3761
2018-10-15 03:26:39 -04:00
eugene yokota 1f8e9c9657
Merge pull request #4410 from eed3si9n/wip/semantic
ThisBuild / enableSemanticDB
2018-10-12 01:00:16 -04:00
Eugene Yokota aa8c1f484b implement SemanticdbPlugin
This makes it easier to enable SemanticDB build wide.
2018-10-11 15:44:15 -04:00
Andrea Peruffo a2607f1da6
Merge branch 'develop' into refactorCancellations 2018-10-11 18:57:57 +01:00
eugene yokota e759d17b2e
Merge pull request #4397 from andreaTP/lspCompletions
[sbt-server] LSP completions support
2018-10-11 12:49:23 -04:00
andrea 02b19752eb refactoring of server cancellation request 2018-10-11 14:03:41 +01:00
andrea 34e0fc159c [sbt-server] LSP completions support 2018-10-11 13:34:40 +01:00
Ethan Atkins f579b89577 Fix windows reload loop
On windows* it was possible to get into a loop where the build would
continually restart because for some reason the build.sbt file would get
touched during test (I did not see this behavior on osx). Thankfully,
the repository keeps track of the file hash and when we detect that the
build file has been updated, we check the file hash to see if it
actually changed.

Note that had this bug shipped, it would have been fixable by overriding
the watchOnEvent task in user builds.

The loop would occur if I ran ~filesJVM/test in
https://github.com/swoval/swoval. It would not occur if I ran
test:compile, so the fact that the build file is being touched seems
to be related to the test run itself.
2018-10-10 20:16:29 -07:00
Ethan Atkins d9e8ae18b2 Fix default FileTreeViewConfig
It was possible that on startup, when this function was first invoked,
that the default boot commands are present. This was a problem because
the global file repository is instantiated using the value of this task.
When we start a continuous build, this task gets run again to evaluate
again.

When sbt is started without an implicit task list, then the task is
implicitly shell as indicated by the command "iflast shell". We can use
this to determine whether or not to use the global file system cache or
not.
2018-10-10 18:35:30 -07:00
Ethan Atkins 0a9ae7b4b4 Use FileTreeRepository when interactive or continuous
Ideally we use the FileTreeRepository for interactive sessions by
default. A continuous build is effectively interactive, so I'd like that
case to also use the file tree repository. To avoid breaking scripted
tests, many of which implicitly expect file tree changes to be
instantaneously available, we set interactive to true only if we are not
in a scripted run, which can be verified by checking that the commands
contains "setUpScripted".
2018-10-09 21:52:38 -07:00
Ethan Atkins dc4f705500 Add support to rebuild a '~' task by pressing 'r'
Sometimes a user may want to rerun their task even if the source files
haven't changed. Presently this is a little annoying because you have to
hit enter to stop the build and then up arrow or <ctrl+r> plus enter to
rebuild. It's more convenient to just be able to press the 'r' key to
re-run the task.

To implement this, I had to make the watch task set up a jline terminal
so that System.in would be character buffered instead of line buffered.
Furthermore, I took advantage of the NonBlockingInputStream
implementation provided by jline to wrap System.in. This was necessary
because even with the jline terminal, System.in.available doesn't return
> 0 until a newline character is entered. Instead, the
NonBlockingInputStream does provide a peek api with a timeout that will
return the next unread key off of System.in if there is one available.
This can be use to proxy available in the WrappedNonBlockingInputStream.

To ensure maximum user flexibility, I also update the watchHandleInput Key to
take an InputStream and return an Action. This setting will now receive
the wrapped System.in, which will allow the user to create their own
keybindings for watch actions without needing to use jline themselves.

Future work might make it more straightforward to go back to a line
buffered input if that is what the user desires.
2018-10-09 12:09:42 -07:00
Ethan Atkins 25e97f99f5 Add custom external hooks
For projects with a large number of files, zinc has to do a lot of work
to determine which source files and binaries have changes since the last
build. In a very simple project with 5000 source files, it takes roughly
750ms to do a no-op compile using the default incremental compiler
options. After this change, it takes about 200ms. Of those 200ms, 50ms
are due to the update task, which does a partial project resolution*.

The implementation is straightforward since zinc already provides an api
for overriding the built in change detection strategy. In a previous
commit, I updated the sources task to return StampedFile rather than
regular java.io.File instances. To compute all of the source file
stamps, we simply list the sources and if the source is in fact an
instance of StampedFile, we don't need to compute it, otherwise we
generate a StampedFile on the fly. After building a map of stamped files
for both the sources files and all of the binary dependencies, we simply
diff these maps with the previous results in the changedSources,
changedBinaries and removedProducts methods.

The new ExternalHooks are easily disabled by setting
`externalHooks := _ => None`
in the project build.

In the future, I could see moving ExternalHooks into the zinc project so
that other tools like bloop or mill could use them.

* I think this delay could be eliminated by caching the UpdateResult so
long as the project doesn't depend on any snapshot libraries. For a
project with a single source, the no-op compile takes O(50ms) so caching
the project resolution would make compilation start nearly
instantaneous.
2018-10-09 12:09:42 -07:00
Ethan Atkins 1f996185e1 Only use a file repository for interactive sessions
I realized that using the cache has the potential to cause issues for
batch processing in CI if some tasks assume that a file created by one
task will immediately be visible in the other. With the cache, there is
typically on O(10ms) latency between a file being created and appearing
in the cache (at least on OSX). When manually running commands, that
latency doesn't matter.
2018-10-09 12:09:42 -07:00
Ethan Atkins b155ffb77b Add support for polling some directories
It is not always possible to monitor a directory using OS file system
events. For example, inotify does not work with nfs. To work around
this, I add support for a hybrid FileTreeViewConfig that caches a
portion of the file system and monitors it with os file system
notification, but that polls a subset of the directories. When we query
the view using list or listEntries, we will actually query the file
system for the polling directories while we will read from the cache for
the remainder. When we are not in a continuous build (~ *), there is no
polling of the pollingDirectories but the cache will continue to update
the regular directories in the background. When we are in a continuous
build, we use a PollingWatchService to poll the pollingDirectories and
continue to use the regular repository callbacks for the other
directories.

I suspect that #4179 may be resolved by adding the directories for which
monitoring is not working to the pollingDirectories task.
2018-10-09 12:09:42 -07:00
Ethan Atkins 2b2b84f589 Use FileTreeDataView to collect files
Now that we have the fileTreeView task, we can generalized the process
of collecting files from the view (which may or may not actually cache
the underlying file tree). I moved the implementation of collectFiles
and addBaseSources into the new FileManagement object because Defaults
is already too large of a file. When we query the view, we also need to
register the directory we're listing because if the underlying view is a
cache, we must call register before any entries will be available.
Because FileTreeDataView doesn't have a register method, I implement
registration with a simple implicit class that pattern matches on the
underlying type and only calls register if it is actually a
FileRepository.

A side effect of this change is that the underlying files returned by
collectFiles and appendBaseSources are StampedFile instances. This is so
that in a subsequent commit, I can add a Zinc external hook that will
read these stamps from the files in the source input array rather than
compute the stamp on the fly. This leads to a substantial reduction in
Zinc startup time for projects with many source files. The file filters
also may be applied more quickly because the isDirectory property (which
we check for all source files) is read from a cached value rather than
requiring a stat.

I had to update a few of the scripted tests to use the `1.2.0`
FileTreeViewConfig because those tests would copy a file and then
immediately re-compile. The latency of cache invalidation is O(1-10ms),
but not instantaneous so it's necessary to either use a non-caching
FileTreeView or add a sleep between updates and compilation. I chose the
former.
2018-10-09 12:09:42 -07:00
Ethan Atkins d31fae59f7 Add global file repository task
Every time that the compile task is run, there are potentially a large
number of iops that must occur in order for sbt to generate the source
file list as well as for zinc to check which files have changed since
the last build. This can lead to a noticeable delay between when a build
is started (either manually or by triggered execution) and when
compilation actually begins. To reduce this latency, I am adding a
global view of the file system that will be stored in
BasicKeys.globalFileTreeView.

To make this work, I introduce the StampedFile trait, which augments the
java.io.File class with a stamp method that returns the zinc stamp for
the file. For source files, this will be a hash of the file, while for
binaries, it is just the last modified time. In order to gain access to
the sbt.internal.inc.Stamper class, I had to append addSbtZinc to the
commandProj configurations.

This view may or may not use an in-memory cache of the file system tree
to return the results. Because there is always the risk of the cache
getting out of sync with the actual file system, I both make it optional
to use a cache and provide a mechanism for flushing the cache. Moreover,
the in-memory cache implementation in sbt.io, which is backed by a
swoval FileTreeRepository, has the property that touching a monitored
directory invalidates the entire directory within the cache, so the
flush command isn't even strictly needed in general.

Because caching is optional, the global is of a FileTreeDataView, which
doesn't specify a caching strategy. Subsequent commits will make use of
this to potentially speed up incremental compilation by caching the
Stamps of the source files so that zinc does not need to compute the
hashes itself and will allow for continuous builds to use the cache to
monitor events instead of creating a new, standalone FileEventMonitor.
2018-10-09 12:09:42 -07:00
Ethan Atkins 28fd4a1e61 Add the ability to halt watch on failure
There may be instances where the user may wish to stop the watch if an
error occurs running the task. To facilitate this, I add boolean
parameter, lastStatus, to watchShouldTerminate. The value is computed by
modifying the state used to run the task to have a custom onFailure
command. If the task fails, the returned state will have the onFailure
command will be enqueued at the head of the remaining commands. The
result of the task then becomes true if the custom onFailure is not
present in the remaining commands and false if it is. We don't actually
run this command, so it's just implemented with the identity function.

I also updated Watched.watch to return an Action instead of Unit. This
enables us to return a failed state if Watched.watch returns
HandleError.
2018-10-09 12:09:41 -07:00
Ethan Atkins 7d3d3c71d6 Refactor Watched
This commit reworks Watched to be more testable and extensible. It also
adds some small features. The previous implementation presented a number
of challenges:

1) It relied on external side effects to terminate the watch, which was
   difficult to test
2) It exposed irrelevant implementation details to the user in the
   methods that exposed the WatchState as a parameter.
3) It spun up two worker threads. One was to monitor System.in for user
   input. The other was to poll the watch service for events and write
   them to a queue. The user input thread actually broke '~console'
   because nearly every console session will hit the <enter> key, which
   would eventually cause the watch to stop when the user exited the
   console.

To address (1), I add the shouldTerminate method to WatchConfig. This
takes the current watch iteration is input and if the function returns
true, the watch will stop.

To address (2), I replace the triggeredMessage and watchingMessage keys
with watchTriggeredMessage and watchStartMessage. The latter two keys
are functions that do not take the WatchState as parameters. Both
functions take the current iteration count as a parameter and the
watchTriggeredMessage also has a parameter for the path that triggered
the build.

To address (3), I stop using the sbt.internal.io.EventMonitor and
instead use the sbt.io.FileEventMonitor. The latter class is similar to
the former except that it's polling method accepts a duration, which may
be finite or infinite) and returns all of the events that occurred since
it was last polled. By adding the ability to poll for a finite amount of
time, we can interleave polling for events with polling System.in for
user input, all on the main thread. This eliminates the two extraneous
threads and fixes the '~console' use case I described before.

I also let the user configure the function that reads from System.in via
the watchHandleInput method. In fact, this method need not read from
System.in at all since it's just () => Watched.Action. The reason that
it isn't () => Boolean is that I'd like to leave open the option for the
ability to trigger a build via user input, not just terminating the
watch. My initial idea was to add the ability to type 'r' to re-build in
addition to <enter> to exit. This doesn't work without integrating
jline though because the input is buffered. Regardless, for testing
purposes, it gives us the ability to add a timeout to the watch by
making handleInput return true when a deadline expires.

The tests are a bit wonky because I still need to rely on side effects
in the logging methods to orchestrate the sequence of file events that
I'd like to test. While I could move some of this logic into a
background thread, there still needs to be coordination between the
state of the watch and the background thread. I think it's easier to
reason about when all of the work occurs on the same thread, even if it
makes these user provided functions impure.

I deprecated all of the previous watch related keys that are no longer
used with the new infrastructure. To avoid breaking existing builds, I
make the watchConfig task use the deprecated logging methods if they are
defined in the user's builds, but sbt will not longer set the default
values. For the vast majority of users, it should be straightforward to
migrate their builds to use the new keys.  My hunch is that the of the
deprecated keys, only triggeredMessage is widely used (in conjunction
with the clear screen method) and it is dead simple to replace it with
watchTriggeredMessage.

Note: The FileTreeViewConfig class is not really necessary for this commit.
It will become more important in a subsequent commit which introduces an
optional global file system cache.
2018-10-08 22:00:50 -07:00
Ethan Atkins 28aa1de32a Refactor continuous execution
This commit makes watch event logging work in the '~' command. The
previous design of the command made this difficult, so there is a
significant re-design of the implementation of '~'. I believe that this
redesign will allow the feature to be maintained and improved more
easily moving forward. With the redesign, it is now possible to test the
business logic of the watch command (and I add a rudimentary test that I
will build upon in subsequent commits).

A bonus of this redesign is that now if the user tries to watch an
invalid command, the watch will immediately terminate with an error
rather than get stuck waiting for events when the task can never
possibly succeed.

The previous implementation of the '~' command makes it difficult
to dynamically control the implementation arguments because it is
implemented in the command project which makes it unable to depend on
any task keys that are defined in the build. It works around this by
putting all of it's configuration in the Watched attribute which is
stored globally. This would not have been necessary if the function had
been defined in the main project where it could just extract the value
of the watched task rather than relying on the global attribute value.
Moreover, because it cannot depend on tasks, it makes it nigh impossible
to use the logging framework within the '~' command.

Another issue with the previous implementation is that it's somewhat
difficult to reason about. The executeContinuously has effectively two
entry points: one for the first time the command is run and one for each
subsequent invocation when a new build is triggered. The successive
invocations are triggered by prepending commands to run to the previous
state. This is made recursive by prepending the initial command (that
was prefixed with '~'. Which branch we're in is determined by checking
for the existence of a temporary attribute, that we must ensure that we
remove when the build is stopped. This makes a lot of behavior non-local and
difficult for an outsider who is less familiar with sbt to understand.

Broadly, this refactor does two things:
1) Move the definition of continuous from BasicCommands to BuiltInCommands
2) Re-work the implementation to be executed in code rather than using
   the sbt dsl.

The first part is simple. We just add an implementation of continuous to
BuiltInCommands and remove it from the list of BasicCommands. We need to
leave in the legacy implementation for binary compatibility. I also
moved all of the actual implementation logic into Watched, which makes
maintenance easier since most of the logic is in one place.

The second part is more complicated. Rather than rely on the sbt dsl
(e.g. `(ClearOnFailure :: next :: FailureWall :: repeat :: s)`) to
parse and run the command. We manually parse the command and generate a
task of type `() => State`. We don't actually need to do anything with
the generated state because we're going to return the original state at
the end of the command no matter what. With this task, we can then
create a tail recursive function that repeatedly executes the task until
the watch is terminated.

The parsing is handled in the Watch.command method (which is where I
moved the refactored BasicCommands.continuous implementation). The
actual task running and monitoring is handled in Watched.watch. This
method has no reference to the sbt state, which makes it testable. It sets
up an event monitor and then delegates the recursive monitoring to a
small nested function, Watched.watch.impl. One nice thing about this
approach is that it is very easy to reason about the life cycle of the
EventMonitor. The recursive call is within a try { } finally { } where
the monitor and stdin are guaranteed to be cleared at the end.

Adding support for a custom (and default) watch logger is trivial with
the new infrastructure and is done via the watchLogger TaskKey.

There was a small reporting race condition that was introduced by the
change to (2). Because the new implementation is able to bypass command
parsing for triggered builds, the watch message would usually end up
being printed before the task outcome was fully logged. To work around
this, I made the watch and triggered messages be logged rather than
printed directly to stdout. As a result, the only user visible result of
this change should be that instead of seeing:
"1. Waiting for source changes in project foo... (press enter to interrupt)",
users will now see:
"[info] 1. Waiting for source changes in project foo... (press enter to interrupt)".
2018-10-08 22:00:50 -07:00
Ethan Atkins 5e3f72ad8a Do not append default watch sources with ++=
Using ++= prevents these from being overridden by the user without doing
some nasty runtime manipulations.
2018-10-08 22:00:50 -07:00
Ethan Atkins 97598efcea Bump watch anti-entropy
There have been reports that often a new build will be triggered
immediately after the previous build even when none of the files have
been modified since the start of the last build. This can happen when,
for example, a program implements save with a rename. When that occurs,
a deletion watch event may trigger the build but the corresponding
creation event may be detected outside of the current 40ms window. By
bumping this value to 500ms, we hopefully prevent the majority of these
false triggers. For unusual workflows in which this longer quarantine
period is an issue, the setting can be overridden.
2018-10-08 22:00:50 -07:00
Ethan Atkins 6ffb4108e7 Set swoval.tmpdir property on startup
This change makes the temporary shared library that is created by
the swoval file-tree-views library to be extracted into the sbt global
base directory rather than the temp file. This way if there is a leak of
shared libraries, they can easily be found in ~/.sbt rather than in,
say, /tmp (or the osx/windows equivalent location). The extracted shared
library objects will be in the path ~/.sbt/swoval-jni. There is a
shutdown hook that removes them as well as a garbage collection process
that runs in the background whenever the swoval library is loaded, so
these shouldn't leak uncontrollably.
2018-10-08 22:00:50 -07:00
Ethan Atkins aeadd8db72 Remove unneeded import 2018-10-08 13:59:40 -07:00
Ethan Atkins 6725b39a84 Bump io to 1.3.0-M3
I had to turn off -Xfatal-warnings in commandProj because after updating
io, commandProj depends on the deprecated EventMonitor class. In #4335,
I stop using EventMonitor, but deprecate the Watched class which is both
defined and used (as an unused attribute key) in commandProj. I think we
can probably get rid of Watched in 1.4.x and certainly in a hypothetical
2.x, so hopefully we can restore -Xfatal-warnings sooner than later.

I also had to replace uses of IO.classLocationFile with
IO.classLocationPath to avoid compilation failures due to
-Xfatal-warnings.
2018-10-08 13:59:34 -07:00
Eugene Yokota a9fd111143 Include both explicit plugin resolves and boot resolvers
Fixes #4408
2018-10-07 00:15:58 -04:00
eugene yokota 4867e87eca
Merge pull request #4396 from eed3si9n/wip/progress
"super shell" for sbt
2018-10-06 13:07:11 -04:00
Eugene Yokota c64166ea8d check PluginCross.scala consisntency 2018-10-05 13:32:40 -04:00
Eugene Yokota ca7c7d3841 Fix resolver for compiler bridge
I noticed that we can't resolve the compiler bridge out of snapshot repo.
2018-10-05 04:11:08 -04:00
Eugene Yokota 6dc27a3f6e change to DeleteLine 2018-10-02 08:25:28 -04:00
Eugene Yokota 1465604f8c Demote "Done updating." log to debug level 2018-10-02 08:25:28 -04:00
Eugene Yokota c316c80093 adds task progress
Fixes #4362

This implements an instance of ExecuteProgress that is enabled by default to report progress on the shell.
Combined with the scroll up logger, this takes over the bottom lines of the terminal screen and display a count up clock of the currently executing task.
2018-10-02 08:25:28 -04:00
eugene yokota e2923541b1
Merge pull request #4384 from andreaTP/lspCancellationSupport
Implementing cancellation requests for LSP server
2018-10-01 15:00:43 -04:00
andrea d7c55a3d82 Implementing cancellation requests for LSP server 2018-10-01 14:39:26 +01:00
Dale Wijnand b174a13b9c
Fix scaladoc for DslAggregate 2018-09-28 10:41:08 +01:00
eugene yokota 1fc0cd0314
Merge pull request #4377 from eed3si9n/wip/openjdk11
Scala 2.12.7 and linesIterator
2018-09-28 02:56:37 -04:00
Eugene Yokota 2389106aa6 Use linesIterator for JDK 11 compat 2018-09-27 12:41:47 -04:00
andrea 15d11f8f55 Fix: #4370 2018-09-25 12:57:09 +01:00
eugene yokota 550c068b98
Merge pull request #4369 from eed3si9n/wip/task-sequential
Implement sequential[B](tasks: Seq[Initialize[Task[B]]])
2018-09-19 23:19:16 -04:00
Yusuke Izawa b47ca1b9ef Implement sequential[B](tasks: Seq[Initialize[Task[B]]]) and remove useless comment outs 2018-09-19 20:54:18 -04:00
kai-chi 3f1ce8b9a0 expand tilde prefix 2018-09-19 18:35:59 +02:00
Eugene Yokota 3e1dac5161 -Xfatal-warnings in most subprojects 2018-09-18 11:47:55 -04:00
Eugene Yokota f2bc786fd1 cleaning up a few compiler warnings 2018-09-17 22:30:14 -04:00
eugene yokota 04a4456dd0
Merge pull request #4313 from raboof/supportJdk11AndJavaHomeInCrossJava
Support discovering JDK 11 in JAVA_HOME for CrossJava
2018-09-17 12:34:31 -04:00
eugene yokota 20573e8631
Merge pull request #4341 from Falmarri/feature/show-output-4184
Add `print` command that mimics `show`, but to standard output, not the logger
2018-09-17 12:33:25 -04:00
eugene yokota d3b79ae2e1
Merge pull request #4356 from eed3si9n/wip/remove-global-dir-warning
Remove old warning about global dir
2018-09-17 12:33:06 -04:00
eugene yokota 3f1ae8bd55
Merge pull request #4355 from eed3si9n/wip/one-letter
Allow single-letter aliases
2018-09-16 02:58:45 -04:00
eugene yokota 6b6c52b2d8
Merge pull request #4306 from sirocchj/fix-4236
Fixes #4236
2018-09-16 02:57:09 -04:00
Eugene Yokota f2f1d06329 Remove old warning about global dir
Fixes #1054
2018-09-16 01:55:18 -04:00
Eugene Yokota a2ef775edc Add scaladoc for Cross 2018-09-16 01:51:53 -04:00
falmarri cc143db6f7 Make `show` output on stdout rather than to the logger
Fixes #4184
2018-09-15 18:35:04 -07:00