Commit Graph

312 Commits

Author SHA1 Message Date
Ethan Atkins f126206231 Fix incremental task evaluation semantics
While writing documentation for the new file management/incremental
task evaluation features, I realized that incremental task evaluation
did not have the correct semantics. The problem was that calls to
`.previous` are not scoped within the current task. By this, I mean that
say there are tasks foo and bar and that the defintion of bar looks like

bar := {
    val current = foo.value
    foo.previous match {
        case Some(v) if v == current => // value hasn't changed
        case _ => process(current)
    }
}

The problem is that foo.previous is stored in
effectively (foo / streams).value.cacheDirectory / "previous". This
means that it is completely decoupled from foo. Now, suppose that the
user runs something like:
> set foo := 1
> bar // processes the value 1
> set foo := 2
> foo
> bar // does not process the new value 2 because foo was called, which updates the previous value

This is not an unrealistic scenario and is, in fact, common if the
incremental task evaluation is changed across multiple processing steps.
For example, in the make-clone scripted test, the linkLib task processes
the outputs of the compileLib task. If compileLib is invoked separately
from linkLib, then when we next call linkLib, it might not do anything
even if there was recompilation of objects because the objects hadn't
changed since the last time we called compileLib.

To fix this, I generalizedthe previous cache so that it can be keyed on
two tasks, one is the task whose value is being stored (foo in the
example above) and the other is the task in which the stored task value
is retrieved (bar in the example above). When the two tasks are the
same, the behavior is the same as before.

Currently the previous value for foo might be stored somewhere like:

base_directory/target/streams/_global/_global/foo/previous

Now, if foo is stored with respect to bar, it might be stored in

base_directory/target/streams/_global/_global/bar/previous-dependencies/_global/_gloal/foo/previous

By storing the files this way, it is easy to remove all of the previous
values for the dependencies of a task.

In addition to changing how the files are stored on disk, we have to store
the references in memory differently. A given task can now have multiple
previous references (if, say, two tasks bar and baz both depend on the
previous value). When we complete the results, we first have to collect
all of the successful tasks. Then for each successful task, we find all
of its references. For each of the references, we only complete the
value if the scope in which the task value is used is successful.

In the actual implemenation in Previous.scala, there are a number places
where we have to cast to ScopedKey[Task[Any]]. This is due to
limitations of ScopedKey and Task being type invariant. These casts are
all safe because we never try to get the value of anything, we only use
the portion of the apis of these types that are independent of the value
type. Structural typing where ScopedKey[Task[_]] gets inferred to
ScopedKey[Task[x]] forSome x is a big part of why we have problems with
type inference.
2019-08-09 12:18:22 -07:00
Ethan Atkins a3ac4c76a6 Bump scalafmt
Intellij had issues resolving 2.0.0-RCX so it will be nice to be using
the latest.
2019-07-18 12:40:21 -07:00
Ethan Atkins cad89d17a9 Add support for in memory cache store
It can be quite slow to read and parse a large json file. Often, we are
reading and writing the same file over and over even though it isn't
actually changing. This is particularly noticeable with the
UpdateReport*. To speed this up, I introduce a global cache that can be
used to read values from a CacheStore. When using the cache, I've seen
the time for the update task drop from about 200ms to about 1ms. This
ends up being a 400ms time savings for test because update is called for
both Compile / compile and Test / compile.

The way that this works is that I add a new abstraction
CacheStoreFactoryFactory, which is the most enterprise java thing I've
ever written. We store a CacheStoreFactoryFactory in the sbt State.
When we make Streams for the task, we make the Stream's
cacheStoreFactory field using the CacheStoreFactoryFactory. The
generated CacheStoreFactory may or may not refer to a global cache.

The CacheStoreFactoryFactory may produce CacheStoreFactory instances
that delegate to a Caffeine cache with a max size parameter that is
specified in bytes by the fileCacheSize setting (which can also be set
with -Dsbt.file.cache.size). The size of the cache entry is estimated by
the size of the contents on disk. Since we are generally storing things
in the cache that are serialized as json, I figure that this should be a
reasonable estimate. I set the default max cache size to 128MB, which is
plenty of space for the previous cache entries for most projects. If the
size is set to 0, the CacheStoreFactoryFactory generates a regular
DirectoryStoreFactory.

To ensure that the cache accurately reflects the disk state of the
previous cache (or other cache's using a CacheStore), the Caffeine cache
stores the last modified time of the file whose contents it should
represent. If there is a discrepancy in the last modified times (which
would happen if, say, clean has been run), then the value is read from
disk even if the value hasn't changed.

* With the following build.sbt file, it takes roughly 200ms to read and
parse the update report on my compute:

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.3"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1"

This is because spark-sql has an enormous number of dependencies and the
update report ends up being 3MB.
2019-07-11 17:45:16 -07:00
Dale Wijnand e86a63ca1b
Drop the remaining TupleSyntax usage 2019-05-29 14:43:18 +01:00
Ethan Atkins 81ce14d58c Allow calling TaskKey.previous in input tasks
I discovered that it wasn't possible to call .previous in an input task.
While I understand why you can't call .previous on an InputKey, I think
it makes sense to allow calling .previous on a TaskKey within an input
task.
2019-04-25 15:58:01 -07:00
Eugene Yokota 1e157b991a apply formatting 2019-04-20 03:23:54 -04:00
Ethan Atkins 9cdeb7120e Add StateTransform class
This commit cleans up the approach for transforming the sbt state upon
completion of a task returning State. I add a new approach where a task
can return an instance of StateTransform, which is just a wrapper around
State. I then update EvaluateTask to apply this stateTransform rather
than the (optional) state transformation that may be stored in the Task
info parameter. By requiring that the user return StateTransform rather
than State directly, we ensure that no existing tasks that depend on the
state transformation function embedded in the Task info break. In sbt 2,
I could see the possibility of making this automatic (and probably
removing the state transformation function via attribute).

The problem with using the transformState attribute key is that it is
applied non-deterministically. This means that if you decorate a task
returning State, then the state transformation may or may not be
correctly applied.

I tracked this non-determinism down to the stateTransform
method in EvaluateTask. It iterates through the task result map and
chains all of the defined transformState attribute values. Because the
result is a map, this order is not specified. This chaining is arguably
a bad design because State => State does not imply commutivity. Indeed,
the problem here was that my state transformation functions were
constant functions, which are obviously non-commutative. I believe that
this logic likely written under the assumption that there would be no
more than one of these tranformations in a given result map.
2019-03-30 16:39:10 -07:00
Ethan Atkins e868c43fcc Refactor Watched
This is a huge refactor of Watched. I produced this through multiple
rewrite iterations and it was too difficult to separate all of the
changes into small individual commits so I, unfortunately, had to make a
massive commit. In general, I have tried to document the source code
extensively both to facilitate reading this commit and to help with
future maintenance.

These changes are quite complicated because they provided a built-in
like api to a feature that is implemented like a plugin. In particular,
we have to manually do a lot of parsing as well as roll our own
task/setting evaluation because we cannot infer the watch settings at
project build time because we do not know a priori what commands the
user may watch in a given session. The dynamic setting and task
evaluation is mostly confined to the WatchSettings class in Continuous.
It feels dirty to do all of this extraction by hand, but it does seem to
work correctly with scopes.

At a high level this commit does four things:
1) migrate the watch implementation to using the InputGraph to collect
   the globs that it needs to monitor during the watch
2) simplify WatchConfig to make it easier for plugin authors to write
   their own custom watch implementations
3) allow configuration of the watch settings based on the task(s) that
   is/are being run
4) adds an InputTask implemenation of watch.

Point #1 is mostly handled by Point #3 since I had to overhaul how _all_
of the watch settings are generated. InputGraph already handles both
transitive inputs and triggers as well as legacy watchSources so not
much additional logic is needed beyond passing the correct scoped keys
into InputGraph.

Point #3 require some structural changes. The watch settings cannot in
general be defined statically because we don't know a priori what tasks
the user will try and watch. To address this, I added code that will
extract the task keys for all of the commands that we are running. I
then manually extract the relevant settings for each command. Finally, I
aggregate those settings into a single WatchConfig that can be used to
actually implement the watch. The aggregation is generally
straightforward: we run all of the callbacks for each task and choose
the next watch state based on the highest priority Action that is
returned by any of the callbacks.

Because I needed Extracted to pull out the necessary settings, I was
forced to move a lot of logic out of Watched and into a new singleton,
Continuous, that exists in the main project (Watched is in the command
project). The public footprint of Continuous is tiny. Even though I want
to make the watch feature flexible for plugin authors, the
implementation and api remain a moving target so I do not want to be
limited by future binary compatibility requirements. Anyone who wants to
live dangerously can access the private[sbt] apis via reflection or by
adding custom code to the sbt package in their plugin (a technique I've
used in CloseWatch).

Point #2 is addressed by removing the count and lastStatus from the
WatchConfig callbacks. While these parameters can be useful, they are
not necessary to implement the semantics of a watch. Moreover, a status
boolean isn't really that useful and the sbt task engine makes it very
difficult to actually extract the previous result of the tasks that were
run. After this refactor, WatchConfig has a simpler api. There are fewer
callbacks to implement and the signatures are simpler. To preserve the
_functionality_ of making the count accessible to the user specifiable
callbacks, I still provided settings like watchOnInputEvent that accept
a count parameter, but the count is actually tracked externally to
Watched.watch and incremented every time the task is run.

Moreover, there are a few parameters of the watch: the logger and
transitive globs, that cannot be provided via settings. I provide
callback settings like watchOnStart that mirror the WatchConfig
callbacks except that they return a function from Continuous.Arguments
to the needed callback. The Continuous.aggregate function will check if
the watchOnStart setting is set and if it is, will pass in the needed
arguments. Otherwise it will use the default watchOnStart implementation
which simulates the existing behavior by tracking the iteration count in
an AtomicInteger and passing the current count into the user provided
callback. In this way, we are able to provide a number of apis to the
watch process while preserving the default behavior.

To implement #4, I had to change the label of the `watch` attribute key
from "watch" to "watched". This allows `watch compile` to work at the
sbt command line even thought it maps to the watchTasks key. The actual
implementation is almost trivial. The difference between an
InputTask[Unit] and a command is very small. The tricky part is that the
actual implementation requires applying mapTask to a delegate task that
overrides the Task's info.postTransform value (which is used to
transform the state after task evaluation). The actual postTransform
function can be shared by the continuous task and continuous command.
There is just a slightly different mechanism for getting to the state
transformation function.
2019-03-30 16:38:56 -07:00
Ethan Atkins ed06e18fab Add InputGraph
This commit adds functionality to traverse the settings graph to find
all of the Inputs settings values for the transitive dependencies of the
task. We can use this to build up the list of globs that we must watch
when we are in a continuous build. Because the Inputs key is a setting,
it is actually quite fast to fetch all the values once the compiled map
is generated (O(2ms) in the scripted tests, though I did find that it
took O(20ms) to generate the compiled map).

One complicating factor is that dynamic tasks do not track any of
their dynamic dependencies. To work around this, I added the
transitiveDependencies key. If one does something like:
foo := {
  val _ = bar / transitiveDependencies
  val _ = baz / transitiveDependencies
  if (System.getProperty("some.prop", "false") == "true") Def.task(bar.value)
  else Def.task(baz.value)
}
then (foo / transitiveDependencies).value will return all of the inputs
and triggers for bar and baz as well as for foo.

To implement transitiveDependencies, I did something fairly similar to
streams where if the setting is referenced, I add a default
implementation. If the default implementation is not present, I fall
back on trying to extract the key from the commandLine. This allows the
user to run `show bar / transitiveDependencies` from the command line
even if `bar / transitiveDependencies` is not defined in the project.

It might be possible to coax transitiveDependencies into a setting, but
then it would have to be eagerly evaluated at project definition time
which might increase start up time too much.  Alternatively, we could
just define this task for every task in the build, but I'm not sure how
expensive that would be. At any rate, it should be straightforward to
make that change without breaking binary compatibility if need be. This
is something to possibly explore before the 1.3 release if there is any
spare time (unlikely).
2019-03-30 16:38:44 -07:00
Eugene Yokota adfb2ece1a install official sbt based on project/build.properties 2019-03-25 02:10:48 -04:00
Ethan Atkins 571b179574 Add dsl for collecting globs
Right now, the sbt.internal.io.Source is something of a second class
citizen within sbt. Since sbt 0.13, there have been extension classes
defined that can convert a file to a PathFinder but no analog has been
introduced for sbt.internal.io.Source.

Given that sbt.internal.io.Source was not really intended to be part of
the public api (just look at its package), I think it makes sense to
just replace it with Glob. In this commit, I add extension
methods to Glob and Seq[Glob] that make it possible to easily
retrieve all of the files for a particular Glob within a task. The
upshot is that where previously, we'd have had to write something like:

watchSources += Source(baseDirectory.value / "src" / "main" / "proto", "*.proto", NothingFilter)

now we can write

watchGlobs += baseDirectory.value / "src" / "main" / "proto" * "*.proto"

Moreover, within a task, we can now do something like:
foo := {
  val allWatchGlobs: Seq[File] = watchGlobs.value.all
  println(allWatchSources.mkString("all watch source files:\n", "\n", ""))
}
Before we would have had to manually retrieve the files.

The implementation of the dsl uses the new GlobExtractor class which
proxies file look ups through a FileTree.Repository. This makes it so
that, by default, all file i/o using Sources will use the default
FileTree.Repository. The default is a macro that returns
`sbt.Keys.fileTreeRepository.value: @sbtUnchecked`. By doing it this
way, the default repository can only be used within a task definition
(since it delegates to `fileTreeRepository.value`). It does not,
however, prevent the user from explicitly providing a
FileTree.Repository instance which the user is free to instantiate
however they wish.

Bonus: optimize imports in Def.scala and Defaults.scala
2019-03-22 07:53:41 -07:00
Ethan Atkins 82cfbe83a7 Fix doc warning
The scaladoc task was warning:
'Could not find any member to link for "LinterLevelLowPriority"'. Given
that LinterLevelLowPriority is a package private trait, it shouldn't be
mentioned by name.
2019-01-31 17:04:40 -08:00
Ethan Atkins df6f3bd888 Switch to regular comment instead of scaladoc
Having this as scaladoc adds an unmoored doc comment compiler warning
because no documentation is generated for the class that is being
documented.
2019-01-31 17:04:40 -08:00
Dale Wijnand 32b342e9a8
Mention SAM conversions 2019-01-29 15:47:50 +00:00
Dale Wijnand 53c6299c94
Implement Append for Function1 2019-01-29 09:12:11 +00:00
Dale Wijnand 14bffefef9
Cleanup Append 2019-01-27 16:20:59 +00:00
Ethan Atkins 01b2e86a54 Fix Def cannot be used inside a taskDyn #3110
The illegalReference check did not actually validate whether the illegal
reference actually referred to an M[_] (which is pretty much always
Initialize[_]]). The means by which this failure was induces were fairly
obscure and go through multiple levels of macro transformations that I
attempt to explain in the comment in IllegalReferenceSpec.

Fixes #3110
2018-12-11 21:49:42 -08:00
Ethan Atkins d42b3ee2fd Make the TaskLinterDSL warn by default
I am generally of the opinion that a linter should not abort progress by
default. I do, however, think that it should be on by default, making
warn a happy compromise.
2018-12-11 18:21:46 -08:00
Ethan Atkins 32792f2b9d Make the task linter configurable
The user should be able to configure whether or not the task linting is
strictly enforced. In my opinion, the linter is generally pretty good
about warning users that a particular definition may not behave the way
the user expects. Unfortunately, it is fairly common that the user
explicitly wants this behavior and making the linter abort compilation
is a frustrating user experience.

To fix this, I add the LinterLevel trait to a new sbt.dsl package in the
core-macros project. The user can configure the linter to:
1) abort when an issue is detected. This is the current default behavior.
2) print a warning when an issues is detected
3) skip linting all together

Linter configuration is managed by importing the corresponding implicit
object from the LinterLevel companion object.

Fixes #3266
2018-12-11 18:21:44 -08:00
Ethan Atkins 2e027640d3 Add solutions for dynamic task evaluation
There are many cases where one would want to force evaluation of the
task even when contained in a lambda (see
https://github.com/sbt/sbt/issues/3266). The @sbtUnchecked annotation is
one way to disable the linter that prevents this, but it is obscure. If
the annotation is to exist, I think it should be presented as a
solution.
2018-12-11 12:45:49 -08:00
Ethan Atkins 1c2bab093b Switch to more functional style in BaseTaskLinterDSL
I found it somewhat difficult to reason about the state of the tree
Traverser because of the usage of mutable variables and data structures.
For example, I determined that the uncheckedWrappers were never used
non-locally so a Set didn't seem to be the right data structure. It was
reasonably straightforward to switch to a more functional style by
parameterizing the method local traverser class.

Bonus:
  - remove unused variable disableNoValueReport
  - add scaladoc to document the input parameters of the traverser class
    so that it's a bit easier to understand for future maintainers.
2018-12-11 12:45:49 -08:00
Ethan Atkins ab2df045ab Lint TaskLinterDSL for intellij
This cleans up all of the yellow intellij warnings for me. It still
complains about some typos, but those manifest as green squiggled lines
which are less annoying.
2018-12-11 12:45:49 -08:00
Eugene Yokota ca7c7d3841 Fix resolver for compiler bridge
I noticed that we can't resolve the compiler bridge out of snapshot repo.
2018-10-05 04:11:08 -04:00
Eugene Yokota 2389106aa6 Use linesIterator for JDK 11 compat 2018-09-27 12:41:47 -04:00
Eugene Yokota ef49a95b7d address more warnings 2018-09-18 17:45:24 -04:00
Eugene Yokota 4ff4f6e45e Update header 2018-09-14 04:53:36 -04:00
xuwei-k ae1fdff968 use SAM type 2018-07-09 13:06:34 +09:00
Eugene Yokota 14a31634e7 Code formatting 2018-06-27 21:06:40 -04:00
eugene yokota eb942f8e5f
Merge pull request #4003 from eed3si9n/wip/opt-delegation3
optimize scope delegation
2018-06-27 15:31:21 -04:00
Eugene Yokota 28e90ea09b Merge branch '1.1.x' into wip/merge-1.1.x 2018-04-29 14:31:30 -04:00
eugene yokota 851ce6ed83
Merge pull request #4090 from eed3si9n/wip/unused-task
Fixes linter that detects missing .value
2018-04-26 12:25:09 -04:00
Eugene Yokota 06d7be0365 handle X / y 2018-04-26 04:44:12 -04:00
Dale Wijnand 8f4b8abb7b
Run scalafmt & test:scalafmt 2018-04-24 16:12:10 +01:00
Dale Wijnand dfff1ed928
Merge pull request #4032 from eed3si9n/wip/servertest
improve server testing
2018-04-18 07:12:53 +02:00
Eugene Yokota 190dc23f70 Fixes linter that detects missing .value
Fixes #4079

#3216 introduced a linter that checks against missing `.value`, but the tree only checked for `Ident`. This doesn't work because in reality the symbols of build.sbt are transformed to `$somehash.npmInstallTask` where `somehash` is the wrapper object we create. Similarly for the built-in keys, they are presented as `sbt.Keys.compile`.

With this change unused task will fail to load the build with the following message:

```
/sbt-4079/build.sbt:26: error: The key `compile` is not being invoked inside the task definition.

Problem: Keys missing `.value` are not initialized and their dependency is not registered.

Solution: Replace `compile` by `compile.value` or remove it if unused.

  compile
  ^
/sbt-4079/build.sbt:27: error: The key `npmInstallTask` is not being invoked inside the task definition.

Problem: Keys missing `.value` are not initialized and their dependency is not registered.

Solution: Replace `npmInstallTask` by `npmInstallTask.value` or remove it if unused.

  npmInstallTask
  ^
```
2018-04-11 01:31:38 -04:00
Dale Wijnand 54a6262f7f
Merge pull request #4041 from dwijnand/Scoped.equals
Give Scoped a default equals/hashCode implementation
2018-04-10 06:10:55 +01:00
Dale Wijnand 3692db9068
Give Scoped a default equals/hashCode implementation 2018-04-04 15:04:53 +01:00
Dale Wijnand 8c1337455d
Fix migrate URL
Fixes #4062
2018-04-04 14:26:03 +01:00
Dale Wijnand af3eec7c31
Merge pull request #4045 from dwijnand/Deprecate-Scope.transformTaskName
Deprecate Scope.transformTaskName
2018-03-28 16:09:39 +01:00
eugene yokota 02acf9380e
Merge pull request #4042 from dwijnand/Reuse-Scoped.scoped-methods
Re-use Scoped.scoped* methods
2018-03-28 09:41:09 -04:00
Dale Wijnand 2e86eed151
Split BuildSettingsInstances out of SlashSyntaxSpec.scala 2018-03-27 16:32:34 +01:00
Dale Wijnand 165aac858e
Merge pull request #4040 from dwijnand/dedup/SlashSyntaxSpec
Dedup keys in SlashSyntaxSpec
2018-03-27 14:54:59 +01:00
Dale Wijnand 35decb6ee5
Deprecate Scope.transformTaskName 2018-03-27 09:59:07 +01:00
Dale Wijnand d19329666a
Re-use Scoped.scoped* methods 2018-03-27 09:24:12 +01:00
Dale Wijnand 3c66f39744
Allow SlashSyntaxSpec to be Scalafmt formatted
Previously:

    [warn] /d/sbt/main-settings/src/test/scala/sbt/SlashSyntaxSpec.scala:44: error: illegal start of simple expression
    [warn] )
    [warn]     ^
2018-03-27 09:22:07 +01:00
Dale Wijnand 42b61a4f8e
Dedup keys in SlashSyntaxSpec 2018-03-27 09:22:06 +01:00
Dale Wijnand 51ff72872b
De-generalise expectValue in SlashSyntaxSpec 2018-03-27 09:22:06 +01:00
Dale Wijnand 006527d246
Re-order SlashSyntaxSpec arbitraries 2018-03-27 09:22:06 +01:00
Eugene Yokota 67e1e4a9ff improve server testing
previously I was using separate thread in a forked test to test the server, but that is not enough isolation to run multiple server tests.
This adds `RunFromSourceMain.fork(workingDirectory: File)`, which allows us to run a fresh sbt on the given working directory. Next, I've refactored the stateful client-side buffer to a class `TestServer`.
2018-03-24 12:09:41 -04:00
Eugene Yokota d4cdb11b53 typo fix 2018-03-13 17:47:11 +09:00
Eugene Yokota d2f2a90d5e handroll for-expression
Ref https://github.com/sbt/sbt/pull/3979

Based on #3979 this handrolls the for-express using while.
2018-03-10 18:43:43 -05:00
Dale Wijnand 10275489d8
Merge pull request #3987 from dwijnand/document-Rich-Init-classes
Document RichInitX classes
2018-03-07 09:44:19 +00:00
Dale Wijnand 621ad2b553
Cleanup TaskNegSpec 2018-03-06 11:59:27 +00:00
Dale Wijnand d26085155d
Cleanup InputWrapper imports 2018-03-06 11:54:11 +00:00
Dale Wijnand 77abb9ee29
Cleanup InputConvert 2018-03-06 11:54:10 +00:00
Dale Wijnand 3921c45563
Document RichInitX classes 2018-03-06 10:37:42 +00:00
Dale Wijnand 90cd60f3b9
Merge branch '1.1.x' into merge-1.1.x-into-1.x
* 1.1.x:
  Use Java's redirectInput rather than sys.process's connectInput
  Re-write toolboxClasspath to use sbt-buildinfo
  Cleanup generateToolboxClasspath
  Upgrade to sbt-buildinfo 0.8.0
  Fix how fullClasspath is defined in TestBuildInfo
  delete buildinfo.BuildInfo from sbt main

Conflicts:
	project/plugins.sbt
2018-02-28 10:30:53 +00:00
Dale Wijnand 27fe5a6957
Re-write toolboxClasspath to use sbt-buildinfo 2018-02-23 13:20:19 +00:00
Colin Dean 3e11b3f000 Fixes link to documentation for deprecated 0.10/0.12 DSL syntax
Fixes sbt/website#558
2018-01-24 23:13:00 -05:00
Dale Wijnand 113656aed1
Remove compile warnings 2018-01-16 11:17:01 +00:00
Dale Wijnand a90832b593
Remove all warnings from mainProj 2017-12-14 15:40:03 +00:00
Dale Wijnand 2390fdfac6
Remove all warnings from mainSettingsProj 2017-12-14 13:16:23 +00:00
Dale Wijnand 9f1d60be60
Rewrite to polymorphic function syntax 2017-10-25 10:23:46 +01:00
Dale Wijnand f662fdda8e
Rewrite to function syntax 2017-10-25 10:22:48 +01:00
Simon Schäfer 93e08b7ee7 Fix warnings about unused pattern vars in various projects 2017-10-19 13:07:24 +02:00
Dale Wijnand 668ace98ee
Remove some duplication when only Scoped is required 2017-10-08 19:07:38 +01:00
Dale Wijnand 5e0b080f51
Fix some minor errors 2017-10-07 18:42:26 +01:00
Dale Wijnand ef4828cfc2
Add slash syntax for AttrKey that parity with "in" syntax 2017-10-07 13:12:17 +01:00
Dale Wijnand d19c350687
Correct name of WithoutScope properties 2017-10-07 13:12:16 +01:00
Dale Wijnand 8af3110c00
Add .previous tests to SlashSyntaxTest 2017-10-07 13:12:16 +01:00
Dale Wijnand fb28c201c8
Make scoped keys more frequent 2017-10-07 13:12:16 +01:00
Dale Wijnand 74ac7d9e07
Drop redudant label in expectValue 2017-10-07 13:12:15 +01:00
Dale Wijnand 025075efd0
Dedup, cleanup & cover all key types 2017-10-07 13:12:15 +01:00
Dale Wijnand 165dc794ca
Extract gen{Input,Setting,Task}Key 2017-10-07 13:12:15 +01:00
Dale Wijnand 18d615701f
Remove WithScope wrapping 2017-10-07 13:12:15 +01:00
Dale Wijnand 0bcd7c3b6d
Remove boilerplate around sbtSlashSyntaxRichConfiguration 2017-10-07 12:08:32 +01:00
Dale Wijnand e5898111fe
Replace CanScope with Scoped.ScopingSetting 2017-10-07 12:07:19 +01:00
Eugene Yokota 2a6385fd94 Remove thunk for slash syntax
Ref #3606, #3611, and #3613

This removes unnecessary thunk for slash syntax.
The semantics using this approach is strictly better than the previous `in (ref, config, task)`. By removing the thunk, we retain `(a / b) / c == a / b / c`.

See the following example:

```scala
scala> import sbt._, Keys._
scala> val t: TaskKey[Unit] = (test in Test)
t: sbt.TaskKey[Unit] = TaskKey(This / Select(ConfigKey(test)) / This / test)

scala> ThisBuild / t
ThisBuild / t
res1: sbt.TaskKey[Unit] = TaskKey(Select(ThisBuild) / Select(ConfigKey(test)) / This / test)

scala> ThisBuild / t / name
ThisBuild / t / name
res2: sbt.SettingKey[String] = SettingKey(Select(ThisBuild) / Select(ConfigKey(test)) / Select(test) / name)
```

so far so good? Now look at this:

```
scala> scala> name in (ThisBuild, t)
name in (ThisBuild, t)
res3: sbt.SettingKey[String] = SettingKey(Select(ThisBuild) / This / Select(test) / name)
```

`Test` configuration knowledge is lost! For `in (..)` maybe it was ok because mostly we don't use unscoped keys, but that's the difference between `in (..)` and `/`.

Fixes #3605
2017-10-06 15:47:46 -04:00
Dale Wijnand 5b03379693
Extract withScope 2017-10-06 17:41:06 +01:00
Dale Wijnand 9b526f54bf
Rarely include custom scopes in key generators 2017-10-06 15:41:28 +01:00
Dale Wijnand db87e4c871
Add SlashSyntaxSpec to validate syntax parity 2017-10-06 15:07:18 +01:00
Dale Wijnand 97ddc1ffb7
Copy the non-runtime parts of project/unified to SlashSyntaxTest 2017-10-06 15:07:17 +01:00
Dale Wijnand f22843f91c
Move SlashSyntax to the settings component 2017-10-06 09:52:34 +01:00
eugene yokota 1251817cf2 Merge pull request #3612 from dwijnand/scope-toString
Handle Global in Scope#toString
2017-10-05 17:43:25 -04:00
Dale Wijnand 2a5ba9475d
Handle Global in Scope#toString
Print "Global" instead of "Zero / Zero / Zero".
2017-10-05 20:29:42 +01:00
Dale Wijnand f4b2fc4228
Correct handling of resolving ThisProject
In ca71b4b902 I went about fixing the
inexhaustive matching in Scope's resolveProjectBuild and
resolveProjectRef. Looking back the change was wrong.

For resolveProjectBuild the new implementation is less wrong, but still
not great, seeing as it doesn't actually do any build resolving.

For resolveProjectRef the new implementation now blows up instead of
lies. Which means it's less leneant, more "fail-fast".

isProjectThis is unused; remnant of the pre-AutoPlugin days when build
settings where defined in Plugin.settings.
2017-10-05 19:00:48 +01:00
Dale Wijnand a41727fb17
Add, configure & enforce file headers 2017-10-05 09:03:40 +01:00
Eugene Yokota 60f2498c0a Implement toString for keys
toString added for REPL testing:

```
scala> Zero / Zero / Zero / name
res0: sbt.SlashSyntax.ScopeAndKey[sbt.SettingKey[String]] = Zero / Zero / Zero / name
```
2017-10-05 02:46:09 -04:00
Eugene Yokota b0306b738e Add whitespaces in Show of scoped keys
```
Provided by:
    ProjectRef(uri("...."), "root") / Test / test
Dependencies:
    Test / executeTests
    Test / test / streams
    Test / state
    Test / test / testResultLogger
```
2017-09-28 03:34:49 -04:00
Eugene Yokota 33a01f3ceb Unified slash syntax
Fixes sbt/sbt#1812

This adds unified slash syntax for both sbt shell and the build.sbt DSL.
Instead of the current `<project-id>/config:intask::key`,
this adds `<project-id>/<config-ident>/intask/key` where <config-ident> is the Scala identifier notation for the configurations like `Compile` and `Test`.

This also adds a series of implicits called `SlashSyntax` that adds `/` operators to project refererences, configuration, and keys such that the same syntax works in build.sbt.

These examples work for both from the shell and in build.sbt.

    Global / cancelable
    ThisBuild / scalaVersion
    Test / test
    root / Compile / compile / scalacOptions
    ProjectRef(uri("file:/xxx/helloworld/"),"root")/Compile/scalacOptions
    Zero / Zero / name

The inspect command now outputs something that can be copy-pasted:

    > inspect compile
    [info] Task: sbt.inc.Analysis
    [info] Description:
    [info] 	Compiles sources.
    [info] Provided by:
    [info] 	ProjectRef(uri("file:/xxx/helloworld/"),"root")/Compile/compile
    [info] Defined at:
    [info] 	(sbt.Defaults) Defaults.scala:326
    [info] Dependencies:
    [info] 	Compile/manipulateBytecode
    [info] 	Compile/incCompileSetup
    [info] Reverse dependencies:
    [info] 	Compile/printWarnings
    [info] 	Compile/products
    [info] 	Compile/discoveredSbtPlugins
    [info] 	Compile/discoveredMainClasses
    [info] Delegates:
    [info] 	Compile/compile
    [info] 	compile
    [info] 	ThisBuild/Compile/compile
    [info] 	ThisBuild/compile
    [info] 	Zero/Compile/compile
    [info] 	Global/compile
    [info] Related:
    [info] 	Test/compile
2017-09-28 01:01:43 -04:00
eugene yokota 45765583a6 Merge branch '1.0.x' into 1.0.x 2017-09-15 19:47:08 -04:00
Răzvan Flavius Panda 0124a8ad0e Fix unused imports warnings 2017-09-15 16:35:08 +01:00
Dale Wijnand 936733b2b1
Cleanup 2017-09-13 16:03:51 +01:00
Dale Wijnand d3d105515a
Don't warn on by-name keys
.. which is apparently what the type of bare vals in *.sbt files are.

Fixes #3299, again.
2017-09-11 14:46:33 +01:00
Dale Wijnand 8d9a3a35d2
Fix warnings in TaskLinterDSL.scala 2017-09-11 14:45:40 +01:00
eugene yokota 33d3ba9d7c Merge pull request #3438 from Duhemm/source-appender
[1.0.x] `Append` instance to add `File` to `Seq[Source]`
2017-08-23 18:31:58 -04:00
Dale Wijnand 4f0d2f9ffd
Scalafmt 1.2.0 2017-08-14 15:47:52 +01:00
Martin Duhem 5815f1db0a
`Append` instance to add `File` to `Seq[Source]` 2017-08-14 15:47:17 +02:00
Dale Wijnand 805b76f3d4
Add back, re-configure & re-enable Scalafmt 2017-08-10 16:35:23 +01:00
Eugene Yokota 467617a4b9 Implement withRank 2017-07-25 01:50:53 -04:00
Martin Duhem 9dfe1a8406
Allow `.value` on SettingKey inside ifs & lambdas
Calling `.value` on a SettingKey doesn't trigger any execution and
doesn't have any side effect, so we can safely allow calls to `.value`
inside conditionals and lambdas.

Fixes #3299
2017-07-21 16:26:45 +02:00
Martin Duhem 347914191a Adapt to new library management API 2017-07-15 18:09:40 -04:00
Martin Duhem 3fe068a4c1 Adapt to changes in ConsoleAppender 2017-07-15 17:15:42 +02:00
Eugene Yokota 42a1e2a291 Improve the default prompt
Fixes #3313

This changes the default prompt to include "sbt:" + current project name.
See https://twitter.com/eed3si9n/status/884914670351659009
2017-07-13 14:58:10 -04:00
Dale Wijnand 0fe0de5fb3
Move the tuple enrichments into sbt.TupleSyntax
This undeprecates the syntax, but at the same times moves it out of
implicit scope, therefore requiring a 'import TupleSyntax._' to opt-in
to the old syntax.
2017-06-27 12:42:46 +01:00
Dale Wijnand c5b21ab9d9
Format Structure 2017-06-26 13:49:18 +01:00
jvican c85fb95851
Remove unnecessary infra for .value DSL check 2017-05-28 18:36:14 +02:00
jvican 6a31251a01
Add tests that check `sbtUnchecked` is respected 2017-05-28 18:31:45 +02:00
jvican b017eaee39
Support DSL detection for nested ifs and anons
Before, we were not preserving the value `insideXXX`. This commit makes
sure that we handle much more complex scenarios and we report them
successfully. Have a look at the tests.
2017-05-28 18:31:44 +02:00
jvican ded281b5c2
Add missing .value DSL check
The missing .value analysis is dumb on purpose because it's expensive.

Detecting valid use cases of idents whose type is an sbt key is
difficult and dangerous because we may miss some corner cases. Instead,
we report on the easiest cases in which we are certain that the user
does not want to have a stale key reference. Those are idents in the rhs
of val definitions with `_` as name and idents in statement position
inside blocks.

In the future, we can cover all val definitions no matter what their
name is. Unfortunately, doing so will come at the cost of speed: we have
to run the unused name analysis in `TypeDiagnostics` and expose it from
the power context in `ContextUtil`.

This is good enough for now. If users express interest in having a
smarter analysis, we can always consider trying the unused name
analysis. I am not sure how slow it will be -- hopefully it won't be
that much.
2017-05-28 18:08:58 +02:00
eugene yokota acf4a5ffc2 Merge pull request #3220 from eed3si9n/fport/3135
[fport] Fix += interference with sbt-buildinfo
2017-05-27 00:15:32 -04:00
Eugene Yokota bfc2d85d54 Fix += interference with sbt-buildinfo
The macro pattern match was too general. This makes it tighter.

Fixes #3132
2017-05-26 21:04:33 -04:00
Eugene Yokota 5dc46e3cf2 fix restligeist macro
Fixes sbt/sbt#3157

Before:

    PgpSettings.scala:99: exception during macro expansion:
    [error] java.lang.RuntimeException: Unexpected macro application tree (class scala.reflect.internal.Trees$Apply): PgpKeys.pgpStaticContext.<<=(sbt.this.Scoped.t2ToApp2[sbt.File, sbt.File](scala.Tuple2.apply[sbt.SettingKey[sbt.File], sbt.SettingKey[sbt.File]](PgpKeys.pgpPublicRing, PgpKeys.pgpSecretRing)).apply[com.jsuereth.pgp.cli.PgpStaticContext]({
    [error]   ((publicKeyRingFile: sbt.File, secretKeyRingFile: sbt.File) => SbtPgpStaticContext.apply(publicKeyRingFile, secretKeyRingFile))

After:

    build.sbt:18: error: `<<=` operator is removed. Use `key := { x.value }` or `key ~= (old => { newValue })`.
    See http://www.scala-sbt.org/1.0/docs/Migrating-from-sbt-012x.html
        publishLocal <<= foo // publishLocal.dependsOn(foo)
                     ^
    [error] sbt.compiler.EvalException: Type error in expression
2017-05-26 01:03:26 -04:00
jvican bd59f64727
Rename `sbt.unchecked` to `sbt.sbtUnchecked`
As per Eugene's suggestion.
2017-05-25 19:05:27 +02:00
jvican ca3acc4e52
Improve if check and prohibit value inside anon
This commit does the following things:

* Removes the boolean from the instance context passes to the linter.
* Prohibits the use of value inside anonymous functions.
* Improves the previous check of `value` inside if.

The improvements have occurred thanks to the fix of an oversight in the
traverser. As a result, several implementation of tasks have been
rewritten because of new compilation failures by both checks.

Note that the new check that prohibits the use of value inside anonymous
functions ignores all the functions whose parameters have been
synthesized by scalac (that can happen in a number of different
scenarios, like for comprehensions). Other scripted tests have also been
fixed.

Running `.value` inside an anonymous function yields the following
error:

```
[error] /data/rw/code/scala/sbt/main-settings/src/test/scala/sbt/std/TaskPosSpec.scala:50:24: The evaluation of `foo` inside an anonymous function is prohibited.
[error]
[error] Problem: Task invocations inside anonymous functions are evaluated independently of whether the anonymous function is invoked or not.
[error]
[error] Solution:
[error]   1. Make `foo` evaluation explicit outside of the function body if you don't care about its evaluation.
[error]   2. Use a dynamic task to evaluate `foo` and pass that value as a parameter to an anonymous function.
[error]
[error]       val anon = () => foo.value + " "
[error]                        ^
```
2017-05-25 17:21:29 +02:00
jvican 41dce9e568
Make sure that macro linter doesn't fail spuriously 2017-05-25 15:33:06 +02:00
jvican 8de2bfe461
Fix error message 2017-05-25 15:33:06 +02:00
jvican 2b12721a68
Add fully-fledged macro check for value inside if
`.value` inside the if of a regular task is unsafe. The wrapping task
will always execute the value, no matter what the if predicate yields.

This commit adds the infrastructure to lint code for every sbt DSL
macro. It also adds example of neg tests that check that the DSL checks
are in place.

The sbt checks yield error for this specific case because we may want to
explore changing this behaviour in the future. The solutions to this are
straightforward and explained in the error message, that looks like
this:

```
EXPECTED: The evaluation of `fooNeg` happens always inside a regular task.

PROBLEM: `fooNeg` is inside the if expression of a regular task.
  Regular tasks always evaluate task inside the bodies of if expressions.

SOLUTION:
  1. If you only want to evaluate it when the if predicate is true, use a dynamic task.
  2. Otherwise, make the static evaluation explicit by evaluating `fooNeg` outside the if expression.
```

Aside from those solutions, this commit also adds a way to disable any
DSL check by using the new `sbt.unchecked` annotation. This annotation,
similar to `scala.annotation.unchecked` disables compiler output. In our
case, it will disable any task dsl check, making it silent.

Examples of positive checks have also been added.

There have been only two places in `Defaults.scala` where this check has
made compilation fail.

The first one is inside `allDependencies`. To ensure that we still have
static dependencies for `allDependencies`, I have hoisted up the value
invocation outside the if expression. We may want to explore adding a
dynamic task in the future, though. We are doing unnecessary work there.

The second one is inside `update` and is not important because it's not
exposed to the user. We use a `taskDyn`.
2017-05-25 15:33:00 +02:00
jvican b4299e7f34
Add check of task invocation inside `if`
This commit adds the first version of the checker that will tell users
if they are doing something wrong.

The first version warns any user that write `.value` inside an if
expression.

As the test integration is not yet working correctly and messages are
swallowed, we have to println to get info from the test.
2017-05-24 17:27:18 +02:00
jvican 5b7180cfa7
Reorganize test structure for `mainSettings` 2017-05-24 11:27:04 +02:00
jvican 03daae49e2
Add test util to check neg macro tests 2017-05-24 10:21:52 +02:00
Eugene Yokota 2082f37e2a Rename Global as scope component to Zero
Fixes #3178

While working on the Scopes and Scope Delegation document, I noticed that the term Global in sbt is used for two different meaning.

1. Universal fallback scope component `*`
2. An alias for GlobalScope

This disambiguates the two by renaming ScopeAxis instance to Zero.
Since this is mostly internal to sbt code, the impact to the user should be minimal.
2017-05-08 11:41:48 -04:00
Eugene Yokota da046791e0 Apply Scalafmt formatting 2017-04-21 04:48:31 -04:00
Eugene Yokota 1ec7e8d975 format: off / format: on 2017-04-21 03:11:48 -04:00
Eugene Yokota 9e02995ac0 Bump to Zinc 1.0.0-X14 2017-04-18 00:56:22 -04:00
Dale Wijnand 91354497ab
Drop deprecated InputTask apply method 2017-04-11 16:44:53 +01:00
Dale Wijnand 0d32f6fc76
Drop File/Seq[File] setting enrichments 2017-04-11 16:42:43 +01:00
Dale Wijnand 4a08b65cb2
Drop ProjectReference implicit lifts 2017-04-11 16:40:38 +01:00
Dale Wijnand a6015793e2
Code formatting only changes 2017-04-11 14:32:38 +01:00
Dale Wijnand be3fa7e66d
Just whitespace & comment changes 2017-04-11 11:48:24 +01:00
eugene yokota 42b8da18c3 Merge pull request #3076 from eed3si9n/fport/2908
[fport] Fix triggeredBy/storeAs/etc using :=
2017-04-04 20:08:05 -07:00
Dale Wijnand f77eeed77e Avoid missleading, link the syntax migration guide
Fixes #2818
2017-04-04 16:52:31 -04:00
Dale Wijnand 7d5dd9999d
Remove some code duplication between TaskInstance and MultiInTask 2017-04-03 17:34:17 +01:00
Dale Wijnand ece85b44bd Merge pull request #3032 from dwijnand/setting-query-json
Start handling default types when serialising query setting values
2017-03-28 10:45:46 +01:00
Dale Wijnand 2bc5ba02f3
Add three more missing WeakTypeTags
.. accidentally removed in 12c2734052
2017-03-27 14:36:56 +01:00
Dale Wijnand 52de082b2e
Add OptJsonWriter to SettingKey 2017-03-27 14:15:11 +01:00
Dale Wijnand ad2f91e357
Extra getName/getImplicit in KeyMacro 2017-03-27 14:15:11 +01:00
Guillaume Martres 747aa48c9c Fix inputTaskDyn not working
This fixes the following error when trying to use inputTaskDyn in a build:

[error] /tmp/sbt_8316130f/input-task-dyn/build.sbt:11: error: Macro
expansion contains free type variable T defined by wrap in
InputConvert.scala:76:20. Have you forgotten to use c.WeakTypeTag
annotation for this type parameter? If you
have troubles tracking free type variables, consider using -Xlog-free-types (out-1)
[error]     runFoo := Def.inputTaskDyn { (out-1)
[error]                                ^ (out-1)
[info] [error] sbt.compiler.EvalException: Type error in expression (out-3) (out-1)

I have no idea what the error means, I just implemented the suggested fix.
2017-03-24 17:41:14 +01:00
Eugene Yokota 180bdfd129 Bump underlying modules to latest 2017-03-23 12:41:24 -04:00
Dale Wijnand 206c3e6d4d
Introduce Def displayBuildRelative/showBuildRelativeKey 2017-03-05 13:42:15 +00:00
Dale Wijnand e67cd6948b
Fix a bunch but not all compile warnings 2017-03-03 01:33:44 +01:00
Eugene Yokota 51f7d2e24a Adds an Append instance that extracts taskValue
This adds a macro-level hack to support += op for sourceGenerators and resourceGenerators using RHS of Initialize[Task[Seq[File]]].
When the types match up, the macro now calls `.taskValue` automatically.
2017-01-22 22:53:27 -05:00
Josh Soref 66ec720884 spelling: specify 2017-01-20 08:27:28 +00:00
Josh Soref 35b55abf0c spelling: semantics 2017-01-20 08:28:56 +00:00
Josh Soref 840dabf6b5 spelling: implementation 2017-01-20 08:17:44 +00:00
eugene yokota e54d4ed8fd Merge pull request #2921 from eed3si9n/fport/2784
[fport] Show deprecations in build.sbt
2017-01-15 23:19:19 -05:00
Dale Wijnand 74cddc254e Deprecate tuple enrichments. Fixes #2763 2017-01-15 09:01:43 -05:00
Eugene Yokota 38cf2cd1b2 Improve deprecation message 2017-01-15 08:54:51 -05:00
Dale Wijnand 7fcfec8b8e
-sbinary/+sjson-new, -datatype/+contraband & upgrades
* start to replace sbinary with sjson-new
* upgrade from sbt/datatype to sbt/contraband
* upgrade and migrate to new sbt modules APIs
2017-01-05 21:59:00 +00:00
Dale Wijnand 43821667bf
Upgrade scalariform version 2016-12-11 12:13:11 +00:00
Dale Wijnand f3a6e9d0c1
Add some singleton type arguments 2016-10-05 11:12:24 -05:00
Dale Wijnand 2f84e05282
Migrate to blackbox.Context 2016-10-05 11:12:24 -05:00
kenji yoshida 27fe8eb6f7 remove unused imports (#2719) 2016-08-30 07:29:17 +01:00
Eugene Yokota 01bd4add3c Replace the old operators with Restligeist macros
The old operators `<<=`, `<+=`, and `<++=` are now replaced with
Restligeist macros that will always fail during compile-time but can
display migration guide as the error message.

This would provide better upgrade experience than simply removing the
methods and displaying `<<= is not a member of sbt.TaskKey`.
2016-08-29 23:08:27 -04:00
eugene yokota 9f21bb451d [sbt 1.0] Remove .value extension method from input tasks (#2710)
* Remove .value from input tasks. Ref #2709

Calling `.value` method on an input task returns `InputTask[A]`, which
is completely unintuitive and often results to a bug.

In most cases `.evaluated` should be called, which returns `A` by
evaluating the task. Just in case `InputTask[A]` is needed,
`toInputTask` method is now provided.

* Fixed test

* Rename toInputTask to inputTaskValue
2016-08-29 21:11:32 +02:00
Dale Wijnand 84c611af36 Remove unused vals/defs 2016-07-12 14:31:35 +01:00
Dale Wijnand deea82542c Remove unused imports 2016-07-12 11:55:10 +01:00
Dale Wijnand 12c2734052 Pattern match some if/else's 2016-07-07 18:21:25 +01:00
Dale Wijnand 4c75d778b9 Group imports 2016-07-07 18:21:25 +01:00
Dale Wijnand 32760bed55 Remove some fatal exception catching 2016-07-07 18:21:25 +01:00
Dale Wijnand ca71b4b902 Cleanup mainSettingsProj 2016-06-21 08:09:30 +01:00
Eugene Yokota ee272d780e Reorganize directory structure 2016-05-06 16:01:49 -04:00