The qualifier of the `.value` call may contain `DefTree`s (e.g.
vals, defs) or `Function` trees. When we snip them out of the
tree and graft them into a new context, we must also call
`changeOwner`, so that the symbol owner structure and the tree
structure are coherent.
Failure to do so resulted in a crash in the compiler backend.
Fixes#1150
The qualifier of the `.value` call may contain `DefTree`s (e.g.
vals, defs) or `Function` trees. When we snip them out of the
tree and graft them into a new context, we must also call
`changeOwner`, so that the symbol owner structure and the tree
structure are coherent.
Failure to do so resulted in a crash in the compiler backend.
Fixes#1150
This commit makes the code source compatible across Scala 2.10.3
and https://github.com/scala/scala/pull/3452, which is proposed
for inclusion in Scala 2.11.0-RC1.
We only strictly need the incremental compiler to build on Scala
2.11, as that is integrated into the IDE. But we gain valuable
insight into compiler regressions by building *all* of SBT with
2.11.
We only got there recently (the 0.13 branch of SBT now fully cross
compiles with 2.10.3 and 2.11.0-SNAPSHOT), and this aims to keep
things that way.
Once 2.10 support is dropped, SBT macros will be able to exploit
the new reflection APIs in 2.11 to avoid the need for casting
to compiler internals, which aren't governed by binary compatibility.
This has been prototyped by @xeno-by: https://github.com/sbt/sbt/pull/1121
Since the fix for SI-2066, Scala 2.11 calls logicallyEnclosingMember on the
`x` in the expansion of the task macro:
InitializeInstance.app[[T0[x]](T0[java.io.File], T0[java.io.File]), Seq[java.io.File]]
This exposed the fact that SBT has created `T0` with `NoSymbol` as
the owner. This led to the a SOE.
I will also change the compiler to be more tolerant of this, but we
can observe good discipline in the macro and pick a sensible owner.
Still TODO for auto-plugins/logic:
* property-based tests for logic system
* user documentation
* (optional) 'about plugins' or similar to show more information about the auto-plugins for a project
* (deferred) allow AutoPlugin to inject Commands directly?
* (deferred) provide AutoPlugin functionality to arbitrary scopes instead of just at the Project level?
This requires a Format[T] to be implicitly available at the call site and requires the task
to be referenced statically (not in a settingDyn call). References to previous task values
in the form of a ScopedKey[Task[T]] + Format[T] are collected at setting load time in the
'references' setting. These are used to know which tasks should be persisted (the ScopedKey)
and how to persist them (the Format).
When checking/delegating previous references, rules are slightly different.
A normal reference from a task t in scope s cannot refer to t in s unless
there is an earlier definition of t in s. However, a previous reference
does not have this restriction. This commit modifies validateReferenced
to allow this.
TODO: user documentation
TODO: stable selection of the Format when there are multiple .previous calls on the same task
TODO: make it usable in InputTasks, specifically Parsers
A hash for given name in a source file is computed by combining
hashes of all definitions with given name. When hashing a single
definition we take into account all information about it except nested
definitions. For example, if we have following definition
class Foo[T] {
def bar(x: Int): Int = ???
}
hash sum for `Foo` will include the fact that we have a class with
a single type parameter but it won't include hash sum of `bar` method.
Computed hash sums are location-sensitive. Each definition is hashed along
with its location so we properly detect cases when definition's signature
stays the same but it's moved around in the same compilation unit.
The location is defined as sequence of selections. Each selection consists
of a name and name type. The name type is either term name or type name.
Scala specification (9.2) guarantees that each publicly visible definition
is uniquely identified by a sequence of such selectors.
For example, if we have:
object Foo {
class Bar { def abc: Int }
}
then location of `abc` is Seq((TermName, Foo), (TypeName, Bar))
It's worth mentioning that we track name-hash pairs separately for
regular (non implicit) and implicit members. That's required for name
hashing algorithm because it does not apply its heuristic when implicit
members are being modified.
Another important characteristic is that we include all inherited members
when computing name hashes.
Here comes the detailed list of changes made in this commit:
* HashAPI has new parameter `includeDefinitions` that allows
shallow hashing of Structures (where we do not compute hashes
recursively)
* HashAPI exposes `finalizeHash` method that allow one to capture
current hash at any time. This is useful if you want to hash a list of
definitions and not just whole `SourceAPI`.
* NameHashing implements actual extraction of public definitions,
grouping them by simple name and computing hash sums for each group
using HashAPI
* `Source` class (defined in interface/other file) has been extended to
include `_internalOnly_nameHashes` field. This field stores
NameHashes data structure for given source file. The NameHashes
stores two separate collections of name-hash pairs for regular and
implicit members.
The prefix `_internalOnly_` is used to indicate that this is not an
official incremental compiler's or sbt's API and it's for use by
incremental compiler internals only. We had to use such a prefix
because the `datatype` code generator doesn't support emitting access
modifiers
* `AnalysisCallback` implementation has been modified to gather all
name hashes and store them in the Source object
* TestCaseGenerators has been modified to implement generation of
NameHashes
* The NameHashingSpecification contains a few unit tests that make sure
that the basic functionality works properly
Tracking of used names is a component needed by the name hashing
algorithm. The extraction and storage of used names is active only when
`AnalysisCallback.nameHashing` flag is enabled and it's disabled by
default.
This change constists of two parts:
1. Modification of Relations to include a new `names` relation
that allows us to track used names in Scala source files
2. Implementation of logic that extracts used names from Scala
compilation units (that correspond to Scala source files)
The first part is straightforward: add standard set of methods in
Relations (along with their implementation) and update the logic which
serializes and deserializes Relations.
The second part is implemented as tree walk that collects all symbols
associated with trees. For each symbol we extract a simple, decoded name
and add it to a set of extracted names. Check documentation of
`ExtractUsedNames` for discussion of implementation details.
The `ExtractUsedNames` comes with unit tests grouped in
`ExtractUsedNamesSpecification`. Check that class for details.
Given the fact that we fork while running tests in `compiler-interface`
subproject and tests are ran in parallel which involves allocating
multiple Scala compiler instances we had to bump the default memory limit.
This commit contains fixes for gkossakowski/sbt#3, gkossakowski/sbt#5 and
gkossakowski/sbt#6 issues.
The previous name of the flag was rather specific: it indicated
whether the new source dependency tracking is supported by given Relations
object. However, there will be more functionality added to Relations that
is specific to name hashing algorithm. Therefore it makes sense to name
the flag as just `nameHashing`.
I decided to rename Relations implementation classes to be more
consistent with the name of the flag and with the purpose they serve.
The flag in AnalysisCallback (and classes implementing it) has been
renamed as well.
Previously incremental compiler was extracting source code
dependencies by inspecting `CompilationUnit.depends` set. This set is
constructed by Scala compiler and it contains all symbols that given
compilation unit refers or even saw (in case of implicit search).
There are a few problems with this approach:
* The contract for `CompilationUnit.depend` is not clearly defined
in Scala compiler and there are no tests around it. Read: it's
not an official, maintained API.
* Improvements to incremental compiler require more context
information about given dependency. For example, we want to
distinguish between dependency on a class when you just select
members from it or inherit from it. The other example is that
we might want to know dependencies of a given class instead of
the whole compilation unit to make the invalidation logic more
precise.
That led to the idea of pushing dependency extracting logic to
incremental compiler side so it can evolve indepedently from Scala
compiler releases and can be refined as needed. We extract
dependencies of a compilation unit by walking a type-checked tree
and gathering symbols attached to them.
Specifically, the tree walk is implemented as a separate phase that
runs after pickler and extracts symbols from following tree nodes:
* `Import` so we can track dependencies on unused imports
* `Select` which is used for selecting all terms
* `Ident` used for referring to local terms, package-local terms
and top-level packages
* `TypeTree` which is used for referring to all types
Note that we do not extract just a single symbol assigned to `TypeTree`
node because it might represent a complex type that mentions
several symbols. We collect all those symbols by traversing the type
with CollectTypeTraverser. The implementation of the traverser is inspired
by `CollectTypeCollector` from Scala 2.10. The
`source-dependencies/typeref-only` test covers a scenario where the
dependency is introduced through a TypeRef only.
The fix was made possible by the very helpful information provided by @retronym.
This commit does two key things:
1. changes the owner when splicing original trees into new trees
2. ensures the synthetic trees that get spliced into original trees do not need typechecking
Given this original source (from Defaults.scala):
...
lazy val sourceConfigPaths = Seq(
...
unmanagedSourceDirectories := Seq(scalaSource.value, javaSource.value),
...
)
...
After expansion of .value, this looks something like:
unmanagedSourceDirectories := Seq(
InputWrapper.wrapInit[File](scalaSource),
InputWrapper.wrapInit[File](javaSource)
)
where wrapInit is something like:
def wrapInit[T](a: Any): T
After expansion of := we have (approximately):
unmanagedSourceDirectories <<=
Instance.app( (scalaSource, javaSource) ) {
$p1: (File, File) =>
val $q4: File = $p1._1
val $q3: File = $p1._2
Seq($q3, $q4)
}
So,
a) `scalaSource` and `javaSource` are user trees that are spliced into a tuple constructor after being temporarily held in `InputWrapper.wrapInit`
b) the constructed tuple `(scalaSource, javaSource)` is passed as an argument to another method call (without going through a val or anything) and shouldn't need owner changing
c) the synthetic vals $q3 and $q4 need their owner properly set to the anonymous function
d) the references (Idents) $q3 and $q4 are spliced into the user tree `Seq(..., ...)` and their symbols need to be the Symbol for the referenced vals
e) generally, treeCopy needs to be used when substituting Trees in order to preserve attributes, like Types and Positions
changeOwner is called on the body `Seq($q3, $q4)` with the original owner sourceConfigPaths to be changed to the new anonymous function.
In this example, no owners are actually changed, but when the body contains vals or anonymous functions, they will.
An example of the compiler crash seen when the symbol of the references is not that of the vals:
symbol value $q3 does not exist in sbt.Defaults.sourceConfigPaths$lzycompute
at scala.reflect.internal.SymbolTable.abort(SymbolTable.scala:49)
at scala.tools.nsc.Global.abort(Global.scala:254)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.genLoadIdent$1(GenICode.scala:1038)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genLoad(GenICode.scala:1044)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase$$anonfun$genLoadArguments$1.apply(GenICode.scala:1246)
at scala.tools.nsc.backend.icode.GenICode$ICodePhase$$anonfun$genLoadArguments$1.apply(GenICode.scala:1244)
...
Other problems with the synthetic tree when it is spliced under the original tree often result in type mismatches or some other compiler error that doesn't result in a crash.
If the owner is not changed correctly on the original tree that gets spliced under a synthetic tree, one way it can crash the compiler is:
java.lang.IllegalArgumentException: Could not find proxy for val $q23: java.io.File in List(value $q23, method apply, anonymous class $anonfun$globalCore$5, value globalCore, object Defaults, package sbt, package <root>) (currentOwner= value dir )
...
while compiling: /home/mark/code/sbt/main/src/main/scala/sbt/Defaults.scala
during phase: global=lambdalift, atPhase=constructors
...
last tree to typer: term $outer
symbol: value $outer (flags: <synthetic> <paramaccessor> <triedcooking> private[this])
symbol definition: private[this] val $outer: sbt.BuildCommon
tpe: <notype>
symbol owners: value $outer -> anonymous class $anonfun$87 -> value x$298 -> method derive -> class BuildCommon$class -> package sbt
context owners: value dir -> value globalCore -> object Defaults -> package sbt
...
The problem here is the difference between context owners and the proxy search chain.
Fix the problem with unstable names synthesized for existential
types (declared with underscore syntax) by renaming type variables
to a scheme that is guaranteed to be stable no matter where given
the existential type appears.
The sheme we use are De Bruijn-like indices that capture both position
of type variable declarion within single existential type and nesting
level of nested existential type. This way we properly support nested
existential types by avoiding name clashes.
In general, we can perform renamings like that because type variables
declared in existential types are scoped to those types so the renaming
operation is local.
There's a specs2 unit test covering instability of existential types.
The test is included in compiler-interface project and the build
definition has been modified to enable building and executing tests
in compiler-interface project. Some dependencies has been modified:
* compiler-interface project depends on api project for testing
(test makes us of SameAPI)
* dependency on junit has been introduced because it's needed
for `@RunWith` annotation which declares that specs2 unit
test should be ran with JUnitRunner
SameAPI has been modified to expose a method that allows us to
compare two definitions.
This commit also adds `ScalaCompilerForUnitTesting` class that allows
to compile a piece of Scala code and inspect information recorded
callbacks defined in `AnalysisCallback` interface. That class uses
existing ConsoleLogger for logging. I considered doing the same for
ConsoleReporter. There's LoggingReporter defined which would fit our
usecase but it's defined in compile subproject that compiler-interface
doesn't depend on so we roll our own.
ScalaCompilerForUnit testing uses TestCallback from compiler-interface
subproject for recording information passed to callbacks. In order
to be able to access TestCallback from compiler-interface
subproject I had to tweak dependencies between interface and
compiler-interface so test classes from the former are visible in the
latter. I also modified the TestCallback itself to accumulate apis in
a HashMap instead of a buffer of tuples for easier lookup.
An integration test has been added which tests scenario
mentioned in #823.
This commit fixes#823.
As pointed out by @harrah in #705, both beginSource and endSource are
not used in sbt internally for anything meaningful.
We've discussed an option of deprecating those methods but since they
are not doing anything meaningful Mark prefers to have compile-time
error in case somebody implements or calls those methods. I agree with
that hence removal.
Josh Suereth
Jason Zaugg
Grzegorz Kossakowski
Mark Harrah
Dan Sanduleac
Benjy
Bruno Bieth