In Scala 2.10.4, this macro can produce a stack overflow :
def foo(a: Any): Any = macro impl
def impl(c: Context)(a: c.Expr[Any]): c.Expr[Any] = a
Here, an application such as `foo(someVal)` will produce the expansion
`someVal`. As expected, `someVal` has `original` tree `foo(someVal)`,
but if we inspect this tree, we will find that `someVal` has an
original tree, but it shouldn't.
Moreover, in Scala 2.11, some macros have their own application as
`original` trees.
See sbt/sbt#1237 for a description of these problems.
This commit fixes these two problems.
Fixessbt/sbt#1237
It has been reported in sbt/sbt#1237 that stack overflows may occur during the
extraction of used names (and later of dependencies between files). This
problem has been introduced by sbt/sbt#1163, which was about recording the
dependencies of macro arguments.
When a macro is expanded, the compiler attaches the tree before expansion to
the tree representing the expanded macro. As of Scala 2.11-RC3, some macros
have themselves attached as original tree, which caused the same macro to be
inspected over and over until a stack overflow.
This commit solves this problem by making sure that the original of a macro
expansion will be inspected if and only if it is different from the expanded
tree.
Fixessbt/sbt#1237
Add a unit test which checks whether we capture dependencies introduced
by arguments to macros. Those dependencies are special because macros
get expanded during type checking and arguments to macros are not visible
during regular tree walk.
It was not possible to make `ScalaCompilerForUnitTesting` compile several
files in different runs, which means that it was not possible to compile
and use a macro in a test case, since macros cannot be used in the same
compilation run that defines them.
This commit allows a test case to provide multiple grouped snippets of
code that will be compiled in separate runs.
For instance :
List(Map(<snippet A>, <snippet B>), Map(<snippet C>))
Here, <snippet A> and <snippet B> will be compiled together, and then
<snippet C> will be compiled, and will be able to use symbols defined
in <snippet A> or <snippet B>.
Macros take arguments as trees and return some other trees; both of
them have dependencies but we see trees only after expansion and
recorded only those dependencies.
This commit solves this problem by looking into the attachments of the
trees that are supposed to contain originals of macro expansions and
recording dependencies of the macro before its expansion.
Tracking of used names is a component needed by the name hashing
algorithm. The extraction and storage of used names is active only when
`AnalysisCallback.nameHashing` flag is enabled and it's disabled by
default.
This change constists of two parts:
1. Modification of Relations to include a new `names` relation
that allows us to track used names in Scala source files
2. Implementation of logic that extracts used names from Scala
compilation units (that correspond to Scala source files)
The first part is straightforward: add standard set of methods in
Relations (along with their implementation) and update the logic which
serializes and deserializes Relations.
The second part is implemented as tree walk that collects all symbols
associated with trees. For each symbol we extract a simple, decoded name
and add it to a set of extracted names. Check documentation of
`ExtractUsedNames` for discussion of implementation details.
The `ExtractUsedNames` comes with unit tests grouped in
`ExtractUsedNamesSpecification`. Check that class for details.
Given the fact that we fork while running tests in `compiler-interface`
subproject and tests are ran in parallel which involves allocating
multiple Scala compiler instances we had to bump the default memory limit.
This commit contains fixes for gkossakowski/sbt#3, gkossakowski/sbt#5 and
gkossakowski/sbt#6 issues.
Add documentation which explains how a general technique using implicits
conversions is employed in Compat class. Previously, it was hidden inside
of Compat class.
Also, I changed `toplevelClass` implementation to call
`sourceCompatibilityOnly` method that is designed for the purpose
of being a compatibility stub.
The scala/scala@2d4f0f1859 removes the
`toplevelClass` method. The recent change from
aac19fd02b introduces dependency on that
method. Combination of both changes makes incremental compiler incompatible
with Scala 2.11.
This change introduces a compatibility hack that brings back source
compatibility of incremental compiler with Scala 2.8, 2.9, 2.10 and 2.11.
The compatibility hack is making clever use implicit conversions that
can provide dummy method definitions for methods removed from Scala
compiler.
Also, the code that depends on `enclosingTopLevelClass` has been refactored
so the dependency is more centralized.
The previous name of the flag was rather specific: it indicated
whether the new source dependency tracking is supported by given Relations
object. However, there will be more functionality added to Relations that
is specific to name hashing algorithm. Therefore it makes sense to name
the flag as just `nameHashing`.
I decided to rename Relations implementation classes to be more
consistent with the name of the flag and with the purpose they serve.
The flag in AnalysisCallback (and classes implementing it) has been
renamed as well.
The documentation of `Relations.inheritance` mentions an oddity of Scala's
type checker which manifests itself in what is being tracked by that
relation in case of traits being first parent for a class/trait.
Add a test case which verifies that this oddity actually exists and it's
not harmful because it doesn't break an invariant between `memberRef`
and `inheritance` relations.
Flip `memberRefAndInheritanceDeps` flag to true which allows us to
test `memberRef` and `inheritance` relations instead of `direct` and
`publicInherited` as it was previously done.
There a few changes to extracted dependencies from public members:
* F doesn't depend on C by inheritance anymore. The dependency on
C was coming from self type. This shows that dependencies from self
types are not considered to be dependencies introduces by inheritance
anymore.
* G depends on B by member reference now. This dependency is introduced
by applying type constructor `G.T` and expanding the result of the
application.
* H doesn't depend on D by inheritance anymore. That dependency was
introduced through B which inherits from D. This shows that only
parents (and not all base classes) are included in `inheritance`
relation.
NOTE: The second bullet highlights a bug in the old dependency tracking
logic. The dependency on B was recorded in `publicInherited` but not in
`direct` relation. This breaks the contract which says that
`publicInherited` is a subset of `direct` relation.
This a change to dependencies extracted from non-public members:
* C depends on A by inheritance and D depends on B by inheritance now;
both changes are of the same kind: dependencies introduced by
inheritance are tracked for non-public members now. This is necessary
for name hashing correctness algorithm
Add specs2 specification (unit test) which documents current dependency
extraction logic's behavior. It exercises `direct` and `publicInherited`
relations.
This test is akin to `source-dependencies/inherited-dependencies` scripted
test. We keep both because this test will diverge in next commit to test
`memberRef` and `inheritance` relations.
The idea behind adding this test and then modifying the
`memberRefAndInheritanceDeps` flag so we test `memberRef` and `inheritance`
is that we can show precisely the differences between those two dependency
tracking mechanisms.
Adding source dependency on itself doesn't really bring any value so
there's no reason to do it. We avoided recording that kind of dependencies
by performing a check in `AnalysisCallback` implementation. However, if we
have another implementation like `TestCallback` used for testing we do
not benefit from that check.
Therefore, the check has been moved to dependency phase were dependencies
are collected.
Add `extractDependenciesFromSrcs` method to ScalaCompilerForUnitTest
class which allows us to unit test dependency extraction logic.
See the comment attached to the method that explain the details of
how it should be used.
Refactor ScalaCompilerForUnitTesting by introducing a new method
`extractApiFromSrc` which better describes the intent than
`compileSrc`. The `compileSrc` becomes a private, utility method.
Also, `compileSrc` method changed it's signature so it can take
multiple source code snippets as input. This functionality will
be used in future commits.
Previously incremental compiler was extracting source code
dependencies by inspecting `CompilationUnit.depends` set. This set is
constructed by Scala compiler and it contains all symbols that given
compilation unit refers or even saw (in case of implicit search).
There are a few problems with this approach:
* The contract for `CompilationUnit.depend` is not clearly defined
in Scala compiler and there are no tests around it. Read: it's
not an official, maintained API.
* Improvements to incremental compiler require more context
information about given dependency. For example, we want to
distinguish between dependency on a class when you just select
members from it or inherit from it. The other example is that
we might want to know dependencies of a given class instead of
the whole compilation unit to make the invalidation logic more
precise.
That led to the idea of pushing dependency extracting logic to
incremental compiler side so it can evolve indepedently from Scala
compiler releases and can be refined as needed. We extract
dependencies of a compilation unit by walking a type-checked tree
and gathering symbols attached to them.
Specifically, the tree walk is implemented as a separate phase that
runs after pickler and extracts symbols from following tree nodes:
* `Import` so we can track dependencies on unused imports
* `Select` which is used for selecting all terms
* `Ident` used for referring to local terms, package-local terms
and top-level packages
* `TypeTree` which is used for referring to all types
Note that we do not extract just a single symbol assigned to `TypeTree`
node because it might represent a complex type that mentions
several symbols. We collect all those symbols by traversing the type
with CollectTypeTraverser. The implementation of the traverser is inspired
by `CollectTypeCollector` from Scala 2.10. The
`source-dependencies/typeref-only` test covers a scenario where the
dependency is introduced through a TypeRef only.
Fix the problem with unstable names synthesized for existential
types (declared with underscore syntax) by renaming type variables
to a scheme that is guaranteed to be stable no matter where given
the existential type appears.
The sheme we use are De Bruijn-like indices that capture both position
of type variable declarion within single existential type and nesting
level of nested existential type. This way we properly support nested
existential types by avoiding name clashes.
In general, we can perform renamings like that because type variables
declared in existential types are scoped to those types so the renaming
operation is local.
There's a specs2 unit test covering instability of existential types.
The test is included in compiler-interface project and the build
definition has been modified to enable building and executing tests
in compiler-interface project. Some dependencies has been modified:
* compiler-interface project depends on api project for testing
(test makes us of SameAPI)
* dependency on junit has been introduced because it's needed
for `@RunWith` annotation which declares that specs2 unit
test should be ran with JUnitRunner
SameAPI has been modified to expose a method that allows us to
compare two definitions.
This commit also adds `ScalaCompilerForUnitTesting` class that allows
to compile a piece of Scala code and inspect information recorded
callbacks defined in `AnalysisCallback` interface. That class uses
existing ConsoleLogger for logging. I considered doing the same for
ConsoleReporter. There's LoggingReporter defined which would fit our
usecase but it's defined in compile subproject that compiler-interface
doesn't depend on so we roll our own.
ScalaCompilerForUnit testing uses TestCallback from compiler-interface
subproject for recording information passed to callbacks. In order
to be able to access TestCallback from compiler-interface
subproject I had to tweak dependencies between interface and
compiler-interface so test classes from the former are visible in the
latter. I also modified the TestCallback itself to accumulate apis in
a HashMap instead of a buffer of tuples for easier lookup.
An integration test has been added which tests scenario
mentioned in #823.
This commit fixes#823.
As pointed out by @harrah in #705, we might want to merge both API
and dependency phases so we should mention that in the comment explaining
phase ordering constraints instead.
I'd still like to keep the old comment in the history (as separate commit)
because it took me a while to figure out cryptic issues related to
continuations plugin so it's valuable to keep the explanation around in
case somebody else in the future tries to mess around with dependencies
defined by sbt.
This is the first step towards using new mechanism for dependency
extraction that is based on tree walking.
We need dependency extraction in separate phase because the code
walking trees should run before refchecks whereas analyzer phase runs
at the very end of phase pipeline.
This change also includes a work-around for phase ordering issue with
continuations plugin. See included comment and SI-7217 for details.
As pointed out by @harrah in #705, both beginSource and endSource are
not used in sbt internally for anything meaningful.
We've discussed an option of deprecating those methods but since they
are not doing anything meaningful Mark prefers to have compile-time
error in case somebody implements or calls those methods. I agree with
that hence removal.
The test case compiles a project without and with this
setting and checks that a warning is and isn't emitted
respectively.
It's a multi-project build; this bug didn't seem to turn
up in a single-project build.
This way we have a little bit more clear separation
between compiler phase logic and the core logic responsible for
processing each compilation unit and extracting an api for it.
As added benefit, we have a little bit less of mutable state
(e.g. sourceFile doesn't need to be a var anymore).
The API extraction logic contains some internal caches that are
required for correctness. It wasn't very clear if they have to
be maintained during entire phase run or just during single compilation
unit processing. It looks like they have to be maintained during
single compilation unit processing and refactored code both
documents that contracts and implements it in the API phase.
Move collection (a class `Compat`) of compatibility hacks into separate
file. This aids understanding of the code as both Analyzer and API make
use of that class and keeping it `Analyzer.scala` file suggested that
it's used only by Analyzer.
Incremental compiler didn't have any explicit logic to handle
cancelled compilation so it would go into inconsistent state.
Specifically, what would happen is that it would treat cancelled
compilation as a compilation that finished normally and try to
produce a new Analysis object out of partial information collected
in AnalysisCallback. The most obvious outcome would be that the
new Analysis would contain latest hashes for source files. The
next time incremental compiler was asked to recompile the same files
that it didn't recompile due to cancelled compilation it would think
they were already successfully compiled and would do nothing.
We fix that problem by following the same logic that handles compilation
errors, cleans up partial results (produced class files) and makes sure
that no Analysis is created out of broken state.
We do that by introducing a new exception `CompileCancelled`
and throwing it at the same spot as an exception signalizing compilation
errors is being thrown. We also modify `IncrementalCompile` to
catch that exception and gracefully return as there was no compilation
invoked.
NOTE: In case there were compilation errors reported _before_
compilation cancellations was requested we'll still report them
using an old mechanism so partial errors are not lost in case
of cancelled compilation.
In summary this commit:
* drops type normalization in api phase but keeps dealiasing
* fixes#736 and marks corresponding test as passing
I discussed type normalization with @adriaanm and according to him
sbt shouldn't call that method. The purpose of this method to
convert to a form that subtyping algorithm expects. Sbt doesn't need
to call it and it's fairly expensive in some cases.
Dropping type normalization also fixes#726 by not running into
stale cache in Scala compiler problem described in SI-7361.
The infrastructure for resident compilation still exists,
but the actual scalac-side code that was backported is removed.
Future work on using a resident scalac will use that invalidation
code directly from scalac anyway.
While trying to determine binary dependencies sbt lookups class files
corresponding to symbols. It tried to do that for packages and most of the
time would fail because packages don't have corresponding class file
generated. However, in case of case insensitive file system, combined
with special nesting structure you could get spurious dependency.
See added test case for an example of such structure.
The remedy is to never even try to locate class files corresponding to
packages.
Fixes#620.
Extract the api after picklers, since that way we see the same symbol
information/structure irrespective of whether we were typechecking
from source / unpickling previously compiled classes.
Previously, the apiExtractor phase ran after typer.
Since this fix is hard to verify with a test (it's based on the
conceptual argument above, and anecdotal evidence of incremental
compilation of a big codebase), we're providing a way to restore the
old behaviour: run sbt with -Dsbt.api.phase=typer.
This fixes#609.
Martin Duhem
Brian McKenna
Josh Suereth
Eugene Yokota
Grzegorz Kossakowski
Jason Zaugg
Mark Harrah
Adriaan Moors