This patch breaks the dependency that Zinc had on sbt's Library
Management (LM) library abstraction (sbt/librarymanagement) and thus,
transitively, on Ivy.
The reason Zinc depends on LM in the first place is to dynamically
download the compiler bridge sources JAR, which it then compiles in
order to bridge from binary-compatible Zinc to the never-binary-stable
(and most of the time source-compatible) compiler API.
From my research (that is, using GitHub Search) the only user of this
feature is sbt, with all other integrations (e.g. pants, bloop, mill)
providing the compiler bridge sources JAR directly (an alternative API
entry point).
Therefore the dependency on LM and that integration code could be
untangled from Zinc and moved into sbt. That would also give an
opportunity to move the code in LM (back) into sbt, which I also think
would be a good idea.
For now this patch leaves that integration code in the already existing
`zinc-ivy-integration` module, but which is now no longer a dependency
of any other module of Zinc, specifically it is no longer a dependency
of the `zinc` and `zincScripted` modules. I think, though,
zinc-ivy-integration should remain in the zinc repo, as a part of the
build, being tested in CI, until it's been moved (back) into sbt. I'd
be happy to take care of both the remaining move and removal tasks.
In order to allow removing LM, Ivy and zinc-ivy-integration as
transitive dependencies of Zinc, this patch has to make a breaking
change to the `ZincUtil` object in the zinc module. Despite it being
defined in the `sbt.internal.inc` package (i.e. not declared as public
API) ZincUtil is in fact a used (Scala) API of Zinc (usages discovered
via GitHub Search). Therefore I chose to leave the object in the zinc
module and only drop from the object the two methods that directly
depend on LM's API. Specifically (1) the `getDefaultBridgeModule`
method and (2) the `scalaCompiler` overload that depends on LM were
removed. These methods now live in a new `ZincLmUtil` object in
zinc-ivy-integration, which sbt can switch to using.
In terms of risk, from my research (GitHub Search) the only users of
those methods are sbt and bloop. I need to confirm but I think bloop
will be able to manage this breaking change (it looks it might be just
fallback usage? TBC). For sbt we don't actual support using new Zinc
with old sbt, so sbt can also just update its usage on its next release.
Any other usage risks being broken but given it's internal, non-public
API I think it's fair game.
The remaining changes are in the testing so that zinc-ivy-integration's
test suite continues to test its download-and-compile behaviour, while
the rest of Zinc switches to using the prebuilt compiler bridges,
reusing and generalising some pre-existing zincScripted code (as well as
re-wiring some modules) so it can be used by zinc's tests too.
This refactors the compiler bridge unit test to use the normal Zinc facility,
namely AnalyzingCompiler that's built on Scala 2.12, but is capable of driving
the compiler bridge built on non-2.12.
This allows us to run the unit tests without any additional dependencies published for Scala 2.13.0-M5.
Fixes https://github.com/sbt/sbt/issues/3405
Ref https://github.com/scala/scala-xml/issues/195
sbt's `run` is emulated using a classloader trick that includes ScalaInstance as the parent classloader under the classpath. The problem is the ScalaInstance classloader currently contains both compiler, library, and their transitive JARs:
```scala
res0: Array[java.io.File] = Array(scala-library.jar, scala-compiler.jar, jline.jar, scala-reflect.jar, scala-xml_2.12.jar)
```
This could have been causing various issues, but most recently it showed up as wrong version of scala-xml getting prioritized over what's passed by the user.
1. new field loaderLibraryOnly is added to xsbti.ScalaInstance.
2. it is initialized to the library loader if the launcher creates it, otherwise create layered loader here.
This aims to isolate the library loader, and retain the perf.
* 1.0.x: (28 commits)
Split compiler bridge tests to another subproject
Implement compiler bridge for 2.13.0-M2
Add yourkit acknoledgement in the README
"sbt '++ 2.13.0-M2!' compile" does not work with sbt 1.0.0
Add header to cached hashing spec
Add headers to missing files
Fix#332: Add sbt-header back to the build
Update sbt-scalafmt to 1.12
Make classpath hashing more lightweight
Fix#442: Name hash of value class should include underlying type
source-dependencies/value-class-underlying: fix test
Ignore null in generic lambda tparams
Improve and make scripted parallel
Fix#436: Remove annoying log4j scripted exception
Fix#127: Use `unexpanded` name instead of `name`
Add pending test case for issue/127
source-dependencies / patMat-scope workaround
Fixes undercompilation on inheritance on same source
Add real reproduction case for sbt/zinc#417
Add trait-trait-212 for Scala 2.12.3
...
Conflicts:
internal/zinc-apiinfo/src/main/scala/sbt/internal/inc/ClassToAPI.scala
project/build.properties
zinc/src/main/scala/sbt/internal/inc/MixedAnalyzingCompiler.scala
The ClassToAPI conflict is due to:
* https://github.com/sbt/zinc/pull/393 (a 1.x PR), conflicting with
* https://github.com/sbt/zinc/pull/446 (a 1.0.x PR).
The build.properties conflict is due to different PRs bumping
sbt.version from 1.0.0 to 1.0.2 to 1.0.3. (#413, #418, #453).
The MixedAnalyzingCompiler conflict is due to:
* https://github.com/sbt/zinc/pull/427 (a 1.x PR), conflicting with
* https://github.com/sbt/zinc/pull/452 (a 1.0.x PR).
Splitting compiler bridge tests to another subproject because while the bridge itself can be compiled with just compiler-interface, util-interface, and Scala Compiler as dependencies, the testing introduces more (such as IO). This creates problem for new Scala versions where IO or test libraries do not exist yet (e.g. Scala 2.13.0-M2).
This also removes the Mima test due to the lack of 2.13 bridge for Zinc 1.0.0.
Compiler bridge just needs to compile itself against the interface and Scala compiler, so there's no need to run Mima test.
We have to look into the reason why the shaded version of the compiler
plugin reports the following error and halts the build:
```
protoc-jar: executing: [/tmp/protocjar2818613521400194680/bin/protoc.exe, --plugin=protoc-gen-scala=/tmp/protocbridge4921434211196915044, --scala_out=grpc:/data/rw/code/scala/zinc/internal/zinc-persist/target/scala-2.12/src_managed/main, -I/data/rw/code/scala/zinc/internal/zinc-persist/src/main/protobuf, -I/data/rw/code/scala/zinc/internal/zinc-persist/target/protobuf_external, /data/rw/code/scala/zinc/internal/zinc-persist/src/main/protobuf/schema.proto]
java.lang.AbstractMethodError: Method scalapb/ScalaPbCodeGenerator$.run(Lcom/google/protobuf/compiler/PluginProtos$CodeGeneratorRequest;)Lcom/google/protobuf/compiler/PluginProtos$CodeGeneratorResponse; is abstract
at scalapb.ScalaPbCodeGenerator$.run(ScalaPbCodeGenerator.scala)
at protocbridge.frontend.PluginFrontend$.$anonfun$runWithBytes$1(PluginFrontend.scala:56)
at scala.util.Try$.apply(Try.scala:209)
at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:54)
at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:67)
at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$1(PosixPluginFrontend.scala:25)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:653)
at scala.util.Success.$anonfun$map$1(Try.scala:251)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:287)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:140)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
```
Before, we were using `sbt.inc`. As we have settled to define all the
interfaces in the `xsbti.compile` namespaces, it's not worth it to force
users remember other locations for the interfaces forwarding the default
implementations in Scala.
* Move `IncrementalCompilerUtil` to `ZincCompilerUtil`.
* Change `IncrementalCompilerUtil` to an interface with static methods.
* Remove bug in which ScalaInstance implementation was leaking to the
public API.
This is the proof that we're now testing the Zinc public API because
we're resolving with the ivy options and resolution specified in the
default compiler bridge provider.
Expose default component provider through a static method in
`ZincCompilerBridge`. This method is necessary to invoke the bridge.
Removes the previous global lock and component provider implementations
in `BridgeProviderSpecification`.
* Adds a method to fetch the default implementation of the compiler
provider.
* Adds a method to get a default `IvyConfiguration` from a small number
of parameters.
The public API is implemented in Java to ensure binary compatibility no
matter what happens to the internal Scala implementations. It also makes
it easier for Java users to use the API and avoid importing MODULE$'s
and Scala generated class files.'
This commit makes sure that we always check that we compile against the
latest two versions in Scala 2.10, Scala 2.11 and Scala 2.12.
This commit also removes the null loggers and uses normal loggers to
check what the output of the bridge provider is. This output is
important and the rationale to do this is that it should be visible for
us in order to detect cases where there's a regression or the output is
modified in some way. Output is short anyway.
* Remove unused code in `BridgeProviderSpecification` and test the real
code in the default provider.
* Don't use temporary directory to download the Scala jars, retrive them
in a directory specified by the user.
* Use `java.net.ClassLoader` instead of `ScalaClassLoader`.
* Use the `ScalaInstance` interface, not the implementation. Remove any
reference to the implementation.
* Add interface for the provider.
* Rename `IvyComponentCompiler` to `ZincComponentCompiler`.
* Split `ZincComponentCompiler` and `ZincComponentManager`.
* Define `IfMissing` and `InvalidComponent` into independent files.
* Rename variables and internal API to be clearer.
The compiler bridge dependency resolution had `missingOk = true` flag on. This would silently succeed even when one of the dependency is missing, and it would randomly fail bridge compilation, which is very confusing.
This commit addresses several concers:
* Document better the Zinc internal implementation.
* Getting the implementation ready for moving things from internal to
public APIs. This will be done in a follow-up commit.
* Removing `inputs` and `setups` from the `IncrementalCompilerUtil` as
they exist mainly to interface between Scala code and Java code. The
reality is that both Java and Scala developers can easily call the
constructors by themselves and don't learn more API methods.
* Removes unused imports, that emit stupid warnings.
* Make code more readable, especifically in the analyzing compilers for
both Java and Scala, which was hardly understandable and documented.
* Make `AnalyzingJavaCompiler` to extend `JavaCompiler` as
`AnalyzingCompiler` does for `ScalaCompiler`. In an upcoming commit,
`AnalyzingCompiler` will be renamed into `AnalyzingScalaCompiler` for
consistency reasons.
* Replace use of `sbt.util.Logger` helpers by `sbt.util.InterfaceUtil`.
This migrates the `ComponentManager` from librarymanagement, and refactors it to clarify the double caching of the compiler bridge.
The main purpose of this refactoring is to fixsbt/sbt#2539. As it stands, becaue Ivy cache directory is being used as the secondary cache of the compiled compiler bridge, it's annoying to work with the compiler bridge. Instead of using "Ivy cache", the new `ZincComponentManager` accepts `secondaryCacheDir: Option[File]`, which during test defaults to `target/zinc-components`.
Fixessbt/sbt#2539
Dale Wijnand
Eugene Yokota
Kazuhiro Sera
Łukasz Wawrzyk
exoego
Allan Timothy Leong
jvican
Martin Duhem