The compiler interface subclasses `scala.tools.nsc.Global`,
and loading this new subclass before each `compile` task forces
HotSpot JIT to deoptimize larges swathes of compiled code. It's
a bit like SBT has rigged the dice to always descend the longest
ladder in a game of Snakes and Ladders.
The slowdown seems to be larger with Scala 2.12. There are a number
of variables at play, but I think the main factor here is that
we now rely on JIT to devirtualize calls to final methods in traits
whereas we used to emit static calls. JIT does a good job at this,
so long as classloading doesn't undo that good work.
This commit extends the existing `ClassLoaderCache` to encompass
the classloader that includes the compiler interface JAR. I've
resorted to adding a var to `AnalyzingCompiler` to inject the
dependency to get the cache to the spot I need it without binary
incompatible changes to the intervening method signatures.
The issue is that when you manually set a ScalaInstance, i.e. not one from Ivy, the
classpath which is returned for any given configuration ONLY uses Ivy. This means that
the legitimate Scala JAR files that need to be on the classpath are missing from the list.
For some reason, the way we instantiate tests uses an unfiltered classloader against the
ScalaInstance, *BUT* the thread-context-classloader DOES use a filtered instance by
classpath. This add the hook into the TestFramework runner creation so that
the classpath accurately reflects the jars needed.
cc @rkuhn
Construction of Scala providers was already properly synchronized jvm and machine-wide.
The cache on top of construction was not and neither was the newer ClassLoaderCache.
This could cause the same Scala version to be loaded in multiple class loaders, taking
up more permgen space and possibly decreasing performance due to less effective jit.
The issue is very rare in practice for 0.13 because of the low probability of contention
on ClassLoaderCache. This is because the work for a cache miss is mainly the construction
of a URLClassLoader. In 0.12, however, the work potentially involved network access and
class loading (not just class loader construction), thus greatly increasing the probability
of contention and thus duplicate work (i.e. class loader construction).
When there is contention, multiple class loaders are constructed and then preserved by the
scalaInstance task in each project throughout the first task execution. Only when multiple
scalaInstance tasks execute simultaneously and only during the first execution does this occur.
(Technically, it could still happen later, but it doesn't in practice.)
This means that the number of duplicate class loaders should quickly saturate instead of growing
linearly with the number of projects. It also means that the impact depends on the exact
tree structure of projects. A linear chain of dependencies will be unaffected, but a build with
independent leaves may be limited by the number of cores. The number of cores affects
the number of threads typically used by the task engine, which limits the number of concurrently
executing scalaInstance tasks.
In summary, this might affect the first, cold compilation of a multi-module project with
independent leaves on a multi-core machine with Scala version different from the version used
for sbt. It might increase the maximum permgen requirements as well as slow the jit compilation
by up to one task execution. Subsequent compilations should be unaffected and the permgen
utilization return to be as expected.
Specifically, when the Scala version for sbt is the same as that for the project being built,
the jars in UpdateReport should be the same as those in ScalaProvider. This is because the
loader will come from the ScalaProvider, which uses jars in the boot directory instead of the
cache. The first part of the fix for #661 checks that loaded classes come from the classpath
and so they need to line up.
It is now used for consoleProject, run, and test. This loader verifies
that all classes loaded through it came from a particular classpath or
from the "root" loader. Root loader here is the launcher loader so that
those classes with native bindings come from that shared loader.
1. Scala jars won't be copied to the boot directory, except for those needed to run sbt.
2. Scala SNAPSHOTs behave like normal SNAPSHOTs. In particular, running `update` will properly re-resolve the dynamic revision.
3. Scala jars are resolved using the same repositories and configuration as other dependencies.
4. Classloaders (currently, Scala classloaders) are cached by the timestamps of entries instead of Scala class loaders being cached by version.
TODO: Support external dependency configuration
Changed the order of parameters in getScala method.
Changed the key name to scalaOrganization (scala-organization).
Augmented description of the key.
Minor fixes.
Adding scalaOrg key that specifies organization (artifactId) of scala used in the project. The change does not affect version checks for dependecies and LauncherConfiguration.
Modified scalaProvider cache in Launcher to use (scalaOrg, version) as a key.
Downloaded jars are stored in the folder scala-.../lig-<scalaOrg> if scalaOrg is not default.
scala-org is an advanced setting so it can not be used in build.sbt.
Jason Zaugg
Guillaume Martres
Pierre DAL-PRA
henry
Josh Suereth
Eugene Yokota
Mark Harrah
Grzegorz Kossakowski
Peter Vlugter
Vojin Jovanovic
Vojin Jovanovic