There is a large fixed overhead of about O(7minutes) per travis run. We
seem to be limited to five build agents at a time so I consolidated the
.travis.yml into five builds. I tried to distribute the tasks fairly
evenly so that all of the builds take about the same amount of time. On
my personal travis build, where I only get three build agents, the total
run time of the build dropped to 50 minutes from one hour forty minutes.
This check doesn't actually make sense anymore with the new
ClassLoaderCache. In the old ClassLoaderCache, there were separate
layers for the snapshots and regular jars. The test was verifying that
only the snapshot layer was invalidated but now there is just one layer.
The TestDependencies strategy doesn't work as of 1.3.0-RC1 because some
of the tests need to access resources. The ScalaLibrary strategy is
broken in 1.3.0-RC1. That leaves Flat. I tested that with 1.3.0-SNAPSHOT
off the latest master, that the AllLibraryJars strategy works fine, but
I need this change now to get travis to work.
This new versions layers the classpath so that the
test-interface-1.0.jar appears before the scala library. This allows us
to bypass reconstructing the AppConfiguration at startup which reduces
the metaspace utilization of sbt.
In an interactive session, it's possible for task evaluation to trigger
an OOM: Metaspace but for sbt to continue working after that failure.
Moreover, the metaspace oom can be caused by using a dependency
classloader layer. If the user changes the layering strategy, they may
be able to re-run their command successfully.
The dotty sbt-bridge module assumes that it's going to get a
URLClassLoader from which it can extract all of the classpath urls. That
doesn't work with the old wrapped classloader because its classpath was
empty. As a nasty workaround, I override the getURLs method, which is
where it gets the URLs from. After this change the
compiler-project/dotty-compiler-plugin test passes.
Instead of caching based on the classpath of the resources, we should
instead cache based on the actual resource files. This commit achieves
that by adding the classpathFiles key which just transforms the
attributed classpath to a Seq[Path]. This implicitly generates the
outputFileStamps key for classpathFiles which we can use to read the
stamps (the file stamp entries for the classpath should get filled by
the compile task so this shouldn't actually cause any additional io).
The ShareRuntimeDependenciesLayerWithTestDependencies strategy doesn't
really work with resources, so it makes sense to get rid of it. Without
the share layer, there is no point in having separate
RuntimeDependencies and TestDependencies layers so I consolidated them
to Dependencies.
If we really care about binary/source compatibility for the 1.3.0-RCx
series, I can restore the traits and objects and set them private[sbt].
I think it was kind of a bug that they existed at all given the issue
with resources so it makes sense to just remove them.
This makes debugging a bit easier in the eclipse memory analyzer tool
since we get a more specific classloader type than URLClassLoader and by
giving the class a meaningful name, we can tell from where it
originated.
This commit removes the ClassLoaderCache that I'd added for the purpose
of caching layered classloaders. Instead, we will use the state's global
ClassLoaderCache. This is better both because it centralizes the
classloader caching and because the new ClassLoaderCache will evict
unused classloaders when the jvm is under memory pressure.
I also add a new layer for the resources that goes between the scala
library layer and the dependency layer. This should help in cases where
users depend on libraries that require access to resources, e.g.
logback.xml.
In sbt 1.3.0-RC1, when the dependency layers are used, the dependencies
cannot see the project resources. This can be problematic when the
library requires certain files like logback.xml to work. This introduces
a scripted test for this case.
The next commit should fix this test.
It was possible to make new classloaders for the scala library and other
jars with each new scala instance. To avoid this, I audited all of the
places within sbt where we make a ScalaInstance and ensure that we
instantiate them in such a way that the classloaders are retrieved
through the state's ClassLoaderCache.
After this change, I found from a heap dump that it was possible to run
test in a project that uses scala 2.12.8 and have only ONE classloader
for the scala library present in the heap dump. With older versions,
there were would be up to 3 or 4 in most heap dumps.
This commit adds a new ClassLoaderCache that builds on the
ClassLoaderCache that is present in zinc (and can be used to build an
instance of the zinc ClassLoaderCache to preserve compatibility). It
differs from the zinc classloader cache that it does not use direct
SoftReferences to classloaders. Instead, we create a wrapper loader
that can't load any classes and just delegates to its parent. This
allows us to add a thread that reaps the soft reference to the wrapper
loader. Crucially, we add a custom SoftReference class that has a strong
reference to the underlying classloader. This allows us to call close on
the strong reference.
The one issue with this approach is that we can't
rescue the jvm from crashing with an OOM: metaspace because the jvm
doesn't give us a chance to close and dereference the underlying
classloaders before it crashes. It WILL collect classloaders under
normal memory pressure, just not metaspace pressure. To fix this, I
check if the MaxMetaspaceSize is set via an MxBean and, if it is, we
fill the cache with regular soft references. We are going to change the
bash script to not set -XX:MaxMetaspaceSize by default so most builds
should probably end up correctly closing the classloaders after this
change. But we should break existing builds that set MaxMetaspaceSize
but don't crash.
As part of this commit, I audited all of the places where we were
instantiating ClassLoaderCache instances and instead pass in the
state's ClassLoaderCache instance. This reduces the total number of
classloaders created.
Using a lazy val causes the log manager to hold onto a reference to the
state. These would accumulate with each task evaluation. I found that
that in the beanpuree project, that if I ran compile 10 times in a row,
the heap usage was 40mb lower after this change.
This was problematic because it had no dependency on the compile task
which meant that any other task in the config would pick up those
fileOutputs which did not make sense. I noticed this because
(resources / outputFileStamps).value would include class files.
If a test threw a NoClassDefFoundError, it was uncaught and the user got
an obscure error. After this change, they will be warned that the
classloader layering strategy may be at fault and will get instructions
on how to fix it.
To minimize classloading and consistency between sbt instances launched
with the latest launcher compared to old launchers, I overhauled code
that replaces the app configuration and meta build classloader at
startup. The goals of this change for legacy launchers were:
1) Do not ever load the scala-library.jar from the app provider class loader.
2) Close the class loaders that are below the topLoader in the class
loading hierarcy
For the new launcher, we simply want to avoid modifying the loader at
all.
I added the SbtParserInit class so that it was more straightforward to
preload the global instance using reflection. We now use reflection to
instantiate an SbtParserInit instance for both the legacy and new
launcher cases to simplify the logic.
After this change, the legacy loader still uses somewhat more metaspace
than the new loader, but the difference seems to be O(10MB), which
should only impact projects that were close their MaxMetaspaceSize to
begin with.
I verified using javap that none of the code in this class uses the
scala standard library which should help metaspace since we don't load
much of the scala standard library until we enter xMainImpl.run.
I verified manually that ExternalHooks were still applied by default but
that I could set the incOptions in the Test and Compile configs so that
they weren't used.
Fixes#4624
Fixes#4712
This adds a specialized DependencyResolution instance called `scalaCompilerBridgeDependencyResolution` to download the compiler bridge. It has its own list of resolvers set by `scalaCompilerBridgeResolvers`. For backward compatibility, it will append `externalResolvers.value` as well.
Ethan Atkins
eugene yokota
Guillaume Martres