If the sbt snapshot builds are cached, then we don't actually dogfood
sbt when we publish a local sbt version because sbt will load with the
version from the last build
Moving forward, it would be great for sbt to use the latest code to run
the scripted tests. This could help detect regressions that aren't
noticed by manual dogfooding. We can do this by first publishing
1.4.0-SNAPSHOT binaries before running tests. I didn't want to do this
for all of the builds because it does make each build that publishes
binaries take about a minute longer. The last build in the matrix is
much faster than the other builds though so I made just that build use
the snapshot version. Since that build almost always finishes first,
this should have little effect on the actual time to run ci even though
the cpu time will increase a bit.
The travis tty does not work with virtual io. We should disable virtual
io by default when sbt detects that it's being run in CI but until we
release 1.4.0-M3 or 1.4.0-RC1, we need this flag to fix the travis
output.
The swoval javafmt plugin uses the google java formatter (which I
believe is the only widely used java formatter) to format source files.
It does not provide an automatic javafmtOnCompile method like the
scalafmt plugin so I had to manually implement that functionality. In
general the java formatter is much faster than scalafmt so the impact of
having javafmtOnCompile set to true is very low.
The existing .travis.yml in the util project runs:
mimaReportBinaryIssues
scalafmtCheckAll
whitesourceCheckPolicies
test
The first three are already handled by the first build in the matrix. I
add the util tests to the first jdk11 build and the jdk8 build.
There is a large fixed overhead of about O(7minutes) per travis run. We
seem to be limited to five build agents at a time so I consolidated the
.travis.yml into five builds. I tried to distribute the tasks fairly
evenly so that all of the builds take about the same amount of time. On
my personal travis build, where I only get three build agents, the total
run time of the build dropped to 50 minutes from one hour forty minutes.
The supershell output is distracting in CI. I added a system property,
sbt.ci, to explicitly set whether or not sbt is running in a ci build.
It was not at all obvious to me if the BUILD_NUMBER or CI environment
variables were set on travis or appveyor.
Normally I'd include these with the previous commit, but the diff is so
large that I put them in their own commit. The tests handle 5 scenarios:
1) akka-actor-system -- a project that has Akka as a dependency and a
simple main method that creates and terminates an ActorSystem. What
is interesting about this test is that if scriptedBufferLog := false,
we notice that the first call to run is slow, but subsequent calls to
run and test are fast. The test does at least ensure that recycling
the runtime layer in test works ok.
2) jni -- verifies that a project with native libraries will be able to
load the library with each run. It actually swaps out the underlying
library so that the it really ensures that the library is reloaded
between runs.
3) library-mismatch -- verifies that the layered classloaders can work
when the test dependencies are incompatible with the runtime
dependencies. In this test, the test dependencies use an api in a
library called foo-lib that isn't available in the version used by
the runtime dependencies. Because of this incompatibility, the test
will not work if Test / layeringStrategy := LayeringStrategy.Full.
4) scalatest -- verifies that a test runs using the scalatest framework
5) utest -- verifies that a test runs using the utest framework
The reason for (4) and (5) is to ensure that both the in sourced test
frameworks and external frameworks work with the new loaders.
whitesourceOnPush calls whitesourceCheckPolicies and whitesourceUpdate on push.
Since Travis CI secrets are not available during PR from a fork, there's no point in calling these during the PR validation.
This commit reworks Watched to be more testable and extensible. It also
adds some small features. The previous implementation presented a number
of challenges:
1) It relied on external side effects to terminate the watch, which was
difficult to test
2) It exposed irrelevant implementation details to the user in the
methods that exposed the WatchState as a parameter.
3) It spun up two worker threads. One was to monitor System.in for user
input. The other was to poll the watch service for events and write
them to a queue. The user input thread actually broke '~console'
because nearly every console session will hit the <enter> key, which
would eventually cause the watch to stop when the user exited the
console.
To address (1), I add the shouldTerminate method to WatchConfig. This
takes the current watch iteration is input and if the function returns
true, the watch will stop.
To address (2), I replace the triggeredMessage and watchingMessage keys
with watchTriggeredMessage and watchStartMessage. The latter two keys
are functions that do not take the WatchState as parameters. Both
functions take the current iteration count as a parameter and the
watchTriggeredMessage also has a parameter for the path that triggered
the build.
To address (3), I stop using the sbt.internal.io.EventMonitor and
instead use the sbt.io.FileEventMonitor. The latter class is similar to
the former except that it's polling method accepts a duration, which may
be finite or infinite) and returns all of the events that occurred since
it was last polled. By adding the ability to poll for a finite amount of
time, we can interleave polling for events with polling System.in for
user input, all on the main thread. This eliminates the two extraneous
threads and fixes the '~console' use case I described before.
I also let the user configure the function that reads from System.in via
the watchHandleInput method. In fact, this method need not read from
System.in at all since it's just () => Watched.Action. The reason that
it isn't () => Boolean is that I'd like to leave open the option for the
ability to trigger a build via user input, not just terminating the
watch. My initial idea was to add the ability to type 'r' to re-build in
addition to <enter> to exit. This doesn't work without integrating
jline though because the input is buffered. Regardless, for testing
purposes, it gives us the ability to add a timeout to the watch by
making handleInput return true when a deadline expires.
The tests are a bit wonky because I still need to rely on side effects
in the logging methods to orchestrate the sequence of file events that
I'd like to test. While I could move some of this logic into a
background thread, there still needs to be coordination between the
state of the watch and the background thread. I think it's easier to
reason about when all of the work occurs on the same thread, even if it
makes these user provided functions impure.
I deprecated all of the previous watch related keys that are no longer
used with the new infrastructure. To avoid breaking existing builds, I
make the watchConfig task use the deprecated logging methods if they are
defined in the user's builds, but sbt will not longer set the default
values. For the vast majority of users, it should be straightforward to
migrate their builds to use the new keys. My hunch is that the of the
deprecated keys, only triggeredMessage is widely used (in conjunction
with the clear screen method) and it is dead simple to replace it with
watchTriggeredMessage.
Note: The FileTreeViewConfig class is not really necessary for this commit.
It will become more important in a subsequent commit which introduces an
optional global file system cache.
The previous scalafmt plugin had two problems:
* Caching with Coursier did not work correctly
* It failed after the upgrade to 1.0 in my computer (from a clean fork):
```
[error] (testingProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (runProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (taskProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (stdTaskProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (actionsProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (protocolProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (commandProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (mainSettingsProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (mainProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (sbtProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (scriptedPluginProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] (scriptedSbtProj/compile:scalafmtInc) java.lang.NoClassDefFoundError: Could not initialize class scala.meta.package$
[error] Total time: 19 s, completed May 24, 2017 10:44:56 AM
```
This commit replaces the previous scalafmt integration by the one
created by Lucidsoftware, big shoutout! (/cc @pauldraper)
https://github.com/lucidsoftware/neo-sbt-scalafmt
Some plugins remain commented out, for now.
sbt-doge is no longer needed because a variant of it has been folded
into sbt 1.
For some reason scripted requires src/doc jars of sbt, so switch back to
using `publishAll` rather than `publishLocalBinAll`. :(
Also, the sys.prop change in scripted is to force log4j2 to not use a
thread context classloader, and avoid the following:
ERROR StatusLogger Unable to create custom ContextSelector. Falling back to default.
java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.async.AsyncLoggerContextSelector to org.apache.logging.log4j.core.selector.ContextSelector
at java.lang.Class.cast(Class.java:3369)
at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:201)
at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOfProperty(LoaderUtil.java:226)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.createContextSelector(Log4jContextFactory.java:97)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.<init>(Log4jContextFactory.java:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:94)
at org.apache.logging.log4j.spi.ThreadContextMapFactory.createThreadContextMap(ThreadContextMapFactory.java:73)
at org.apache.logging.log4j.ThreadContext.init(ThreadContext.java:223)
at org.apache.logging.log4j.ThreadContext.<clinit>(ThreadContext.java:202)
at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createDefaultInjector(ContextDataInjectorFactory.java:83)
at org.apache.logging.log4j.core.impl.ContextDataInjectorFactory.createInjector(ContextDataInjectorFactory.java:67)
at org.apache.logging.log4j.core.lookup.ContextMapLookup.<init>(ContextMapLookup.java:34)
at org.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:117)
at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:125)
at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:46)
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:650)
at org.apache.logging.log4j.core.layout.PatternLayout.createDefaultLayout(PatternLayout.java:487)
at sbt.internal.util.ConsoleAppender.<init>(ConsoleAppender.scala:245)
at sbt.internal.util.ConsoleAppender$.apply(ConsoleAppender.scala:196)
at sbt.internal.util.ConsoleLogger.<init>(ConsoleAppender.scala:42)
at sbt.internal.util.ConsoleLogger$.apply(ConsoleAppender.scala:34)
at sbt.test.ScriptedRunner.run(ScriptedTests.scala:221)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at Scripted$.doScripted(Scripted.scala:125)
at $0483e89d182e9d240274$.$anonfun$scriptedTask$5(build.sbt:301)
I've tried to put together some scripted tests to remove the overhead of
compiling the whole sbt, which is around 3 minutes every time.
This new structure *should* make the scripted tests run faster.
Aside from this, we do some more tweaks:
* Increase memory.
* Fork processes to compile and run (to see if it makes a difference).
* Pass in the server flag to sbt.
Ethan Atkins
Eugene Yokota
Anil Kumar Myla
Ethan Atkins
Filipe Regadas
Dale Wijnand
jvican