It is not uncommon in large projects for the jvm to silently be running
frequent full gcs in the background. This can slow progress to a crawl.
Usually the fix is to bump the -Xmx parameter but if the users do not
realize that their tasks are slow because of gc thrashing, they may not
think to do that. This PR adds a monitor that hooks into the jvm's event
notification system to keep track of how much time is spent in GC. If
the ratio of the amount of time in gc to the total elapsed time exceeds
some threshold, we emit a warning.
I was motivated to do this because publishLocal can take forever in the
zinc project because a 1G heap isn't big enough.
These tasks show up during task progress and they clutter up the
display. Since my understanding is that both of these tasks are more or
less just waiting for other work to complete, I don't think they are
helpful for debugging.
When the number of tasks running exceeds the limit for the number of
progress tasks to display by 1. Say the limit is 2 and there are three
tasks running, then we display
| ... (1 other tasks)
| => foo 2s
| => bar 3s
This looks bad considering we could just display what the task actually is.
The akka-http project doesn't load because it gets an Appender with
MainAppender.defaultBacked which returns an sbt Appender rather than a
log4j appender now. It then passes that appender into
bindLoggerAppenders which doesn't work because bindLoggerAppenders was
expecting a log4j appender rather than an sbt Appender..
The zinc scripted project depends on all of the compiler bridges. As a
result the introduction of the strict scala binary version check in
f8139da192 broke zinc scripted. This
commit reverts to the old behavior in the non scala sandwich case.
I also switched to a for comprehension instead of a pattern match
because this is a rare case where I think it made the code significantly
more readable.
While in a continuous build, when the user enters ctrl+c into the sbt
server console (not a thin client connection) when sbt has been launched
in interactive mode, the server exits. This commit makes it so that
instead we just cancel the watch. As a result, if sbt was started in
batch mode, e.g. `sbt ~compile`, ctrl+c will still exit sbt but in
interactive mode ctrl+c will take the user back to the shell.
I occassionally end up in a state where watch input does not seem to be
read. To rule out the possibility that the background thread that reads
input has not successfully started, this commit makes it so that we
block until the thread signals that it has started via a CountDownLatch.
The diff is superficially big because of an indentation change at the
bottom.
When forking, a test could be run as many times as its `TestDef` matched
`Fingerprint`s. When forking is disabled, sbt correctly runs the tests
only once.
The sbt Server is initialized with a callback onIncomingSocket. That
callback was created in CommandExchange and held references to a build
structure and a state. Neither the state nor structure would ever go out
of scope so they effectively leaked. It is possible for each
NetworkChannel to access a recent instance of state through the
CommandExchange.withState method. Using this, we can eliminate the
references to state and build structure in the onIncomingSocket
callback. In the sbt project, this reduced the memory utilization by
about 50mb on startup.
Intellij invokes sbt with "-Djline.terminal=jline.UnsupportedTerminal"
which Terminal rewrites to the value none. When that property is set, we
should be using a jline dumb terminal. While
https://github.com/sbt/sbt/pull/5788 did fix the import functionality,
jline 3 was still emitting some ansi characters to the intellij console.
When we feed a dumb terminal to the jline 3 line reader, the ansi
control characters go away.
On linux and mac, entering ctrl+c will automatically kill any forked
processes that were created by the sbt server because sigint is
automatically forwarded to the child process. This is not the case on
windows where it is necessary to forcibly kill these processes.
On windows with jline3, inputting ctrl+c in the sbt console just causes
the input stream to return -3 unlike mac and linux where ctrl+c always
signals.
Fixes https://github.com/sbt/sbt/issues/5791
The intellij import currentlly works by forking an sbt process and
writing command input through the process input stream. To make this
work, we need the SimpleTerminal (which is used when sbt is run with
-Dsbt.log.noformat=true) to be able to read input.
Attaching the input to the simple terminal caused watch tests to fail on
windows. This can be fixed by checking if the byte read from the input
stream is -1 and ignoring it if so.
The sbt.log.noformat parameter should be treated very similarly to
sbt.io.virtual. When it is true, we should just use the raw io streams
for the process. This came up because of
https://github.com/sbt/sbt/issues/5784 which reported that intellij
imports were not working and that ansi control characters were being
written to the output.
There can be race conditions where we try to interrupt and join a ui
thread before it becomes interruptible by blockign on a queue. To
workaround this, we can add the JoinThread class which adds an
extension method Thread.joinFor that takes a FiniteDuration parameter.
This variant of join will repeatedly interrupt and attempt to join the
thread for up to 10 milliseconds before retrying until the limit is
reached. If the limit is reached, we print a noisy error to the console.
I'm not 100% sure if we are leaking threads in the latest sbt version
but this gives me more piece of mind that either we are always
successfully joining the threads or we will be alerted if the joining
fails.
Moving forward, it would be great for sbt to use the latest code to run
the scripted tests. This could help detect regressions that aren't
noticed by manual dogfooding. We can do this by first publishing
1.4.0-SNAPSHOT binaries before running tests. I didn't want to do this
for all of the builds because it does make each build that publishes
binaries take about a minute longer. The last build in the matrix is
much faster than the other builds though so I made just that build use
the snapshot version. Since that build almost always finishes first,
this should have little effect on the actual time to run ci even though
the cpu time will increase a bit.
The CI output logs for the sbt build are truncated when using sbt
1.4.0-M2 because the virtual tty provided by the build agent doesn't
seem to work well with sbt's virtual io.
The travis tty does not work with virtual io. We should disable virtual
io by default when sbt detects that it's being run in CI but until we
release 1.4.0-M3 or 1.4.0-RC1, we need this flag to fix the travis
output.
eugene yokota
Ethan Atkins
Martin Duhem
Adrien Piquerez