When sbt is run with the sbt.log.noformat system property set to true,
no virtual io is used which causes the jline 3 terminal that we creaate
to not work at all. For reasons that I don't understand, it is also
necessary to set the jline.terminal system property to none to make the
dumb terminal work.
A user of 1.4.0-RC1 reported that colors were not being displayed on
fedora 32. I'm not sure if this will fix the issue with fedora, but I
found it confusing how formatEnabledInEnv was set so I refactored things
in a way that I found more clear. I verified that things worked as
expected with -Dsbt.color={true,false} and with -Dsbt.log.format and
-Dsbt.log.noformat.
The RelayAppender should not log directly to console out since it is
supposed to be relaying json log messages to connected clients. This was
manifesting as double printing on some success messages.
When the server is running a command with a long name on behalf of a
client, we truncate the command if it exceeds the length of the
terminal. This is because some of the bsp commands are very long.
Nevertheless, only taking 10 characters was a bit too aggressive.
There was a reddit comment that the user's tty was messed up after they
exited sbt:
https://www.reddit.com/r/scala/comments/io3z2p/sbt_140rc1_released/.
This attempts to fix that issue by restoring the terminal before
exiting. In order to ensure the tty is restored, it's necessary to move
the work off of a background thread and delay sbt exit. This does take
about 150ms on my machine but I figure that isn't a huge deal in the
scheme of things.
When starting sbt via the thin client with 1.4.0-RC1, there is no output
until sbt finishes booting up which is poor ux. The reason is that sbt
only uses virtual io when sbt.io.virtual == true or formatEnabledInEnv
== true and not ci. The default value for formatEnabledInEnv is set
based on whether color is enabled in the environment. This had copied
old logic that turned on color if ansi was enabled but it makes more
sense to check the color property (which is set by the thin client via
an environment variable when it launches sbt) and fall back to whether
or not java.lang.System.console is defined. We also can explicitly set
"-Dsbt.io.virtual=true" when the thin client launches sbt since the thin
client relies on this behavior. By doing it in both places, the sbtn
for 1.4.0-RC1 will display boot output for newer versions of sbt.
Bonus: don't call ConsoleAppender.formatEnabledInEnv which just calls
back to Terminal.formatEnabledInEnv
When the number of tasks running exceeds the limit for the number of
progress tasks to display by 1. Say the limit is 2 and there are three
tasks running, then we display
| ... (1 other tasks)
| => foo 2s
| => bar 3s
This looks bad considering we could just display what the task actually is.
The akka-http project doesn't load because it gets an Appender with
MainAppender.defaultBacked which returns an sbt Appender rather than a
log4j appender now. It then passes that appender into
bindLoggerAppenders which doesn't work because bindLoggerAppenders was
expecting a log4j appender rather than an sbt Appender..
Intellij invokes sbt with "-Djline.terminal=jline.UnsupportedTerminal"
which Terminal rewrites to the value none. When that property is set, we
should be using a jline dumb terminal. While
https://github.com/sbt/sbt/pull/5788 did fix the import functionality,
jline 3 was still emitting some ansi characters to the intellij console.
When we feed a dumb terminal to the jline 3 line reader, the ansi
control characters go away.
On linux and mac, entering ctrl+c will automatically kill any forked
processes that were created by the sbt server because sigint is
automatically forwarded to the child process. This is not the case on
windows where it is necessary to forcibly kill these processes.
On windows with jline3, inputting ctrl+c in the sbt console just causes
the input stream to return -3 unlike mac and linux where ctrl+c always
signals.
Fixes https://github.com/sbt/sbt/issues/5791
The intellij import currentlly works by forking an sbt process and
writing command input through the process input stream. To make this
work, we need the SimpleTerminal (which is used when sbt is run with
-Dsbt.log.noformat=true) to be able to read input.
Attaching the input to the simple terminal caused watch tests to fail on
windows. This can be fixed by checking if the byte read from the input
stream is -1 and ignoring it if so.
The sbt.log.noformat parameter should be treated very similarly to
sbt.io.virtual. When it is true, we should just use the raw io streams
for the process. This came up because of
https://github.com/sbt/sbt/issues/5784 which reported that intellij
imports were not working and that ansi control characters were being
written to the output.
The CI output logs for the sbt build are truncated when using sbt
1.4.0-M2 because the virtual tty provided by the build agent doesn't
seem to work well with sbt's virtual io.
While running ~scripted that multiple instance of the console terminal
were instantiated which caused problems with reading input. It turned
out that RunFromSourceMain was running in the same jvm process as sbt
and creating a new console terminal in a different classloader. This
both messed up the io of scripted tests when scriptedBufferLog was set
to false but it also made it so that I couldn't exit ~ with <enter>. To
workaround this, I deferred initializaiton of the console terminal to
Terminal.withStreams which is guarded by the sbt.io.virtual system
property.
In dogfooding sbt, I found that the WriteableInputStream used by the
console terminal initialized before it was needed. This would lead to
multiple instances of the WriteableInputStream being created, which
could lead to zombie threads reading from stdin. I'm not 100% sure what
the classloading scenario was that caused this to be a problem but in a
few days of using sbt after these changes, I haven't seem zombie
threads.
With the latest sbt snapshot, the ui would get stuck if the user entered
an empty command. They would be presented with an empty prompt and could
not input any commands. This was caused by the change in
d569abe70a that reset the prompt after a
line was read. I had tried to optimize line reading by ignoring empty
commands in UITask.readline so we wouldn't have to make a new thread.
This optimization wasn't really buying much since it only affects how
quickly the user is reprompted after entering an empty command. Unless a
user is spamming the <enter> key, they shouldn't notice a difference.
This commit adds a few options to supershell:
1. Max items -- sets the max number of tasks to display in the progress
reports. It is pretty hard to read more than a few items in the
progress reports so I set the default limit to 8 and made that
configurable via the superShellMaxTasks parameter. If there are more
than the limit, there is an additional line telling how many additional
tasks are running
2. sleep -- sets how long to sleep between reports. The default is 500ms
to ensure that it updates at least once per second but the previous
value of 100ms is more frequent than necessary
3. threshold -- sets the minimum duration a task has to run before being
printed in the progress reports. The default threshold is increased
from 10ms to 100ms. This introduces a delay of threshold milliseconds
before any progress lines appear and also means that if no tasks ever
exceed the threshold, then no progress is ever displayed.
It turns out that task progress actually introduces a fair bit of
overhead. The biggest issue is that the task progress callbacks block
the Execute main thread. This means that time in those callbacks
delays task evaluation, slowing down sbt. This was not negligible, I was
seeing a lot of the total time of a no-op compile in
https://github.com/jtjeferreira/sbt-multi-module-sample was spent in
TaskProgress callbacks. Prior to these changes, I ran 30 no-op compiles
in that project and the average time was about 570ms. This number got
worse and worse because there were memory leaks in the TaskProgress
object. After these changes, it dropped to 250ms and after jit-ing, it
would drop to about 200ms. I also successfully ran 5000 consecutive
no-op compiles without leaking any memory.
A lot of the overhead of task progress was in adding tasks to the
timings map in AbstractTaskProgress. Tasks were never removed and
ConcurrentHashMap insertion time is proportional to the size of the map
(not sure if it's linear, quadratic or other) which was why sbt actually
got slower and slower the longer it ran. Much of the time was spent
adding tasks to the progress timings.
To fix this, I did something similar to what I did to manage logger
state in https://github.com/jtjeferreira/sbt-multi-module-sample. In
MainLoop, we create a new TaskProgress instance before command
evaluation and clean it up after. Earlier I made TaskProgress an object
to try to ensure there was only one progress thread at a time, and that
introduced the memory leak. In addition to removing the leak, I was able
to improve performance by removing tasks from the timings map when they
completed. Unlike TaskTimings and TaskTraceEvent, we don't care about
tasks that have completed for TaskProgress so it is safe to remove them.
In addition to the memory leaks, I also reworked how the background
threads work. Instead of having one thread that sleeps and prints
progress reports, we now use two single threaded executors. One is a
scheduled executor that is used to schedule progress reports and the
other is the actual thread on which the report is generated. When
progress starts, we schedule a recurring report that is generated every
sleep interval until task evaluation completes. Whenever we add a new
task, if we have haven't previously generated a progress report, we
schedule a report in threshold milliseconds. If the task completes
before the threshold period has elapsed, we just cancel the schedule
report. By doing things this way, we reduce the total number of reports
that are generated. Because reports need to effectively lock System.out,
the less we generate them, the better.
I also modified the internal data structures of AbstractTaskProgress so
that there is a single task map of timings instead of one map for
timings and one for active tasks.
It was a bit tricky to reason about the state of the prompt for a
terminal. To help make things more clear, I reworked things so that the
LineReader always sets the prompt to Pending after it reads a command.
In MainLoop, we cache the prompt value and temporarily set it to Running
while the command is running, which is really how it should have always
been.
In order to make the console task work with scala 2.13 and the thin
client, we need to provide a way for the scala repl to use an sbt
provided jline3 terminal instead of the default terminal typically built
by the repl. We also need to put jline 3 higher up in the classloading
hierarchy to ensure that two versions of jline 3 are not loaded (which
makes it impossible to share the sbt terminal with the scala terminal).
One impact of this change is the decoupling of the version of
jline-terminal used by the in process scala console and the version
of jline-terminal specified by the scala version itself. It is possible
to override this by setting the `useScalaReplJLine` flag to true. When
that is set, the scala REPL will run in a fully isolated classloader. That
will ensure that the versions are consistent. It will, however, for sure
break the thin client and may interfere with the embedded shell ui.
As part of this work, I also discovered that jline 3 Terminal.getSize is
very slow. In jline 2, the terminal attributes were automatically cached with a
timeout of, I think, 1 second so it wasn't a big deal to call
Terminal.getAttributes. The getSize method in jline 3 is not cached and
it shells out to run a tty command. This caused a significant
performance regression in sbt because when progress is enabled, we call
Terminal.getSize whenever we log any messages. I added caching of
getSize at the TerminalImpl level to address this. The timeout is 1
second, which seems responsive enough for most use cases. We could also
move the calculation onto a background thread and have it periodically
updated, but that seems like overkill.
There are cases where if the ui state is changing rapidly, that an
AskUserThread can be created and cancelled in a short time windows. This
could cause problems if the AskUserThread is interrupted during
`LineReader.createReader` which I think can shell out to run some
commands so it is relatively slow. If the thread was interrupted during
the call to `LineReader.createReader` and the interruption was not
handled, then the thread would go into `LineReader.readLine`, which
wouldn't exit until the user pressed enter. This ultimately caused the
ui to break until enter because this zombie line reader would be holding
the lock on the terminal input stream.
We should always hold the print stream lock when calling
progressState.write because otherwise the task progress thread could
concurrently write to stdout.
Prior to these changes, sbt was leaking large amounts of memory via
log4j appenders. sbt has an unusual use case for log4j because it
creates many ephemeral loggers while also having a global logger that is
supposed to work for the duration of the sbt session. There is a lot of
shared global state in log4j and properly cleaning up the ephemeral task
appenders would break global logging. This commit fixes the behavior by
introducing an alternate logging implementation. Users can still use the
old log4j logging implementation but it will be off by default. The
internal implementation is very simple: it just blocks the current
thread and writes to all of the appenders. Nevertheless, I found the
performance to be roughly identical to that of log4j in my sample
project. As an experiment, I did the appending on a thread pool and got
a significant performance improvement but I'll defer that to a later PR
since parallel io is harder to reason about.
Background: I was testing sbt performance in
https://github.com/jtjeferreira/sbt-multi-module-sample and noticed that
performance rapidly degraded after I ran compile a few times. I took a
heap dump and it became obvious that sbt was leaking console appenders.
Further investigation revealed that all of the leaking appenders in the
project were coming from task streams. This made me think that the fix
would be to track what loggers were created during task evaluation and
clear them out when task evaluation completed. That almost worked except
that log4j has an internal append only data structure containing logger
names. Since we create unique logger names for each run, that internal
data structure grew without bound. It looked like this could be worked
around by creating a new log4j Configuration (where that data structure
was stored) but while creating new configurations with each task runs
did fix the leak, it also broke global logging, which was using a
different configuration. At this point, I decided to write an alternate
implementation of the appender api where I could be sure that the
appenders were cleaned up without breaking global logging.
Implementation: I made ConsoleAppender a trait and made it no longer
extends log4j AbstractAppender. To do this, I had to remove the one
log4j specific method, append(LogEvent). ConsoleAppender now has a
method toLog4J that, in most cases, will return a log4j Appender that is
almost identical to the Appenders that we previously used. To manage
the loggers created during task evaluation, I introduce a new class,
LoggerContext. The LoggerContext determines which logging backend to use
and keeps track of what appenders and loggers have been created. We can
create a fresh LoggerContext before each task evaluation and clear it
out, cleaning up all of its resources after task evaluation concludes.
In order to make this work, there were many places where we need to
either pass in a LoggerContext or create a new one. The main magic is
happening in the `next(State)` method in Main. This is where we create a
new LoggerContext prior to command evaluation and clean it up after the
evaluation completes.
Users can toggle log4j using the new useLog4J key. They also can set the
system property, sbt.log.uselog4j. The global logger will use the sbt
internal implementation unless the system property is set.
There are a fairly significant number of mima issues since I changed the
type of ConsoleAppender. All of the mima changes were in the
sbt.internal package so I think this should be ok.
Effects: the memory leaks are gone. I successfully ran 5000 no-op
compiles in the sbt-multi-module-sample above with no degradation of
performace. There was a noticeable degradation after 30 no-op compiles
before.
During the refactor, I had to work on TestLogger and in doing so I also
fixed https://github.com/sbt/sbt/issues/4480.
This also should fix https://github.com/sbt/sbt/issues/4773
Rather than relying on a command, I realized it makes more sense to
explicitly set the terminal for the calling channel in MainLoop. By
doing it this way, we can also ensure that we always reset it to the
previous value.
These were not actually used as far as I could tell. The json codecs
cache showed up as taking up 30MB in a heap dump that I took after
running compile 30 times in a clone of the repro project in
https://github.com/sbt/sbt/issues/5508.
Using the scala reflect library always introduces significant
classloading overhead. We can eliminate the classloading overhead by
generating StringTypeTags at compile time instead.
This sped up average project loading time by a few hundred milliseconds
on my computer. The ManagedLoggedReporter in zinc is still using the
type tag based apis but after the next sbt release, we can upgrade the
zinc apis. We also could consider breaking binary compatibility.
JLine 3 automatically appends a space character to the completion
candidate unless you tell it not to by setting its 'complete' parameter.
This behavior is generally nice because it will automatically complete
something like 'foo/testO<TAB>' to 'foo/testOnly ' which allows the user
to start typing the testname without having to enter space. It does,
however, break scripted completions because it will complete
'scripted wat<TAB>' to 'scripted watch/ '
This commit updates the custom completer to append a " " to the initial
completions and check if there are any additional completions available.
If so, we set the complete flag to true and jline will append a space to
the input when the user presses <TAB> or <ENTER>. Otherwise the old
jline2 behavior where no spaces are ever appended is preserved.
In eb688c9ecd, we started buffering output
to the remote client to reduce flickering. This was causing problems
with the output for the thin client in batch mode. With the delay, it
was possible for the client to exit before all of its output had been
displayed.
Bonus: only display aggregation error message if terminal has success
enabled (the thin client displays its own timing message so the message
in aggregation ended up being a duplicate).
The sbt project load made a number of relatively inefficient
transformations of scala collecitons. I went through and found the slow
parts during project loading and made my best attempt at fixing them.
The most significant changes I made were in places using IMap. An IMap
is more or less a wrapper around an immutable Map. It can be much faster
to construct an IMap by creating a java mutable hashmap, wrapping it a
scala Map that delegates to the underlying java hashmap (with a copy on
write if the map is modified) and constructing the IMap from the wrapped
map. It was also in many cases to parallelize some transformations
wherever the order didn't matter.
After applying all of these changes, I found that loading the sbt
project took generally between 8.5 and 9 seconds on my laptop. With
1.3.13, it hovered around 11.5 seconds. I saw a similar speedup in zinc.
The biggest specific improvement was that generating the compiled map
dropped from between 3.5-4 seconds to pretty consistently being around
1.5 seconds.
The more we call flush, the more likely progress status is to blink. To
reduce the amount of calls to flush, we can instead batch all of the
bytes that are going to be written and wriite them all at once. This
change made progress noticeably less blinky in the zinc project running
the latest sbt snapshot (which frankly was almost siezure inducing when
running publishLocal)
Printing a new line was not great ux. You might see something like:
[info] set current project to project (in build file:project)
sbt:project>
[info] new client connected: network-1
sbt:project>
instead of initially
[info] set current project to project (in build file:project)
sbt:project>
and then after the client connects:
[info] set current project to project (in build file:project)
[info] new client connected: network-1
sbt:project>
This reverts commit b1dcf031a5.
I found that b1dcf031a5 had some
unintended consequences that seemed to mess up the prompt state. The
real problem that it was trying to address was that the prompt was being
interleaved with log messages in some scenarios. There was a different
way to fix that in ProgressState that was both simpler and more
reliable.
I noticed that when reloading the build, that certain errors are logged
by sbt to System.err. These were not shown to a thin client because we
weren't forwarding System.err. This change remedies that.
System.err is handled more simply than System.out. We do not put
System.err through the progress state because generally System.err is
tends to be unbuffered. I had hesitated to add System.err to the
Terminal interface at all to give users an escape hatch but I couldn't
get project loading to work well with the thin client without it.
Ethan Atkins
eugene yokota