Issue #5974 reported that ctrl+d was not working as expected in the
scala 2.13.{2,3} console. This was because we were throwing an exception
whenever ctrl+d was read instead of allowing the jline line reader to
handle it. I can't remember exactly why I added the throw here but I
validated that both the sbt shell and the scala console had the expected
behavior from the comments in #5974 after this change.
When piping an sbt task to a file, the expectation is that sbt will not
write ansi control characters and colors unless the users specifies that
with, e.g. -D.sbt.color=true. With sbt 1.4.0, all output bytes are
routed through the progress output processor which tries to ensure that
progress lines are not interleaved with log lines. The issue was that
the hasProgress flag was being set to true for the server process even
when the formatEnabledInEnv flag was set to false. This caused each log
line to have a leading clear screen before cursor ansi control code
written which would appear in the output file.
When running sbt 1.4.0 with -Dsbt.ci=true and -Dsbt.color=true, there is
no color output. This was because in this scenario, a SimpleTerminal was
used and isAnsiSupported and isColorEnabled were hardcoded to false
rather than reading the values from the system properties.
The user input is not echoed in sbt new because we were switching to raw
mode when creating the console terminal. I can't quite remember why I
was entering raw mode preemptively but it doesn't seem like the best
idea.
In windows, it is necessary for the console mode to include the
ENABLE_PROCESS_INPUT flag in order for ctrl+c to be treated as a signal
rather than a character. In jline 2, the ENABLE_PROCESS_INPUT flag was
preserved whenever the console mode was changed but in jline 3, it was
not. It is easy enough to manually set the flag after entering and
exiting raw mode and setting attributes (which are the apis that change
the console mode on windows).
When sbt is starting up and there is an error with the build loading, we
need to read input from the user to determine to restart the build or
not. What is tricky is that there are potentially two sources of input:
thin clients connected through the boot server socket and the actual sbt
console process. If there are not connected thin clients and no system
console is available, we should return -1 in System.in.read, which will
cause sbt to exit.
On my computer with jEdit, input is not read because the console input
stream was overridden to be a blocking input stream that never returned
input. While debugging the server test hangs, I added some logic to
prevent us from trying to read from System.in when System.console ==
null but I now realize that was a mistake because of situations like
ides or editors launching an sbt shell.
I noticed that when using the scala 2.12 console with the thin client
that there was weird behavior for the first few seconds of the session.
When prompted with 'scala> ' I would type a letter, say v, and the
output would be 'scala>v' instead of 'scala> v'. It turned out that this
was because the NetworkChannel was returning a stale value for
isEchoEnabled. This happened because NetworkChannel has a method
getProperties that is rate limited under the assumption that the
properties rarely change. This made sense for things like
isAnsiSupported or isSuperShellEnabled but not isEchoEnabled. It is
straightforward to fix this by actually getting the terminal attributes
and checking if the echo flag is set.
The normal ansi escape sequence for the end key is \u001B[4~ not
\u001B[4!. This didn't seem to matter in terms of whether or not the
end key worked but it still is worth changing for consistency. Pointed
out in an issue comment in jline 3
(https://github.com/jline/jline3/issues/578#issuecomment-699834705).
If there is no console attached, it doesn't make sense to enter or exit
raw mode. We also don't want to poll from System.in in CI in
SimpleTerminal because threads can get blocked trying to read from
System.in with no possibility of exiting.
It can be useful for plugin and build authors to have access to some of
the virtual terminal properties. For instance, when writing a task that
needs a password, the author may wish to put the terminal in raw mode
with echo disabled. This commit introduces a new Terminal trait at the
sbt level and a corresponding task, terminal, that provides a basic
terminal api. The Terminal returned by the terminal task will correspond
to the terminal that initiated the task so that it should work with sbtn
as well as in console mode.
When sbt is running a background process that logs to stdout, the output
can be inadvertently deleted before it has been printed. When the user
is in the prompt state and a log message comes in, we want to delete the
prompt before we print the log. The problem is println is often
implemented with a write of the content followed by a second write of
the system line separator. When that happened, we would print the
content and then immediately delete it when the newline came in. The fix
is to not clear the prompt if there are any bytes that have been written
without a newline, which was tracked by the currentLineBytes variable.
In a continuous build with rapid triggers, it's possible for the input
thread to be interrupted before it has completed the transition into raw
mode. This isn't a big deal because it will be reset when we enter the
next build or when we re-enter the shell.
The jansi terminal is a little nicer than the jna terminal in windows in
my opinion. In the jna terminal, tab completions use overscores instead
of inverting the colors for the highlighted completion string.
The ansi.caps file doesn't contain any bindings for delete, end or
insert keys. To work around that, we specifiy the defaults in the
WindowsInputStream and override the terminal's getStringCapability
method in the wrapped jline 3 terminal that we pass to the jline 3 line
reader.
The keyState variable that I had taken from the jline 3
AbstractWindowsTerminal was not correctly set which caused the jline
reader to interpret a number of system key events as random strings
rather than escape codes. The arrow keys, for example, did not work.
After setting the keyState variable
There were a number of issues with swithcing between raw and canonical
issues that affected both the server and the thin client. These were
reported in #5863 and #5856. In both cases, there were issues with
reading input or having the input be displayed. Debugging those issues
revealed a number of issues with how we were using the jline 3 system
terminal and the hybrid interaction with the jline 2 terminal. This
commit eliminates all of our internal jline 2 usage. The only remaining
jline 2 usage is that we create and override the global terminal for the
scala console for scala versions < 2.13. By moving away from jline 2, I
was also able to fix#5828, which reported that the home, end and delete
keys were not working.
One of the big issues that this commit addresses is that the
NetworkClient was always performing blocking reads on System.in. This
was problematic because it turns out that you can't switch between raw
and canonical modes when there is a read present. To fix this, the
server now sends a message to the client when it wants to read bytes and
only then does the client create a background thread to read a single
byte.
I also figured out how to set the terminal type properly for the thin
client on windows where we had been manually setting the capabilities to
ansi, which only worked for some keys. This fix required switching to
the WindowsInputStream that I introduced in a prior commit. Before we
were using the jline 2 wrapped input stream which was converting some
system events, like home and end, to the wrong escape sequence mappings.
The remainder of the commit is mostly just converting from jline 2 apis
to jline 3 apis.
I verified that tab completions, the scala console, the ammonite console
and a run task that read from System.in all work with both the server
and the thin client on mac, linux and windows after these changes.
Fixes#5828, #5863, #5856
The old sbt launcher uses jansi 1.11, which is incompatible with jline3.
To work around this, we can use the jna terminal implementation for the
jline system terminal. This commit also switches to using the jline
TerminalBuilder for all system terminals except for the windows system
terminal with the thin client. The jline terminal builder uses
reflection that is difficult to make work with the thin client and it is
much easier to just manually construct the thin client. This is only
necessary for windows because on posix the thin client will fall back to
an implementation that shells out for stty commands.
Reading directly from System.in in windows when in raw mode does not
report special key events like arrows. Previously these keys had worked
because we had been using jline.Terminal.wrapInputIfNeed(System.in).
That jline 2 api unfortunately translated some windows control
characters to the wrong ansi escape sequences which made some keys, like
home and end, not work. This commit introduces a new class
WindowsInputStream that has similar functionality to the
wrapInputIfNeedeed jline api. I initially copied the implementation from
jline.WindowsTerminal and then I copied the escape sequence handling
from AbstractWindowsTerminal in jline3. With this input stream, all of
the keys work as expected on my windows vm and canonical input works as
well.
I'm not sure why the scala.Console.withIn was using proxyInputStream
while the other input streams were set to wrapedSystemIn. At any rate,
using the wrappedSystemIn would likely prevent reading input from
working with the thin client.
In sbt 1.4.0-RC1, if a user ran `sbt console`, the progress lines would
be printed after they had entered the console. This was because the
prompt state was incorrect. To get the prompt in the correct state, we
initialize the prompt to batch and then switch to pending when either
sbt enters the shell or the network client attaches in interactive mode.
We also will now immediately print progress as soon as we enter a skip
task to clear out the progress lines and display the warning about a
running task if there is another client connected while the task is
running.
The SimpleTerminal is used when sbt is run with -Dsbt.log.noformat=true.
There is no reason to disable success messages by default.
Fixes https://github.com/sbt/sbt/issues/5861
When sbt is run with the sbt.log.noformat system property set to true,
no virtual io is used which causes the jline 3 terminal that we creaate
to not work at all. For reasons that I don't understand, it is also
necessary to set the jline.terminal system property to none to make the
dumb terminal work.
A user of 1.4.0-RC1 reported that colors were not being displayed on
fedora 32. I'm not sure if this will fix the issue with fedora, but I
found it confusing how formatEnabledInEnv was set so I refactored things
in a way that I found more clear. I verified that things worked as
expected with -Dsbt.color={true,false} and with -Dsbt.log.format and
-Dsbt.log.noformat.
The RelayAppender should not log directly to console out since it is
supposed to be relaying json log messages to connected clients. This was
manifesting as double printing on some success messages.
When the server is running a command with a long name on behalf of a
client, we truncate the command if it exceeds the length of the
terminal. This is because some of the bsp commands are very long.
Nevertheless, only taking 10 characters was a bit too aggressive.
There was a reddit comment that the user's tty was messed up after they
exited sbt:
https://www.reddit.com/r/scala/comments/io3z2p/sbt_140rc1_released/.
This attempts to fix that issue by restoring the terminal before
exiting. In order to ensure the tty is restored, it's necessary to move
the work off of a background thread and delay sbt exit. This does take
about 150ms on my machine but I figure that isn't a huge deal in the
scheme of things.
When starting sbt via the thin client with 1.4.0-RC1, there is no output
until sbt finishes booting up which is poor ux. The reason is that sbt
only uses virtual io when sbt.io.virtual == true or formatEnabledInEnv
== true and not ci. The default value for formatEnabledInEnv is set
based on whether color is enabled in the environment. This had copied
old logic that turned on color if ansi was enabled but it makes more
sense to check the color property (which is set by the thin client via
an environment variable when it launches sbt) and fall back to whether
or not java.lang.System.console is defined. We also can explicitly set
"-Dsbt.io.virtual=true" when the thin client launches sbt since the thin
client relies on this behavior. By doing it in both places, the sbtn
for 1.4.0-RC1 will display boot output for newer versions of sbt.
Bonus: don't call ConsoleAppender.formatEnabledInEnv which just calls
back to Terminal.formatEnabledInEnv
When the number of tasks running exceeds the limit for the number of
progress tasks to display by 1. Say the limit is 2 and there are three
tasks running, then we display
| ... (1 other tasks)
| => foo 2s
| => bar 3s
This looks bad considering we could just display what the task actually is.
The akka-http project doesn't load because it gets an Appender with
MainAppender.defaultBacked which returns an sbt Appender rather than a
log4j appender now. It then passes that appender into
bindLoggerAppenders which doesn't work because bindLoggerAppenders was
expecting a log4j appender rather than an sbt Appender..
Intellij invokes sbt with "-Djline.terminal=jline.UnsupportedTerminal"
which Terminal rewrites to the value none. When that property is set, we
should be using a jline dumb terminal. While
https://github.com/sbt/sbt/pull/5788 did fix the import functionality,
jline 3 was still emitting some ansi characters to the intellij console.
When we feed a dumb terminal to the jline 3 line reader, the ansi
control characters go away.
On linux and mac, entering ctrl+c will automatically kill any forked
processes that were created by the sbt server because sigint is
automatically forwarded to the child process. This is not the case on
windows where it is necessary to forcibly kill these processes.
On windows with jline3, inputting ctrl+c in the sbt console just causes
the input stream to return -3 unlike mac and linux where ctrl+c always
signals.
Fixes https://github.com/sbt/sbt/issues/5791
Ethan Atkins
xuwei-k
eugene yokota