For reasons I don't fully understand, it is necessary to go in and out
of raw mode to get console input to echo on windows. This was reported
as sbt new not echoing input in #5952.
When the read methods of InputStream that take an Array[Byte] as input
are called, they are supposed to return -1 if there are no bytes
available due to an EOF. Previously it was incorrectly writing the -1 as
a byte in the array and returning 1. Now it correctly returns -1.
The condition for closing the WriteableInputStream was also incorrect.
We should only close the input stream if it returns -1 in raw mode.
Fixes#5999
This channge should turn color on by default in ci builds (see
https://github.com/sbt/sbt/issues/5269). It can still be disabled with
-Dsbt.color=false.
Disabling virtual io prevents the thin client from working. It is
possible that a user may manually start a server with -Dsbt.color=false
but still want to connect a thin client.
Running sbt with -Dsbt.supershell=true and -Dsbt.color=true doesn't work
well because the isAnsiSupported flag is set to false when
-Dsbt.color=false. This causes the processing that ensures that
supershell lines do not get interleaved with normal log lines to be
skipped. To fix this, we can enable ansi codes when supershell is
explicitly enabled.
The ConsoleAppender formatEnabledInEnv field was being used both as an
indicator that ansi codes were supported and that color codes are
enabled. There are cases in which general ansi codes are not supported
but color codes are and these use cases need to be handled separately.
To make things more explicit, this commit adds isColorEnabled and
isAnsiSupported to the Terminal companion object so that we can be more
specific about what the requirements are (general ansi escape codes or
just colors). There are a few cases in ConsoleAppender itself where
formatEnabledInEnv was used to set flags for both color and ansi codes.
When that is the case, we use Terminal.isAnsiSupported because when that
is true, colors should at least work but there are terminals that
support color but not general ansi escape codes.
It is currently the case that all tagged log messages going through a
ConsoleAppender have color codes added and then subsequently stripped
out. This didn't work well in combination with -Dsbt.color=true and
-Dsbt.ci=true because the -Dsbt.color=true was causing
ConsoleAppender.formatEnabledInEnv to be set to true which caused ansi
codes to be enabled. Travis CI supports color codes but not other ansi
escape sequences (this is also often true of embedded shells like in
intellij or jedit). This commit reworks the ConsoleAppender so that we
only add colors if useFormat is enable dand only add
ClearScreenAfterCursor if ansi codes are supported. It also reworks the
ConsoleAppender.write method so that if useFormat is true but
ansiCodesSupported is false that we only remove the non color related
ansi escape sequences so that colors remain.
In canonical mode, System.in will return -1 for ctrl+d on an empty line.
The result of this behavior was that if a user entered ctrl+d during run
in a task that was reading from System.in, sbt would end up exiting
whenever the task exited. This happened because the WriteableInputStream
would close itself when it read -1 from the input stream, which it
assumed meant that the underlying input stream itself had been closed.
When the jline reader tried to read from the closed
WriteableInputStream, it would throw an exception and if the line reader
was for the console channel, it would be interpreted as the user had
inputted ctrl+d in the sbt shell which is supposed to exit sbt. This
change fixes that behavior so that sbt can continue reading input after
the run task exits.
Issue #5974 reported that ctrl+d was not working as expected in the
scala 2.13.{2,3} console. This was because we were throwing an exception
whenever ctrl+d was read instead of allowing the jline line reader to
handle it. I can't remember exactly why I added the throw here but I
validated that both the sbt shell and the scala console had the expected
behavior from the comments in #5974 after this change.
When piping an sbt task to a file, the expectation is that sbt will not
write ansi control characters and colors unless the users specifies that
with, e.g. -D.sbt.color=true. With sbt 1.4.0, all output bytes are
routed through the progress output processor which tries to ensure that
progress lines are not interleaved with log lines. The issue was that
the hasProgress flag was being set to true for the server process even
when the formatEnabledInEnv flag was set to false. This caused each log
line to have a leading clear screen before cursor ansi control code
written which would appear in the output file.
When running sbt 1.4.0 with -Dsbt.ci=true and -Dsbt.color=true, there is
no color output. This was because in this scenario, a SimpleTerminal was
used and isAnsiSupported and isColorEnabled were hardcoded to false
rather than reading the values from the system properties.
The user input is not echoed in sbt new because we were switching to raw
mode when creating the console terminal. I can't quite remember why I
was entering raw mode preemptively but it doesn't seem like the best
idea.
In windows, it is necessary for the console mode to include the
ENABLE_PROCESS_INPUT flag in order for ctrl+c to be treated as a signal
rather than a character. In jline 2, the ENABLE_PROCESS_INPUT flag was
preserved whenever the console mode was changed but in jline 3, it was
not. It is easy enough to manually set the flag after entering and
exiting raw mode and setting attributes (which are the apis that change
the console mode on windows).
When sbt is starting up and there is an error with the build loading, we
need to read input from the user to determine to restart the build or
not. What is tricky is that there are potentially two sources of input:
thin clients connected through the boot server socket and the actual sbt
console process. If there are not connected thin clients and no system
console is available, we should return -1 in System.in.read, which will
cause sbt to exit.
On my computer with jEdit, input is not read because the console input
stream was overridden to be a blocking input stream that never returned
input. While debugging the server test hangs, I added some logic to
prevent us from trying to read from System.in when System.console ==
null but I now realize that was a mistake because of situations like
ides or editors launching an sbt shell.
I noticed that when using the scala 2.12 console with the thin client
that there was weird behavior for the first few seconds of the session.
When prompted with 'scala> ' I would type a letter, say v, and the
output would be 'scala>v' instead of 'scala> v'. It turned out that this
was because the NetworkChannel was returning a stale value for
isEchoEnabled. This happened because NetworkChannel has a method
getProperties that is rate limited under the assumption that the
properties rarely change. This made sense for things like
isAnsiSupported or isSuperShellEnabled but not isEchoEnabled. It is
straightforward to fix this by actually getting the terminal attributes
and checking if the echo flag is set.
The normal ansi escape sequence for the end key is \u001B[4~ not
\u001B[4!. This didn't seem to matter in terms of whether or not the
end key worked but it still is worth changing for consistency. Pointed
out in an issue comment in jline 3
(https://github.com/jline/jline3/issues/578#issuecomment-699834705).
If there is no console attached, it doesn't make sense to enter or exit
raw mode. We also don't want to poll from System.in in CI in
SimpleTerminal because threads can get blocked trying to read from
System.in with no possibility of exiting.
It can be useful for plugin and build authors to have access to some of
the virtual terminal properties. For instance, when writing a task that
needs a password, the author may wish to put the terminal in raw mode
with echo disabled. This commit introduces a new Terminal trait at the
sbt level and a corresponding task, terminal, that provides a basic
terminal api. The Terminal returned by the terminal task will correspond
to the terminal that initiated the task so that it should work with sbtn
as well as in console mode.
When sbt is running a background process that logs to stdout, the output
can be inadvertently deleted before it has been printed. When the user
is in the prompt state and a log message comes in, we want to delete the
prompt before we print the log. The problem is println is often
implemented with a write of the content followed by a second write of
the system line separator. When that happened, we would print the
content and then immediately delete it when the newline came in. The fix
is to not clear the prompt if there are any bytes that have been written
without a newline, which was tracked by the currentLineBytes variable.
In a continuous build with rapid triggers, it's possible for the input
thread to be interrupted before it has completed the transition into raw
mode. This isn't a big deal because it will be reset when we enter the
next build or when we re-enter the shell.
The jansi terminal is a little nicer than the jna terminal in windows in
my opinion. In the jna terminal, tab completions use overscores instead
of inverting the colors for the highlighted completion string.
The ansi.caps file doesn't contain any bindings for delete, end or
insert keys. To work around that, we specifiy the defaults in the
WindowsInputStream and override the terminal's getStringCapability
method in the wrapped jline 3 terminal that we pass to the jline 3 line
reader.
The keyState variable that I had taken from the jline 3
AbstractWindowsTerminal was not correctly set which caused the jline
reader to interpret a number of system key events as random strings
rather than escape codes. The arrow keys, for example, did not work.
After setting the keyState variable
There were a number of issues with swithcing between raw and canonical
issues that affected both the server and the thin client. These were
reported in #5863 and #5856. In both cases, there were issues with
reading input or having the input be displayed. Debugging those issues
revealed a number of issues with how we were using the jline 3 system
terminal and the hybrid interaction with the jline 2 terminal. This
commit eliminates all of our internal jline 2 usage. The only remaining
jline 2 usage is that we create and override the global terminal for the
scala console for scala versions < 2.13. By moving away from jline 2, I
was also able to fix#5828, which reported that the home, end and delete
keys were not working.
One of the big issues that this commit addresses is that the
NetworkClient was always performing blocking reads on System.in. This
was problematic because it turns out that you can't switch between raw
and canonical modes when there is a read present. To fix this, the
server now sends a message to the client when it wants to read bytes and
only then does the client create a background thread to read a single
byte.
I also figured out how to set the terminal type properly for the thin
client on windows where we had been manually setting the capabilities to
ansi, which only worked for some keys. This fix required switching to
the WindowsInputStream that I introduced in a prior commit. Before we
were using the jline 2 wrapped input stream which was converting some
system events, like home and end, to the wrong escape sequence mappings.
The remainder of the commit is mostly just converting from jline 2 apis
to jline 3 apis.
I verified that tab completions, the scala console, the ammonite console
and a run task that read from System.in all work with both the server
and the thin client on mac, linux and windows after these changes.
Fixes#5828, #5863, #5856
The old sbt launcher uses jansi 1.11, which is incompatible with jline3.
To work around this, we can use the jna terminal implementation for the
jline system terminal. This commit also switches to using the jline
TerminalBuilder for all system terminals except for the windows system
terminal with the thin client. The jline terminal builder uses
reflection that is difficult to make work with the thin client and it is
much easier to just manually construct the thin client. This is only
necessary for windows because on posix the thin client will fall back to
an implementation that shells out for stty commands.
Reading directly from System.in in windows when in raw mode does not
report special key events like arrows. Previously these keys had worked
because we had been using jline.Terminal.wrapInputIfNeed(System.in).
That jline 2 api unfortunately translated some windows control
characters to the wrong ansi escape sequences which made some keys, like
home and end, not work. This commit introduces a new class
WindowsInputStream that has similar functionality to the
wrapInputIfNeedeed jline api. I initially copied the implementation from
jline.WindowsTerminal and then I copied the escape sequence handling
from AbstractWindowsTerminal in jline3. With this input stream, all of
the keys work as expected on my windows vm and canonical input works as
well.
I'm not sure why the scala.Console.withIn was using proxyInputStream
while the other input streams were set to wrapedSystemIn. At any rate,
using the wrappedSystemIn would likely prevent reading input from
working with the thin client.
In sbt 1.4.0-RC1, if a user ran `sbt console`, the progress lines would
be printed after they had entered the console. This was because the
prompt state was incorrect. To get the prompt in the correct state, we
initialize the prompt to batch and then switch to pending when either
sbt enters the shell or the network client attaches in interactive mode.
We also will now immediately print progress as soon as we enter a skip
task to clear out the progress lines and display the warning about a
running task if there is another client connected while the task is
running.
The SimpleTerminal is used when sbt is run with -Dsbt.log.noformat=true.
There is no reason to disable success messages by default.
Fixes https://github.com/sbt/sbt/issues/5861
When sbt is run with the sbt.log.noformat system property set to true,
no virtual io is used which causes the jline 3 terminal that we creaate
to not work at all. For reasons that I don't understand, it is also
necessary to set the jline.terminal system property to none to make the
dumb terminal work.
A user of 1.4.0-RC1 reported that colors were not being displayed on
fedora 32. I'm not sure if this will fix the issue with fedora, but I
found it confusing how formatEnabledInEnv was set so I refactored things
in a way that I found more clear. I verified that things worked as
expected with -Dsbt.color={true,false} and with -Dsbt.log.format and
-Dsbt.log.noformat.
The RelayAppender should not log directly to console out since it is
supposed to be relaying json log messages to connected clients. This was
manifesting as double printing on some success messages.
When the server is running a command with a long name on behalf of a
client, we truncate the command if it exceeds the length of the
terminal. This is because some of the bsp commands are very long.
Nevertheless, only taking 10 characters was a bit too aggressive.
There was a reddit comment that the user's tty was messed up after they
exited sbt:
https://www.reddit.com/r/scala/comments/io3z2p/sbt_140rc1_released/.
This attempts to fix that issue by restoring the terminal before
exiting. In order to ensure the tty is restored, it's necessary to move
the work off of a background thread and delay sbt exit. This does take
about 150ms on my machine but I figure that isn't a huge deal in the
scheme of things.
When starting sbt via the thin client with 1.4.0-RC1, there is no output
until sbt finishes booting up which is poor ux. The reason is that sbt
only uses virtual io when sbt.io.virtual == true or formatEnabledInEnv
== true and not ci. The default value for formatEnabledInEnv is set
based on whether color is enabled in the environment. This had copied
old logic that turned on color if ansi was enabled but it makes more
sense to check the color property (which is set by the thin client via
an environment variable when it launches sbt) and fall back to whether
or not java.lang.System.console is defined. We also can explicitly set
"-Dsbt.io.virtual=true" when the thin client launches sbt since the thin
client relies on this behavior. By doing it in both places, the sbtn
for 1.4.0-RC1 will display boot output for newer versions of sbt.
Bonus: don't call ConsoleAppender.formatEnabledInEnv which just calls
back to Terminal.formatEnabledInEnv
Ethan Atkins
eugene yokota