Network client already supports the -bsp command (since
65ab7c94d0). This commit reworks the
BspClient.run method so that it delegates to the NetworkClient. The
advantage to doing it this way is that improvements to starting up the
sbt server by the thin client will automatically propagate to the -bsp
command. The way that it is implemented, all of the output generated
during server startup will be redirected to System.err which is useful
for debugging without messing up the bsp protocol, which relies on only
bsp messages being written to System.out.
The boot server socket was not working correctly when the sbt server was
started by the thin client. This was because it is necessary for us to
create a ConsoleTerminal in order for System.out and System.err to be
properly forwarded to the clients connected over the boot server socket.
As a result, if you started a server instance of sbt with the thin
client, you wouldn't see any output util you connected to the server.
The fix is to just make sure that we create a console terminal if sbt is
run as a subprocess.
When a batch command is run with the thin client, it logs an info
message that the command completed. This is unnecessary given that
completion is implied by the success or failure method that follows. It
made the output look a little different in the thin client vs the
console.
I saw a stacktrace when exiting sbtn on windows due to an interrupted
exception being thrown during thread joining. We only want to throw this
exception if we didn't successfully join the thread. I also noticed that
we would try to join the thread forever. There was supposed to be a
timelimit so that we would eventually stop blocking even if we were
unable to join the thread. The limit was set but not respected.
With sbtn, the system properties are passed in as regular command
arguments. We need to parse them before we call Terminal.withStreams or
else system properties like -Dsbt.color=false are ignored.
EvaluateTask was holding references to SafeState that could be quite
large. This was reported as #5992. In that project, I ran the `ci` task
and observed the OOM as reported. I took a heap dump prior to OOM and
got the retained size graph from visualvm (which took hours to compute).
The lastEvaluatedState was holding a reference to SafeState that was
1.7GB. The project max heap size was set to 2GB. Instead of using the
lastEvaluatedState, we can just use StandardMain.exchange.withState.
The cached instances of state were used for task cancellation and
completions. While it is possible that early on in booting
StandardMain.exchange.withState could return a null state, in practice
this won't happen because it is set early on during the sbt boot
commands.
After this change, I successfully ran the `ci` task in the #5992 issue
project with the same memory parameters as their ci config.
The ConsoleAppender formatEnabledInEnv field was being used both as an
indicator that ansi codes were supported and that color codes are
enabled. There are cases in which general ansi codes are not supported
but color codes are and these use cases need to be handled separately.
To make things more explicit, this commit adds isColorEnabled and
isAnsiSupported to the Terminal companion object so that we can be more
specific about what the requirements are (general ansi escape codes or
just colors). There are a few cases in ConsoleAppender itself where
formatEnabledInEnv was used to set flags for both color and ansi codes.
When that is the case, we use Terminal.isAnsiSupported because when that
is true, colors should at least work but there are terminals that
support color but not general ansi escape codes.
When project/target is a symbolic link, sbt 1.4.0 crashes on startup
because Files.createDirectories will throw a FileAlreadyExistsException.
The fix is to first check if the target directory exists before trying
to create it.
When sbt is starting up and there is an error with the build loading, we
need to read input from the user to determine to restart the build or
not. What is tricky is that there are potentially two sources of input:
thin clients connected through the boot server socket and the actual sbt
console process. If there are not connected thin clients and no system
console is available, we should return -1 in System.in.read, which will
cause sbt to exit.
A user reported that no prompt was displayed when they used sbt from the
jEdit console. The reason no prompt was displayed was because
System.console was null which caused the ConsoleChannel to be
initialized with its prompt set to NoPrompt. I don't remember why it was
doing this but this bug seems worse than whatever it was trying to
address.
In the client test, the sbt server would keep open the the client
connection even after it had exited because the client was only shutting
down its side of the connection. Since in the test it wasn't exiting the
jvm, the read side of the connection was still open.
It is possible for the test cases in ClientTest to block indefinitely.
To avoid that, we can instead run them on a background thread and cancel
the thread if that happens.
CI hung in the server test that checks completions because I broke
reading from the System.in. It occurred to me that we probably shouldn't
prompt indefinitely when the user is running tab completions anyway so I
set a timeout of 5 seconds for the user to respond to input. If they
decline to start a server within the timeout, we will just exit. If they
decline to run compilation within the timeout we just skip the
compilation step (so test or main class names will not be provided for
completions).
There were a number of issues with swithcing between raw and canonical
issues that affected both the server and the thin client. These were
reported in #5863 and #5856. In both cases, there were issues with
reading input or having the input be displayed. Debugging those issues
revealed a number of issues with how we were using the jline 3 system
terminal and the hybrid interaction with the jline 2 terminal. This
commit eliminates all of our internal jline 2 usage. The only remaining
jline 2 usage is that we create and override the global terminal for the
scala console for scala versions < 2.13. By moving away from jline 2, I
was also able to fix#5828, which reported that the home, end and delete
keys were not working.
One of the big issues that this commit addresses is that the
NetworkClient was always performing blocking reads on System.in. This
was problematic because it turns out that you can't switch between raw
and canonical modes when there is a read present. To fix this, the
server now sends a message to the client when it wants to read bytes and
only then does the client create a background thread to read a single
byte.
I also figured out how to set the terminal type properly for the thin
client on windows where we had been manually setting the capabilities to
ansi, which only worked for some keys. This fix required switching to
the WindowsInputStream that I introduced in a prior commit. Before we
were using the jline 2 wrapped input stream which was converting some
system events, like home and end, to the wrong escape sequence mappings.
The remainder of the commit is mostly just converting from jline 2 apis
to jline 3 apis.
I verified that tab completions, the scala console, the ammonite console
and a run task that read from System.in all work with both the server
and the thin client on mac, linux and windows after these changes.
Fixes#5828, #5863, #5856
There are situations in windows where it is possible that a client
attempts to connect to a portfile that it did not have access to. This
can happen if, for example, the server is started in administrator mode
but the client is started as a regular user. When this happens, the
client would try to remove the portfile and start a new server. This new
server would not actually be able to start a server though becuase it
would not be able to open the named point because the other server had
it open. As a result, the client would just hang. The fix is to just
abort the thin client if it gets an access denied exception.
When a json rpc request is specified without any headers, the size of
the message may exceed the buffer that was created for reading hdeaders.
This would cause an exception to be thrown when creating a string from
the header buffer because the number of bytes requested would exceed the
capacity of the buffer. To fix this, we can expand the buffer
dynamically if needed. For the common case when the headers are
specified, this should be a no-op.
Ref https://github.com/sbt/sbt/issues/5665
This adds `--server` command that is immediately filtered out in Main.scala.
The purpose of `--server` is so we can invoke thin client from `sbt` script at some point in the future when Bash script can parse `project/build.properties`.
`sbtn` would need to call `sbt` again to start the server, and at that point the shell script would need to actually invoke the server. The intent of `--server` is to be used as the tie breaker.
Also build users may want to sometimes call `sbt --server`.
I introduced the terminalShellPrompt so that we could generate a prompt
that was colored only if the terminal supported color. Rather than
expose the terminal implementation detail, we can just use a boolean
flag that toggles whether or not color is enabled and sbt can pass in
the value of terminal.isColorEnabled into the function.
sbt 1.4.0 generates the shell prompt using the terminal properties for
the specific terminal for which the prompt is rendered. The mechanism
for doing this broke the prompt for projects that overrode the
shellPrompt key, notably the play plugin. After this change, the play
custom prompt is correctly rendered with 1.4.0-SNAPSHOT.
When starting sbt via the thin client with 1.4.0-RC1, there is no output
until sbt finishes booting up which is poor ux. The reason is that sbt
only uses virtual io when sbt.io.virtual == true or formatEnabledInEnv
== true and not ci. The default value for formatEnabledInEnv is set
based on whether color is enabled in the environment. This had copied
old logic that turned on color if ansi was enabled but it makes more
sense to check the color property (which is set by the thin client via
an environment variable when it launches sbt) and fall back to whether
or not java.lang.System.console is defined. We also can explicitly set
"-Dsbt.io.virtual=true" when the thin client launches sbt since the thin
client relies on this behavior. By doing it in both places, the sbtn
for 1.4.0-RC1 will display boot output for newer versions of sbt.
Bonus: don't call ConsoleAppender.formatEnabledInEnv which just calls
back to Terminal.formatEnabledInEnv
The sbt.log.noformat parameter should be treated very similarly to
sbt.io.virtual. When it is true, we should just use the raw io streams
for the process. This came up because of
https://github.com/sbt/sbt/issues/5784 which reported that intellij
imports were not working and that ansi control characters were being
written to the output.
There can be race conditions where we try to interrupt and join a ui
thread before it becomes interruptible by blockign on a queue. To
workaround this, we can add the JoinThread class which adds an
extension method Thread.joinFor that takes a FiniteDuration parameter.
This variant of join will repeatedly interrupt and attempt to join the
thread for up to 10 milliseconds before retrying until the limit is
reached. If the limit is reached, we print a noisy error to the console.
I'm not 100% sure if we are leaking threads in the latest sbt version
but this gives me more piece of mind that either we are always
successfully joining the threads or we will be alerted if the joining
fails.
When exiting the thin client, an interrupted exception stack trace ends
up being printed because NonFatal doesn't include interrupted
exceptions.
Fixes https://github.com/sbt/sbt/issues/5759
Fixes https://github.com/sbt/sbt/issues/5039
Fixes https://github.com/sbt/sbt/issues/4989
This is take 2 on the semicolon fix by emptying out the completion examples in the multi parser.
```
> set scalaV
```
would complete to
```
> set scalaVersion
```
and more importantly
```
coursierUseSbtCredentials := true
```
errors to
```
sbt:hello> coursierUseSbtCredentials := true
[error] Expected ID character
[error] Not a valid command: coursierUseSbtCredentials
[error] Expected project ID
[error] Expected configuration
[error] Expected ':'
[error] Expected key
[error] Not a valid key: coursierUseSbtCredentials (similar: csrExtraCredentials, credentials)
[error] coursierUseSbtCredentials := true
[error] ^
```
With the latest sbt snapshot, the ui would get stuck if the user entered
an empty command. They would be presented with an empty prompt and could
not input any commands. This was caused by the change in
d569abe70a that reset the prompt after a
line was read. I had tried to optimize line reading by ignoring empty
commands in UITask.readline so we wouldn't have to make a new thread.
This optimization wasn't really buying much since it only affects how
quickly the user is reprompted after entering an empty command. Unless a
user is spamming the <enter> key, they shouldn't notice a difference.
It was a bit tricky to reason about the state of the prompt for a
terminal. To help make things more clear, I reworked things so that the
LineReader always sets the prompt to Pending after it reads a command.
In MainLoop, we cache the prompt value and temporarily set it to Running
while the command is running, which is really how it should have always
been.
In order to make the console task work with scala 2.13 and the thin
client, we need to provide a way for the scala repl to use an sbt
provided jline3 terminal instead of the default terminal typically built
by the repl. We also need to put jline 3 higher up in the classloading
hierarchy to ensure that two versions of jline 3 are not loaded (which
makes it impossible to share the sbt terminal with the scala terminal).
One impact of this change is the decoupling of the version of
jline-terminal used by the in process scala console and the version
of jline-terminal specified by the scala version itself. It is possible
to override this by setting the `useScalaReplJLine` flag to true. When
that is set, the scala REPL will run in a fully isolated classloader. That
will ensure that the versions are consistent. It will, however, for sure
break the thin client and may interfere with the embedded shell ui.
As part of this work, I also discovered that jline 3 Terminal.getSize is
very slow. In jline 2, the terminal attributes were automatically cached with a
timeout of, I think, 1 second so it wasn't a big deal to call
Terminal.getAttributes. The getSize method in jline 3 is not cached and
it shells out to run a tty command. This caused a significant
performance regression in sbt because when progress is enabled, we call
Terminal.getSize whenever we log any messages. I added caching of
getSize at the TerminalImpl level to address this. The timeout is 1
second, which seems responsive enough for most use cases. We could also
move the calculation onto a background thread and have it periodically
updated, but that seems like overkill.
There are cases where if the ui state is changing rapidly, that an
AskUserThread can be created and cancelled in a short time windows. This
could cause problems if the AskUserThread is interrupted during
`LineReader.createReader` which I think can shell out to run some
commands so it is relatively slow. If the thread was interrupted during
the call to `LineReader.createReader` and the interruption was not
handled, then the thread would go into `LineReader.readLine`, which
wouldn't exit until the user pressed enter. This ultimately caused the
ui to break until enter because this zombie line reader would be holding
the lock on the terminal input stream.
Ethan Atkins
Adrien Piquerez
eugene yokota