Commit Graph

2853 Commits

Author SHA1 Message Date
Ethan Atkins 4acf3d14c2 Handle closed executor in network channel flush
I noticed some RejectedExecutionExceptions in travis failures of
ClientTest. This could happen if we try to write output to the
network channel after it has been closed. To avoid this problem, we can
catch RejectedExecutionExceptions and do an immediate flush if the
executor has been shutdown.
2020-07-19 10:09:54 -07:00
adpi2 2c0d09dfb3 Specify full java path in BSP connection details
Use System properties to add java path and classpath to BSP connection details
2020-07-13 13:13:54 +02:00
Ethan Atkins 464ea1b97e Add improved classloader construction short circuit
In order for sbt to function well, it needs the test interface, jansi
and forked jline jars provided by a classloader that is parent to all
other sbt classloaders. To do this for just the test interface jar, I
just checked if the top loader in the app configuration had the correct
name. Now that there are three jars, this is more complicated so I
updated the launcher to create a top loader with the method getEarlyJars
implemented and returning the three needed jars. This is a much more
scalable design.

If sbt is entered with a configuration that does not have a top loader
with the getEarlyJars method defined, then we just fall back on
constructing the default layered classlaoder from the configuration
classpath.

The motivation for this change is that I discovered that sbt immediately
crashed when I tried to run a non-snapshot version. After this change, I
verified that both snapshot and non-snapshot versions of the latest sbt
code could load with both an obsolete and up-to-date launcher.
2020-07-12 17:49:01 -07:00
eugene yokota 6110734383
Merge pull request #5673 from eatkins/client-watch-fixes
UI fixes for remote client in watch
2020-07-12 00:55:03 -04:00
Ethan Atkins baa96be6dc Use unprompt in watch to better manage ui state
The unprompt method will actually kill the ui thread if its running. If
we don't to this, we can get into a weird state where after watch is
exited by <enter>, the ask user thread spins up but before it can print
the prompt, the terminal prompt is changed to Running, which has an
empty prompt. The end result is that jline3 never displays the prompt
even though the line reader is active and reading bytes. When the user
typed <enter> a prompt would appear. They also could input a command and
it would run but it wasn't obvious what would happen since the prompt
was missing.
2020-07-10 14:26:16 -07:00
Ethan Atkins 3ddc7bd744 Remove unneeded println
While I'm sure this was added for a reason, it presently is creating
unnecessary and ugly newlines to the console during `~`.
2020-07-10 14:26:14 -07:00
Ethan Atkins 9dc3c6b17f Use terminal printstream in CheckBuildSources
The build source check is evaluated at times when we can't be completely
sure that global logger is pointing at the terminal that initiated the
reload (which may be a passive watch client). To work around this, we
can inspect the exec to determine which terminal initiated the check and
write any output directly to that terminal.
2020-07-10 13:37:54 -07:00
Ethan Atkins 25e83d8fec Add Terminal.withRawOutput api
In the scala console, it's essential that we not process the bytes that
are written to the terminal by jline.
2020-07-10 13:37:54 -07:00
Ethan Atkins 2ecf5967ee Upgrade LineReader to JLine3
This commit upgrades sbt to using jline3. The advantage to jline3 is
that it has a significantly better tab completion engine that is more
similar to what you get from zsh or fish.

The diff is bigger than I'd hoped because there are a number of
behaviors that are different in jline3 vs jline2 in how the library
consumes input streams and implements various features. I also was
unable to remove jline2 because we need it for older versions of the
scala console to work correctly with the thin client. As a result, the
changes are largely additive.

A good amount of this commit was in adding more protocol so that the
remote client can forward its jline3 terminal information to the server.

There were a number of minor changes that I made that either fixed
outstanding ui bugs from #5620 or regressions due to differences between
jline3 and jline2.

The number one thing that caused problems is that the jline3 LineReader
insists on using a NonBlockingInputStream. The implementation ofo
NonBlockingInputStream seems buggy. Moreover, sbt internally uses a
non blocking input stream for system in so jline is adding non blocking
to an already non blocking stream, which is frustrating.

A long term solution might be to consider insourcing LineReader.java
from jline3 and just adapting it to use an sbt terminal rather than
fighting with the jline3 api. This would also have the advantage of not
conflicting with other versions of jline3. Even if we don't, we may want to
shade jline3 if that is possible.
2020-07-10 13:37:53 -07:00
Ethan Atkins 366c49a764 Aggregate watch events
It is possible for sbt to get into a weird state when in a continuous
build when the auto reload feature is on and a source file and a build
file are changed in a small window of time. If sbt detects the source
file first, it will start running the command but then it will
autoreload when it runs the command because of the build file change.
This causes the watch to get into a broken state because it is necessary
to completely restart the watch after sbt exits.

To fix this, we can aggregate the detected events in a 100ms window. The
idea is to handle bursts of file events so we poll in 5ms increments and
as soon as no events are detected, we trigger a build.
2020-07-10 13:04:08 -07:00
Ethan Atkins eb688c9ecd Buffer output to the remote client
Remote clients sometimes flicker when updating progress. This is especially
noticeable when there are two clients and one of them is running a command,
the other will tend to have some visible flickering and character ghosting.
As an experiment, I buffered calls to flush in the NetworkChannel output
stream and the artifacts went away.
2020-07-09 17:17:15 -07:00
Ethan Atkins 6aa1333adb Don't log systemOut messages in jsonRpcNotify
Whatever debugging utility these may have is not worth spamming the task
logs with these.
2020-07-09 16:54:29 -07:00
Ethan Atkins d826e93ddf Only trigger reload if sources have changed
Running a `~` command in a local build off the latest develop branch
will cause the build to reload even if the build sources were only
touched and not actually modified.
2020-07-09 16:54:29 -07:00
Ethan Atkins de1423d662 Clarify boolean flag
I found this difficult to read.
2020-07-09 16:54:29 -07:00
Ethan Atkins d74a10aad1 Set terminal before watch triggered reload
Without setting the terminal, a remote client is unable to reload the
project if it fails.
2020-07-09 16:54:29 -07:00
Ethan Atkins b1dcf031a5 Unprompt channels during project load
In the situation where sbt was started in server mode and a client is
running a `~` command and a project reload is triggered by a change to
a build source, the console terminal looks like

sbt:foo>
[info] received remote command: ~compile
sbt:foo>
[info] welcome to sbt 1.4.0-SNAPSHOT (Azul Systems, Inc. Java 1.8.0_252)
sbt:foo>
[info] loading global plugins from ~/.sbt/1.0/plugins
sbt:foo>
[info] loading settings for project foo-build from metals.sbt ...
sbt:foo>
[info] loading project definition from
~/foo/project
sbt:foo>
[info] loading settings for project root from build.sbt ...
sbt:foo>
[info] loading settings for project macros from build.sbt ...
sbt:foo>
[info] loading settings for project main from build.sbt ...
sbt:foo>
[info] set current project to foo (in build file:~/foo)
sbt:foo>

This change fixes that by unprompting all channels during project
loading and reprompting them when it completes.
2020-07-09 16:54:29 -07:00
Ethan Atkins 80cd0d5e6b Rename SettingsGraph WatchTransitiveDependencies
This is a more descriptive name and differentiates the object from
`SettingGraph`.
2020-07-03 14:08:26 -07:00
Ethan Atkins 4a8e7c0734 Used cached compiled map in LintUnused
Linting unused keys was adding a significant overhead to sbt project
loading because Def.compiled is so slow. It was around 4 seconds in the
sbt project on my computer.
2020-07-03 14:08:26 -07:00
Ethan Atkins 6565618a15 Cache compiled map during build load
The continuous command recompiles the setting graph into a CompiledMap
data structure so that it can determine which files it needs to
transitively monitor during watch. Generating the CompiledMap can be
very slow for large projects (5 seconds or so on my computer in the sbt
project) and this startup cost is paid every time the user enters a
watch with `~`. To avoid this, we can cache the compile map that is
generated during the initial settings evaluation.

The only real drawback I can see is that the compiled map is guaranteed
to remain in memory so long as the BuildStructure instance that holds it
is alive. Given the performance benefit, this seems like a worthwhile
tradeoff.
2020-07-03 14:08:26 -07:00
Ethan Atkins e4c7747570 Add retry to BspCompileTask.compute
There have been occasional failures on appveyor where an
AccessDeniedException was thrown at this point. AccessDeniedExceptions
thrown during scripted tests can often by resolved with a Retry.
2020-07-03 12:15:55 -07:00
Ethan Atkins f9d5fbf29b Support reboot from remote client
Reboot is a bit tricky for the remote client because the sbt server is
actually shut down during reboot. When sbt shuts down the client, it can
notify the client that the reason is a reboot. The client can then
connect to the recently introduced boot control socket to display the
reboot output and supply input in case the build fails to load. Once the
server has brought back up the server, the client can reconnect. When
the client session is interactive, we're done once we reconnect. When
it's a batch session, the client needs to resend the remaing commands
that have submitted that it hasn't yet run.
2020-06-29 16:41:33 -07:00
Ethan Atkins 77b1e38e41 Add shutdown command
Shutdown was being handled as a special case in CommandExchange. This
promotes it to a full fledged command. Also replace instance of
hard-coded strings with constants.
2020-06-29 16:41:33 -07:00
Ethan Atkins 261084bbb2 Avoid leaking sbt processes
On windows, it is sometimes possible to leak an sbt process if two
processes are started simultaneously by a remote client at the same
time. When this happens, the second process is unable to create a
server because of the first process and it also has no io streams
because the the client detaches its streams. We can detect this
in the shell command and prevent the process from persisting as a
zombie.
2020-06-29 16:41:33 -07:00
Ethan Atkins ae2899baae Notify initiating client before shutdown
When a remote client sent the command `shutdown` through the shell, the
client would log an error and exit with a nonzero exit code because
before shutting down, the server would notify the client that it was
disconnecting it due to shutdown. In this scenario, we actually do not
want the client to log an error since they initiated the shutdown, so
before doing the full shutdown, we shutdown the client that inititated
the shutdown with the flag that tells the client not to log the shutdown
or return a nonzero exit code.
2020-06-29 16:41:33 -07:00
Ethan Atkins 267918958d Prevent simultaneous server booting
One issue with the remote client approach is that it is possible for
multiple clients to start multiple servers concurrently. I encountered
this in testing where in one tmux pane I'd start an sbt server and in
another I might run sbtc before the server had finished loading. This
can actually cause java processes to leak because the second process is
unable to start a server but it doesn't necessarily die after the client
that spawned it exits. This commit prevents this scenario by creating a
server socket before it loads the build and closes once the build is
complete. The client can then receive output bytes and forward input to
the booting server.

The socket that is created during boot is always a local socket, either
a UnixDomainServerSocket or a Win32NamedPipeServerSocket. At the moment,
I don't see any reason to support TCP. This socket also has no impact at
all on the normal sbt server that is started up after the project has
loaded.

The socket is hardcoded to be located at the relative path
project/target/$SOCK_NAME or the named pipe $SOCK_NAME where SOCK_NAME
is a farm hash of the absolute path of the project base directory. There
is no portfile json since there is no need since we don't support TCP.

After the socket is created it listens for clients to whom it relays
input to the console's input stream and relays the process output back
to the client. See the javadoc in BootServerSocket.java for further
details.

The process for forking the server is also a bit more complicated after
this change because the client will read the process output and error
streams until the socket is created and thereafter will only read output
from the socket, not the process.
2020-06-29 16:41:33 -07:00
Ethan Atkins db4878c786 Make progress an object
This commit reworks TaskProgress so that it is a singleton object. By
using a singleton, we ensure that there is at most one progress thread
running at a time. With multiple threads, there can be flickering in the
progress reports.

This fixes https://github.com/sbt/sbt/issues/5547. There also was a bug
that the reference to the progress thread was not reset when the thread
itself exited. As a result, it was possible for progress reporting to
stop while tasks were still running. This seemed to primarily happen in
multi-project builds. It should be fixed by this change.
2020-06-29 09:44:24 -07:00
Ethan Atkins d102c47415 fixup! Add multi-client ui to server
The absence of this case was causing match errors. I think this was
introduced during rebasing.
2020-06-29 08:59:52 -07:00
Ethan Atkins 6796f43ab1 Merge remote-tracking branch 'origin/develop' into wip-sbt-instant-startup 2020-06-26 21:51:58 -07:00
eugene yokota 638aa5b8e6
Merge pull request #5653 from eed3si9n/wip/bumpgiter8
Giter8 0.13.1
2020-06-26 19:44:41 -04:00
eugene yokota e7c3b675e9
Merge pull request #5645 from eatkins/watch-excludes
Filter all watch settings for unused key check
2020-06-26 17:20:12 -04:00
eugene yokota 836134d0aa
Merge pull request #5642 from eatkins/zinc-real-paths
Use real paths for zinc roots
2020-06-26 17:19:36 -04:00
Eugene Yokota 9437fcf2c0 Giter8 0.13.1 2020-06-26 17:09:33 -04:00
adpi2 15178b3ef7 trim internal dependency configs in BSP 2020-06-26 10:10:08 +02:00
Ethan Atkins aaee092c96 Revert "Use print + flush instead of println"
This reverts commit b0a859acb5.
2020-06-25 15:16:21 -07:00
Ethan Atkins ba0d97c9ac Use 'FastTrack' instead of 'Maintenance'
FastTrack may better convey the intent.
2020-06-25 15:12:41 -07:00
Ethan Atkins a2047a0b2c Refactor watch
The existing implementation of watch did not work with the thin client.
In sbt 1.3.0, watch was changed to be a blocking command that performed
manual task evaluation. This commit makes the implementation more
similar to < 1.3.0 where watch modifies the state and after running the
user specified command(s), it enters a blocking command. The new
blocking command is very similar to the shell command.

As part of this change, I also reworked some of the internals of watch
so that a number of threads are spawned for reading file and input
events. By using background threads that write to a single event queue,
we are able to block on the file events and terminal input stream rather
than polling. After this change, the cpu utilization as measured by ps
drops from roughly 2% of a cpu to 0.

To integrate with the network client, we introduce a new UITask that is
similar to the AskUserTask but instead of reading lines and adding execs
to the command queue, it reads characters and converts them into watch
commands that we also append to the command queue.

With this new implementation, the watch task that was added in 1.3.0 no
longer works. My guess is that no one was really using it. It wasn't
documented anywhere. The motivation for the task implementation was that
it could be called within another task which would let users define a
task that monitors for file changes before running. Since this had never
been advertised and is only of limited utility anyway, I think it's fine
to break it.

I also had to disable the input-parser and symlinks tests. I'm not 100%
sure why the symlinks test was failing. It would tend to work on my
machine but fail in CI. I gave up on debugging it. The input-parser test
also fails but would be a good candidate to be moved to the client test
in the serverTestProj. At any rate, it was testing a code path that was
only exercised if the user changed the watchInputStream method which is
highly unlikely to have been done in any user builds.

The WatchSpec had become a nuisance and wasn't really preventing from
any regressions so I removed it. The scripted tests are how we test
watch.
2020-06-25 10:05:59 -07:00
Ethan Atkins d5cbc43075 Add tab completion support to thin client
The sbtc client can provide a ux very similar to using the sbt shell
when combined with tab completions. In fact, since some shells have a
better tab completion engine than that provided by jilne2, the
experience can be even better. To make this work, we add another entry
point to the thin client that is capable of generating completions for
an input string. It queries sbt for the completions and prints the
result to stdout, where they are consumed by the shell and fed into its
completion engine.

In addition to providing tab completions, if there is no server running
or if the user is completing `runMain`, `testOnly` or `testQuick`, the
thin client will prompt the user to ask if they would like to start an
sbt server or if they would like to compile to generate the main class
or test names. Neither powershell nor zsh support forwarding input to
the tab completion script. Zsh will print output to stderr so we
opportunistically start the server or complete the test class names.
Powershell does not print completion output at all, so we do not start a
server or fill completions in that case*. For fish and bash, we prompt
the user that they can take these actions so that they can avoid the
expensive operation if desired.

* Powershell users can set the environment variable SBTC_AUTO_COMPLETE
if they want to automatically start a server of compile for run and test
names. No output will be displayed so there can be a long latency
between pressing <tab> and seeing completion results if this variable is
set.
2020-06-24 20:08:14 -07:00
Ethan Atkins ea823f1051 Add server idle timeout
This commit adds the ability for sbt to automatically shut itself down
if it has been idle for some duration of time. The motivation is that
if the user may not realize they have an sbt server running in the
background that is using resources. We don't want to be too aggressive
with the idle timeout because that can reduce the efficacy of the thin
client. A value of one week is chosen so that users can enjoy a long
weekend and when they return to their computer, they won't have to
restart sbt. If they haven't used the server in at least a week, it
seems prudent to just kill it.
2020-06-24 20:04:13 -07:00
Ethan Atkins f8e06def74 Add win32 named pipe security level option
The sbtipcsocket by default restricts win32 named pipes to only allow
connections from the same login session. This makes connecting to a
remote server not work over ssh. We relax the default slightly in sbt to
allow the owner of the pipe to connect over any logon shell. The user
could restore the old behavior with:
```
Global / windowsServerSecurityLevel := Win32SecurityLevel.LOGON_DACL
```
or, if YOLO
```
Global / windowsServerSecurityLevel := Win32SecurityLevel.NO_SECURITY
```
2020-06-24 20:04:13 -07:00
Ethan Atkins a7cb186924 Add --close-io-streams flag to sbt
When we start sbt with the thin client, we want to close the server io
streams after it loads so that the client exiting won't crash the
server. When we are running the server as part of the server tests, it
is nice to have the server output. By setting the --close-io-streams
flag when we launch the server in the client, we are able to achieve
both.
2020-06-24 20:03:44 -07:00
Ethan Atkins 18cb839c47 Wrap network commands for reporting
Running multi commands (input commands delimited by semi-colons) did not
work with the thin client. The commands would actually run on the
server, but the thin client would exit immediately without displaying
the output. The reason was that MainLoop would report the exec complete
when all it had done was split the original command into its constituent
parts and prepended them to the state command list. To work around this,
when we detect a network source command, we can remap its exec id to a
different id and only report the original exec id after the commands
complete. We also have to keep track of whether or not the command
succeeded or failed so that the reporting command reports the correct
result.

The way its implemented is with the the following steps:
1. set the terminal to the network terminal
2. stash the current onFailure so that we can properly report failures
3. add the new exec id to a map of the original exec id to the generated
   id
4. actually run the command
5. if the command succeeds, add the original exec id to a result map
6. pop the onFailure
7. restore the terminal to console
8. report the result -- if the original exec id is in the result map we
   report success. Otherwise we report failure.

There is also logic in NetworkChannel for finding the original exec id
if reporting one of the artificially generated exec ids because the
client will not be aware of that id.
2020-06-24 20:03:43 -07:00
Ethan Atkins 43e4fa85e3 Add ctrl+c support thin client
When the user presses ctrl+c, we want to cancel any running tasks that
were initiated by that client. This is a bit tricky because we may not
be sure what is running if the client is in interactive mode. To work
around this, we send a cancellation request with the special id
__CancelAll. When the NetworkChannel receives this request, it cancels
the active task if was initiated by the client that sent the
cancellation request. The result it returns to the client indicates if
there were any tasks to be cancelled. If there were and the client was
in interactive mode, we do not exit. Otherwise we exit.
2020-06-24 19:40:18 -07:00
Ethan Atkins ab362397ba Add remote cancellation support
This commit makes it possible for a remote client to cancel a running
task initiated by a different client by typing `cancel` into the shell.
It can be useful if the remote client has run something blocking like
console.

The console task can't safely be interrupted, so instead we write some
newlines filed by ctrl+d to exit the console.
2020-06-24 19:40:17 -07:00
Ethan Atkins ba345dd797 Add multi-client ui to server
This commit makes it possible for the sbt server to render the same ui
to multiple clients. The network client ui should look nearly identical
to the console ui except for the log messages about the experimental
client.

The way that it works is that it associates a ui thread with each
terminal. Whenever a command starts or completes, callbacks are invoked
on the various channels to update their ui state. For example, if there
are two clients and one of them runs compile, then the prompt is changed
from AskUser to Running for the terminal that initiated the command
while the other client remains in the AskUser state. Whenever the client
changes uses ui states, the existing thread is terminated if it is
running and a new thread is begun.

The UITask formalizes this process. It is based on the AskUser class
from older versions of sbt. In fact, there is an AskUserTask which is
very similar. It uses jline to read input from the terminal (which could
be a network terminal). When it gets a line, it submits it to the
CommandExchange and exits. Once the next command is run (which may or
may not be the command it submitted), the ui state will be reset.

The debug, info, warn and error commands should work with the multi
client ui. When run, they set the log level globally, not just for the
client that set the level.
2020-06-24 19:40:17 -07:00
Ethan Atkins 734a1e7641 Add virtual terminal support for network clients
This commit adds support for remote clients to connect to the sbt server
and attach themselves as a virtual terminal. In order to make this work,
each connection must send a json rpc request to attach to the server.
When this is received, the server will periodically query the remote
client to get the terminal properties and capabilities that allow the
remote client to act as a jline terminal proxy. There is also support
for json messages with ids sbt/systemIn and sbt/systemOut that allow io
to be relayed from the remote terminal to the sbt server and back.

Certain commands such as `exit` should be evaluated immediately. To make
this work, we add the concept of a MaintenanceTask. The CommandExchange
has a background thread that reads MaintenanceTasks and evaluates them
on demand. This allows maintenance tasks to be evaluated even when sbt
is evaluating an exec. If it weren't done this way, when the user typed
exit while a different remote connection was running a command, they
wouldn't be able to exit until the command completed.

The ServerIntents in ServerHandler did not handle
JsonRpcResponseMessage because prior to this commit, sbt clients were
primarily making requests to the server. But now the server sends
requests to the client for the terminal properties and terminal
capabilities so it was necessary to add an onResponse handler to
ServerIntent.

I had to move the network channel publishBytes method to run on a
background thread because there were scenarios in which the client
socket would get blocked because the server was trying to write on the
same thread that the read the bytes from the client.

To make the console command work, it is necessary to hijack the
classloader for JLine. In MetaBuildLoader, we put a custom forked JLine
that has a setter for the TerminalFactory singleton. This allows us to
change the terminal that is used by JLine in ConsoleReader. Without this
hack, the scala console would not work for remote clients.
2020-06-24 19:38:42 -07:00
Ethan Atkins 1b03c9b1a9 Make Terminal a trait to support multiple clients
In order to support a multi-client sbt server ux, we need to factor
`Terminal` out into a class instead of a singleton. Each terminal provides
and outputstream and inputstream. In all of the places where we were
previously relying on the `Terminal` singleton we need to update the
code to use `Terminal.get`, which will redirect io to the terminal whose
command is currently running.

This commit does not implement the server side ui for network clients.
It is just preparatory work for the multi-client ui.

The Terminal implementations have thread safe access to the output
stream. For this reason, I had to remove the sychronization on the
ConsoleOut lockObject. There were code paths that led to deadlock when
synchronizing on the lockObject.
2020-06-24 19:22:57 -07:00
Ethan Atkins 120e6eb63d Route TaskProgress through CommandExchange
Rather than going through the console appender logging to make
TaskProgress work, we can instead use the CommandExchange. This will be
useful in future commits where there are multiple terminals that all
need to receive progress. By organizing the TaskProgress this way, we
can store a separate progress state for each terminal and update the
progress for all of the active terminals. We also can set the current
running command in command exchange which will be useful in future
commits to show what command is currently running.

This commit also reworks TaskProgress to always kill its thread when
there are no active tasks. It will start a new thread as soon as there
is another active task.
2020-06-24 19:19:06 -07:00
Ethan Atkins fcfe4333fe Consolidate and optimize input stream json reading
We had similar code for reading json frames from an input stream in
NetworkChannel and ServerConnection. I reworked and consolidated this
logic into a shared method in ReadJsonFromInputStream.

This commit also removes the ObjectMessage reporting methods that
weren't doing anything.
2020-06-24 19:19:06 -07:00
Ethan Atkins b0a859acb5 Use print + flush instead of println
With println, it is possible for lines to get interleaved because
another thread may call flush before the println appends a newline.
2020-06-24 19:19:06 -07:00
Ethan Atkins af5afef271 Add option to skip collectAnalysis on network init
The collectAnalysis task an be a bit slow and delays client connections
from running commands. This commit adds an option to skip the analysis
if it isn't needed. The default behavior is left as it was.
2020-06-24 19:19:06 -07:00
Ethan Atkins fe397351df Remove dead external hooks code
These settings were only used by the external hooks implementation that
was removed.
2020-06-24 19:19:06 -07:00
eugene yokota 778264a319
Merge pull request #5618 from adpi2/topic/metals-support
Add suggestion about semanticdb when Metals connect to sbt
2020-06-24 14:19:04 -04:00
Ethan Atkins 3c51f01872 Dealias baseDirectory in AppConfiguration
In Load.scala and Defaults.scala, the AppConfiguration.baseDirectory is
dealiased when it is a symlink. This commit dealiases the
AppConfiguration.baseDirectory if it is a symlink so that sbt
`appConfiguration.value.baseDirectory` should be the same as
`baseDirectory.value`.
2020-06-23 16:48:14 -07:00
Ethan Atkins 33d5bce75a Filter all watch settings for unused key check
Rather than enumerate all of the watch keys that may appear unused
though they can be used by the `~` command, rework lintUnused to take a
function `String => Boolean` instead of `Set[String] => Boolean`.
2020-06-23 16:01:08 -07:00
Ethan Atkins 95221ed837 Remove dead external hooks code
These settings were only used by the external hooks implementation that
was removed.
2020-06-23 15:06:53 -07:00
Ethan Atkins 63cd8f53c6 Use real paths for zinc base path
The AppConfiguration.baseDirectory is dealiased during project loading.
Not dealiasing the symlink here could cause a discrepancy between the
`baseDirectory` key and the value of the base key in the root paths map.
2020-06-23 15:06:53 -07:00
eugene yokota e323f1f713
Merge pull request #5637 from adpi2/feature/bsp-custom-config
Add BSP support for IntegrationTest and other custom configs
2020-06-23 16:11:14 -04:00
Maksim Ochenashko 71ea117a39 Use contraband for generation of RemoteCacheArtifact 2020-06-21 09:41:56 +03:00
Maksim Ochenashko 0c07fa1851 Allow customization of remote cache artifacts 2020-06-20 14:05:41 +03:00
adpi2 56eed20b34 add support for IntegrationTest and custom configs in BSP
In global bspWorkspace setting, retrieve all projects and all configurations that contain the bspTargetIdentifier setting, so that:
- the IntegrationTest configuration, when added to a project, is automatically associated to a BSP target
- a custom configuration that contains the `Defaults.configSettings` is also associated to a BSP target
2020-06-18 16:32:10 +02:00
eugene yokota 9f9bb9daa2
Merge pull request #5634 from eed3si9n/wip/missingok
Fixes missingOk under Coursier
2020-06-17 21:06:40 -04:00
Eugene Yokota 5cce074b83 Fixes missingOk under Coursier
Fixes https://github.com/sbt/sbt/issues/4707
Ref https://github.com/coursier/sbt-coursier/pull/212
2020-06-17 20:06:25 -04:00
Maksim Ochenashko 49e7d71cbc Delete META-INF folder after remote cache pull 2020-06-17 13:01:47 +03:00
eugene yokota 87e57b5390
Merge pull request #5619 from bjaglin/patch-1
exclude effectful initialize setting key from linting
2020-06-16 19:53:07 -04:00
Adrien Piquerez 0738edc9a5
Apply logging style
Co-authored-by: eugene yokota <eed3si9n@gmail.com>
2020-06-16 09:43:22 +02:00
Maksim Ochenashko 3e8073151b Unify calculation of remoteCacheId and remoteCacheIdCandidates 2020-06-16 09:15:42 +03:00
Brice Jaglin 561d0ef602 exclude effectful initialize setting key from linting 2020-06-16 01:08:15 +02:00
adpi2 159171bba5 Add suggestion about semanticdb when Metals connect to sbt
Try parse the required semanticdbVersion in the initialization request metadata
Issue a warning if the semanticdb plugin is not enabled
Issue a warning if the semanticdb version is lower than the required
2020-06-15 16:57:49 +02:00
Eugene Yokota 8789d20724 -Dsbt.semanticdb=true to enable SemanticDB 2020-06-14 17:14:02 -04:00
Eugene Yokota 837dcbb5a6 SemanticDB 4.3.15 2020-06-14 16:53:36 -04:00
Eugene Yokota 5a37ef14fc Zinc 1.4.0-M6 2020-06-14 15:55:37 -04:00
Eugene Yokota 033ff1d8a5 Make JSON parsing errors more consistent 2020-06-11 20:31:13 -04:00
Eugene Yokota cad84afc6d Drop old application/sbt-x1 protocol 2020-06-11 16:22:07 -04:00
eugene yokota 975e3db43e
Merge pull request #5558 from eed3si9n/wip/selective
Selective functor
2020-06-10 17:39:13 -04:00
eugene yokota a83be809ab
Merge pull request #5552 from eed3si9n/wip/promise
implement Def.promise
2020-06-10 17:38:23 -04:00
Eugene Yokota c8f52e6281 Add atMost parameter 2020-06-10 15:30:39 -04:00
Eugene Yokota b78e4f0919 implement Def.promise
This adds `Def.promise` a facility that wraps `scala.concurrent.Promise`. Project layer, there's an implicit for task-that-returns-promise (`Def.Initialize[Task[PromiseWrap[A]]]`) that would inject `await` method, which returns a task. This is a special task that is tagged with `Tags.Sentinel` so that it will bypass the concurrent restrictions. Since there's no CPU- or IO-bound work, this should be ok.

The purpose of this promise for long-running task to communicate with another task midway.
2020-06-10 15:16:25 -04:00
Eugene Yokota 6dd39e7ab8 try last 5 commits to look for remote cache
In case there are a few local commits ahead of the remote cache, this would still grab the close point in the history to resume the build.
2020-06-10 12:55:30 -04:00
Eugene Yokota 585f8399ba implement RemoteCache
This adds `pushRemoteCache`, `pushRemoteCacheTo`, `pullRemoteCache`, etc to implement cached compilation facility.

In addition, the analysis file location is now made more clear.
2020-06-09 14:28:40 -04:00
eugene yokota a109f3d76d
Merge pull request #5526 from 3rwww1/fix/npe-coursier-null-cred-realm
Fix coursierint NPE when credential realm is null
2020-06-09 14:24:24 -04:00
Eugene Yokota 0d15fe1162 Remove HTTP support without explicit opt-in
Ref https://github.com/sbt/sbt/issues/4905
2020-06-07 01:50:41 -04:00
Eugene Yokota ada490e61d Fixes appResolvers returning None
Fixes https://github.com/sbt/sbt/issues/5582
Ref https://github.com/sbt/sbt/pull/5576

In #5576 I added `m.allowInsecureProtocol`, which causes reflection error for older launcher.jar, which then falls back to None appResolvers.
2020-05-30 17:52:39 -04:00
Eugene Yokota 9264b128ef Lower-case some messages 2020-05-29 02:55:12 -04:00
Eugene Yokota 2bf1bcc884 Add welcome banner with Java version
Fixes https://github.com/sbt/sbt/issues/5544
2020-05-29 02:35:59 -04:00
eugene yokota 24f367fa07
Merge pull request #5576 from eed3si9n/wip/bumplauncher
update to launcher 1.1.4
2020-05-28 17:00:00 -04:00
Eugene Yokota 4bb4c3a9b6 launcher 1.1.4
Forward `m.allowInsecureProtocol` to `MavenRepository`.
2020-05-28 16:21:46 -04:00
Adrien Piquerez a498a20627
Merge branch 'develop' into topic/build-server-protocol 2020-05-28 08:57:23 +02:00
Adrien Piquerez 0789fd7be6 Use java command in BspConnectionDetails 2020-05-25 13:32:48 +02:00
Adrien Piquerez b184be860f Add headers 2020-05-25 10:43:54 +02:00
Adrien Piquerez 914b592fb2 scalafmt 2020-05-25 09:40:54 +02:00
Adrien Piquerez 4f00abf1ba BSP is part of the JVM plugin 2020-05-22 13:39:59 +02:00
Adrien Piquerez c11ee3269c Add BspCompilationTaskProgress 2020-05-22 11:15:47 +02:00
Dale Wijnand ff97fb6068 Avoid making NPEs out of OOMEs!
(cherry picked from commit ba29f65f17c7b9d9ac5bf79d0934cf23438a16b4)
2020-05-19 21:30:53 +01:00
Adrien Piquerez a31747758c Create BSP connection file at server startup 2020-05-18 09:35:14 +02:00
Eugene Yokota 45e8426d27 def taskIf[A](a: A): Initialize[Task[A]]
Make `Def.taskIf` accept pure `A` type, as opposed to `Def.task { ... }`.
2020-05-17 23:44:35 -04:00
Eugene Yokota c6f62293f1 Def.taskIf macro
Def.taskIf accepts an if-expression or a block ending in an if-expression.
2020-05-17 23:36:04 -04:00
Eugene Yokota fc791cc23e Rewrite tasks using Def.ifS 2020-05-17 23:36:04 -04:00
Adrien Piquerez c80fe525c6 add BspClient 2020-05-16 09:52:21 +02:00
Adrien Piquerez 2c8a322bd8 bspWorkspace is a setting 2020-05-16 09:52:21 +02:00
Adrien Piquerez f3bce7e976 fix BuildServerReporter 2020-05-16 09:52:21 +02:00
Adrien Piquerez 24f6a6f290 add BSP buildTarget/dependencySources 2020-05-16 09:52:21 +02:00
Adrien Piquerez baa47c88c3 fix BSP logging and response 2020-05-16 09:52:21 +02:00
Adrien Piquerez 98750817c6 fix internal transitive mgmt in BSP 2020-05-16 09:52:21 +02:00
Adrien Piquerez 068fe2ad0a add BuildServerCapabilities 2020-05-16 09:52:21 +02:00
Adrien Piquerez 3ae42ae9c6 fix scala jars and add capabilities in BuildTarget 2020-05-16 09:52:21 +02:00
Adrien Piquerez 5172775b80 fix classpath in buildTarget/scalacOptions 2020-05-16 09:52:21 +02:00
Adrien Piquerez 454ee61289 separate BSP and LSP handlers + add bspWorkspace task 2020-05-16 09:52:21 +02:00
Adrien Piquerez a415cd0cfc replace LSP compiler reporter by BSP one 2020-05-16 09:52:20 +02:00
Adrien Piquerez f89cef1fd0 add bspCompile task 2020-05-16 09:52:20 +02:00
Eugene Yokota f6ff4da1c5 Disable aggregation for global input tasks 2020-05-16 09:52:20 +02:00
Eugene Yokota d2d0a9ca80 buildTarget/scalacOptions 2020-05-16 09:52:20 +02:00
Eugene Yokota 10b2154d2e Refactor bspBuildTargetSources to be an input task + ScopeFilter 2020-05-16 09:52:20 +02:00
Eugene Yokota 1ccff0ca6d Add Scala Build Target 2020-05-16 09:52:20 +02:00
Eugene Yokota 95f3877bae Add stubs for build/shutdown and build/exit 2020-05-16 09:52:20 +02:00
Eugene Yokota cb93d20492 build server protocol
Initial draft for bsp support.

This shows two communication pattern around BSP.
First, if the request can be handled with the build knowledge is readily available in `NetworkChannel` we can reply immediately. `BuildServerImpl#onBspBuildTargets` is an example for that.

Second, if the request requires `State`, then we can forward the parameter into a custom command, and reply back from a command. `BuildServerProtocol.bspBuildTargetSources` is an example of that since it needs to invoke tasks to generate sources.
2020-05-16 09:52:20 +02:00
Alexandre Archambault f850a9c966 Update coursier to 2.0.0-RC6-4
And warn at start-up if ~/.coursier/cache is found.
2020-05-14 15:54:43 +02:00
Adrien Piquerez ae1df53033 fix relay appender 2020-05-13 14:11:45 +02:00
eugene yokota 4592493617
Merge pull request #5549 from adpi2/issue/json-response
Prevent more than one response per json RPC request
2020-05-12 22:07:11 -04:00
Adrien Piquerez c221f57812 code review 2020-05-12 19:20:43 +02:00
Adrien Piquerez 8df754eeb1 rename publish to either respond or notify 2020-05-12 16:26:33 +02:00
Adrien Piquerez 255a0a6ea6 send response to the source channel only 2020-05-12 14:44:10 +02:00
Adrien Piquerez df293fbfd5 prevent multiple response to a single request 2020-05-12 10:38:47 +02:00
Adrien Piquerez 781584d137 id is mandatory in json rpc responses 2020-05-11 16:51:34 +02:00
Brice Jaglin 331ca6ec9b enable ServiceLoader discovery across classloader layers
java.util.ServiceLoader uses findResources(), which was not
overriden in ReverseLookupClassLoader, causing resources available
in the descendant classloader not to be discovered when a service
loader instance was using the top classloader.
2020-05-08 12:41:44 +02:00
eugene yokota f5eae27c69
Merge pull request #5531 from eatkins/optimize-task-progress
Optimize task progress performance
2020-05-06 23:07:39 -04:00
eugene yokota d2fa0f100e
Merge pull request #5533 from eatkins/ivy-xml
Make IvyXml.writeFiles private[sbt]
2020-05-06 11:46:29 -04:00
Ethan Atkins e3e53df797 Make IvyXml.writeFiles private[sbt]
I got confused an changed the access for the wrong method to support
https://github.com/sbt/sbt/pull/5364.

Partially reverts 5c55393f1b.
2020-05-06 07:43:22 -07:00
Ethan Atkins 9b2bbdd4cc Report progress on background thread
Having the progress reports directly generated in beforeWork delays the
task from being submitted to the executor. This commit moves all of the
reporting onto the background thread to avoid these delays since
progress is less important than task evaluation.
2020-05-05 21:28:08 -07:00
Ethan Atkins f4c11a63ab Re-implement meta build source checking
The old implementation of checkBuildSources can easily take 20ms to run
when called in MainLoop.processCommand. It is rarely faster than 4-5ms.
To reduce this overhead, I stopped using the checkBuildSources task in
processCommand. Instead, I manually cache the build source hashes in a
global state variable and add a file monitor that invalidate the entire
set of source hashes if any changes are detected. This could probably be
more efficient, but I figure that build sources change infrequently
enough that it's fine to just invalidate the entire list of source
hashes.

Because the CheckBuildSources instance is already watching the meta
build, I reworked Continuous to use that FileTreeRepository for the
build sources if it is available.

Bonus: fixes https://github.com/sbt/sbt/issues/5482
2020-05-05 21:26:04 -07:00
Ethan Atkins b829ac9796 Remove FileManagement
This was unused.
2020-05-05 21:25:56 -07:00
Ethan Atkins 0a06bfe2d5 Optimize TaskProgress.containsSkipTasks
I was surprised to find this method in a flamegraph* so I optimized it.
TaskProgress was actually on the hotpath of task evaluation so every
single task was slower to be enqueued with the CompletionService. After
this change, containsSkipTasks dropped out of the flamegraph.

*The flamegraph was for a compile loop where sbt constantly modified a
single source file and re-compiled it. The containsSkipTasks method
appeared in over 2% of the method calls.
2020-05-05 21:04:09 -07:00
frosforever d1cf37b2ca Update Tests.Group to be bin compat by manually implementing case class methods 2020-05-04 09:29:31 -04:00
frosforever 18d51eac58 Add tags to inProccess group as well 2020-05-04 09:29:31 -04:00
frosforever b2942f6321 Add user defined tags to Tests.Group 2020-05-04 09:29:31 -04:00
Erwan Queffelec 23fdb324cb Fix coursierint NPE when credential realm is null
As stated in #5492 and #2366 some artifact hosting services (at least
Azure Artifacts, seems to affect GCP as well) do not offer a stable HTTP
Auth realm that can be safely set in SBT credentials, hence the need to
support null or empty Credentials.

The Ivy (publishing) side of the issue was fixed in sbt/ivy#36

As to the resolving side, This commit is only part of the solution as it
just prevents an NPE and does not consider if coursier itself will fall
back to finding credentials with a null realm that matches the server
hostname.

Related-to: #5492
Related-to: #2366
Signed-off-by: Erwan Queffelec <erwan.queffelec@gmail.com>
2020-05-03 15:53:19 +02:00
Ethan Atkins a1ea5b72f3 Support capital letter options in doLoadFailed
Also display invalid response character.
2020-05-01 13:12:37 -07:00
Ethan Atkins 73235d12e0 Switch to lower case for some messages
There is push to move away from complete sentences in warnings.
2020-05-01 13:12:13 -07:00
Ethan Atkins 24d4b3549b Set default supershell blank zone to 1
We do not need a large blank zone with the virtual system.out.
2020-05-01 12:35:43 -07:00
Ethan Atkins 5c42c20cdc Simplify setting dumbTerm variable 2020-05-01 12:35:43 -07:00
Ethan Atkins 58822cc3f5 Add virtual System.out for supershell
In order to make supershell work with println, this commit introduces a
virtual System.out to sbt. While sbt is running, we override the default
java.lang.System.out, java.lang.System.in, scala.Console.out and
scala.Console.in (unless the property `sbt.io.virtual` is set to
something other than true). When using virtual io, we buffer all of the
bytes that are written to System.out and Console.out until flush is
called. When flushing the output, we check if there are any progress
lines. If so, we interleave them with the new lines to print.

The flushing happens on a background thread so it should hopefully not
impede task progress.

This commit also adds logic for handling progress when the cursor is not
all the way to the left. We now track all of the bytes that have been
written since the last new line. Supershell will then calculate the
cursor position from those bytes* and move the cursor back to the
correct position. The motivation for this was to make the run command
work with supershell even when multiple main classes were specified.

* This might not be completely reliable if the string contains ansi
cursor movement characters.
2020-05-01 12:35:43 -07:00
Ethan Atkins a0012bab75 Apply minimum task progress length threshold
To reduce the noise of supershell, this commit imposes a 10 millisecond
minimum duration before a task will appear in the supershell lines.
2020-05-01 12:35:43 -07:00
Ethan Atkins 9218d3c087 Redraw command prompt after network command
Presently if a server command comes in while in the shell, the client
output can appear on the same line as the command prompt and the command
prompt will not appear again until the user hits enter. This is a
confusing ux. For example, if I start an sbt server and type
the partial command "comp" and then start up a client and run the clean
command followed by a compile, the output looks like:

[info] sbt server started at local:///Users/ethanatkins/.sbt/1.0/server/51cfad3281b3a8a1820a/sock
sbt:scala-compile> comp[info] new client connected: network-1
[success] Total time: 0 s, completed Dec 12, 2019, 7:23:24 PM
[success] Total time: 0 s, completed Dec 12, 2019, 7:23:27 PM
[success] Total time: 2 s, completed Dec 12, 2019, 7:23:31 PM

Now, if I type "ile\n", I get:
[info] sbt server started at local:///Users/ethanatkins/.sbt/1.0/server/51cfad3281b3a8a1820a/sock
ile
[success] Total time: 0 s, completed Dec 12, 2019, 7:23:34 PM
sbt:scala-compile>

Following the same set of inputs after this change, I get:
[info] sbt server started at local:///Users/ethanatkins/.sbt/1.0/server/51cfad3281b3a8a1820a/sock
sbt:scala-compile> comp
[info] new client connected: network-1
[success] Total time: 0 s, completed Dec 12, 2019, 7:25:58 PM
sbt:scala-compile> comp
[success] Total time: 0 s, completed Dec 12, 2019, 7:26:14 PM
sbt:scala-compile> comp
[success] Total time: 1 s, completed Dec 12, 2019, 7:26:17 PM
sbt:scala-compile> compile
[success] Total time: 0 s, completed Dec 12, 2019, 7:26:19 PM
sbt:scala-compile>

To implement this change, I added the redraw() method to LineReader
which is a wrapper around ConsoleReader.drawLine; ConsoleReader.flush().
We invoke LineReader.redraw whenever the ConsoleChannel receives a
ConsolePromptEvent and there is a running thread.

To prevent log lines from being appended to the prompt line, in the
CommandExchange we print a newline character whenever a new command is
received from the network or a network client connects and we believe
that there is an active prompt.
2020-05-01 12:35:43 -07:00
Ethan Atkins 08091d64c1 Log server commands
Prior to this change, if a network command came in, it would run in the
background with no real feedback in the server ui. Prior to this change,
running compile from the thin client would look like:

sbt:scala-compile>
[success] Total time: 1 s, completed Dec 12, 2019, 7:24:43 PM
sbt:scala-compile>

Now it looks like:
sbt:scala-compile>
[info] Running remote command: compile
[success] Total time: 1 s, completed Dec 12, 2019, 7:26:17 PM
sbt:scala-compile>
2020-05-01 12:35:43 -07:00
Ethan Atkins 5afc0f0fdf Don't require newline for load failed commands
It's a bit annoying to have to hit enter here.

Also, this should fix https://github.com/sbt/sbt/issues/5162 because if
there is no System.in attached, the read will return -1 which will cause
sbt to quit.
2020-05-01 12:35:43 -07:00
Ethan Atkins c7b52203a0 Don't require newlines for main classes
There typically are fewer than 10 main classes in a project so allow the
user to just input a single digit in those cases. Otherwise fallback to
a line reader.
2020-05-01 12:35:43 -07:00
Ethan Atkins 44ef718448 Improve run command warning
When the user inputs `run` or `runMain` we shouldn't print the warning
about multiple classes because in the case of run they already will be
prompted to select the classes and in the case of runMain, they are
already required to specify the class name.

Bonus:
 * improve punctuation
 * add clear screen to selector dialogue
 * print selector dialogue in one call to println -- this should prevent
   the possibility of messages from other threads being interlaced with
   the dialogue
2020-05-01 12:35:43 -07:00
Ethan Atkins 7902ec3b7d Add Terminal abstraction
This commit aims to centralize all of the terminal interactions
throughout sbt. It also seeks to hide the jline implementation details
and only expose the apis that sbt needs for interacting with the
terminal.

In general, we should be able to assume that the terminal is in
canonical (line buffered) mode with echo enabled. To switch to raw mode
or to enable/disable echo, there are apis: Terminal.withRawSystemIn and
Terminal.withEcho that take a thunk as parameter to ensure that the
terminal is reset back to the canonical mode afterwards.
2020-05-01 12:35:43 -07:00
Ethan Atkins cd65543d10 Deprecate unused ConsoleUnpromptEvent 2020-05-01 12:28:44 -07:00
Ethan Atkins 2f4c603be6 Stop continuous input thread if System.in is closed
The watch tests take forever on windows because wrapped.read() always
returns -1 during scripted.
2020-05-01 12:28:44 -07:00
Ethan Atkins 7d07bbabbf Fix DefinesClass implementation for jdk > 8
When trying to use any jdk > 8 with the latest sbt, sbt will die in some
projects because it tries to call Locate.defineClass on rt.jar, which
is represented with a DummyVirtualFile and causes a crash.
2020-04-30 20:43:04 -07:00