luke
/
sbt
mirror of https://github.com/sbt/sbt.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #97 from alexarchambault/topic/update-readme 

Rework readme
This commit is contained in:
Alexandre Archambault 2016-01-03 19:33:23 +01:00
parent 5452d77193 7f741f2ef9
commit 997e3f4a80
15 changed files with 766 additions and 171 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

391
README.md
View File

@ -2,7 +2,7 @@
*Pure Scala Artifact Fetching*
A pure Scala substitute for [Aether](http://www.eclipse.org/aether/)
A Scala library to fetch dependencies from Maven / Ivy repositories
[![Build Status](https://travis-ci.org/alexarchambault/coursier.svg?branch=master)](https://travis-ci.org/alexarchambault/coursier)
[![Build status (Windows)](https://ci.appveyor.com/api/projects/status/trtum5b7washfbj9?svg=true)](https://ci.appveyor.com/project/alexarchambault/coursier)
@ -12,11 +12,11 @@ A pure Scala substitute for [Aether](http://www.eclipse.org/aether/)
*coursier* is a dependency resolver / fetcher *à la* Maven / Ivy, entirely
rewritten from scratch in Scala. It aims at being fast and easy to embed
in other contexts. Its very core (`core` module) aims at being
extremely pure, and should be approached thinking algebraically.
extremely pure, and only requires to be fed external data (Ivy / Maven metadata) via a monad.
The `cache` module handles caching of the metadata and artifacts themselves,
and is less so pure than the `core` module, in the sense that it happily
does IO as a side-effect (although it naturally favors immutability for all
does IO as a side-effect (always wrapped in `Task`, and naturally favoring immutability for all
that's kept in memory).
It handles fancy Maven features like
@ -26,42 +26,221 @@ It handles fancy Maven features like
* [properties](http://books.sonatype.com/mvnref-book/reference/resource-filtering-sect-properties.html),
* etc.
It happily resolves dependencies involving modules from the Hadoop ecosystem (Spark, Flink, etc.), that
make a heavy use of these.
and is able to fetch metadata and artifacts from both Maven and Ivy repositories.
It can be used either from the command-line, via its API, or from the browser.
Compared to the default dependency resolution of SBT, it adds:
* downloading of artifacts in parallel,
* better offline mode - one can safely work with snapshot dependencies if these are in cache (SBT tends to try to fail if it cannot check for updates),
* non obfuscated cache (cache structure just mimicks the URL it caches).
It downloads the metadata or the artifacts in parallel (usually, 6 parallel
downloads).
From the command-line, it also has:
* a [launcher](#launch), able to launch apps distributed via Maven / Ivy repositories,
* a [bootstrap](#bootstrap) generator, able to generate stripped launchers of these apps.
## Command-line
Lastly, it can be used programmatically via its [API](#api) and has a Scala JS [demo](#scala-js-demo).
## Table of content
1. [Quick start](#quick-start)
2. [Why](#why)
3. [Usage](#usage)
1. [SBT plugin](#sbt-plugin)
2. [Command-line](#command-line)
3. [API](#api)
4. [Scala JS demo](#scala-js-demo)
4. [Contributors](#contributors)
5. [Projects using coursier](#projects-using-coursier)
## Quick start
The default global cache used by coursier is `~/.coursier/cache/v1`. E.g. the artifact at
`https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.7/scala-library-2.11.7.jar`
will land in `~/.coursier/cache/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.7/scala-library-2.11.7.jar`.
From the SBT plugin, the default repositories are the ones provided by SBT (typically Central or JFrog, and `~/.ivy2/local`).
From the CLI tools, these are Central (`https://repo1.maven.org/maven2`) and `~/.ivy2/local`.
From the API, these are specified manually - you are encouraged to use those too.
* SBT plugin
Enable the SBT plugin by adding
```scala
addSbtPlugin("com.github.alexarchambault" % "coursier-sbt-plugin" % "1.0.0-M2")
```
to `~/.sbt/0.13/plugins/build.sbt` (enables it globally), or to the `project/plugins.sbt` file
of a SBT project. Tested with SBT 0.13.8 / 0.13.9.
* CLI
Download and run its laucher with
```
$ curl -L -o coursier https://git.io/vEpQR && chmod +x coursier && ./coursier --help
```
Note that the launcher itself weights only 8 kB and can be easily
embedded as is in other projects.
The first time it is run, it will download the artifacts required to launch
coursier. You'll be fine the next times :-).
Run an application distributed via artifacts with
```
$ ./coursier launch com.lihaoyi:ammonite-repl_2.11.7:0.5.2
```
The cache of this default launcher defaults to a directory named `.coursier`,
in the same directory as the launcher. This can be changed by manually adjusting
the `COURSIER_CACHE` variable in the first lines of the launcher.
Download and list the classpath of one or several dependencies with
```
$ ./coursier fetch org.apache.spark:spark-sql_2.11:1.5.2 com.twitter:algebird-spark_2.11:0.11.0
Dependencies:
org.apache.spark:spark-sql_2.11:1.5.2
com.twitter:algebird-spark_2.11:0.11.0
Fetching artifacts
/path/to/.coursier/cache/v1/https/repo1.maven.org/maven2/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar
/path/to/.coursier/cache/v1/https/repo1.maven.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar
/path/to/.coursier/cache/v1/https/repo1.maven.org/maven2/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar
/path/to/.coursier/cache/v1/https/repo1.maven.org/maven2/com/typesafe/config/1.2.1/config-1.2.1.jar
...
```
* API
Add to your `build.sbt`
```scala
libraryDependencies ++= Seq(
"com.github.alexarchambault" %% "coursier" % "1.0.0-M2",
"com.github.alexarchambault" %% "coursier-cache" % "1.0.0-M2"
)
```
Add an import for coursier,
```scala
import coursier._
```
To resolve dependencies, first create a `Resolution` case class with your dependencies in it,
```scala
val start = Resolution(
Set(
Dependency(
Module("org.scalaz", "scalaz-core_2.11"), "7.2.0"
),
Dependency(
Module("org.spire-math", "cats-core_2.11"), "0.3.0"
)
)
)
```
Create a fetch function able to get things from a few repositories via a local cache,
```scala
val repositories = Seq(
Cache.ivy2Local,
MavenRepository("https://repo1.maven.org/maven2")
)
val fetch = Fetch.from(repositories, Cache.fetch())
```
Then run the resolution per-se,
```scala
val resolution = start.process.run(fetch).run
```
That will fetch and use metadata.
Check for errors in
```scala
val errors: Seq[(Dependency, Seq[String])] = resolution.errors
```
These would mean that the resolution wasn't able to get metadata about some dependencies.
Then fetch and get local copies of the artifacts themselves (the JARs) with
```scala
import java.io.File
import scalaz.\/
import scalaz.concurrent.Task
val localArtifacts: Seq[FileError \/ File] = Task.gatherUnordered(
resolution.artifacts.map(Cache.file(_).run)
).run
```
## Why
The current state of dependency management in Scala suffers several flaws, that prevent applications to fully
profit from and rely on dependency management. Coursier aims at addressing these by making it easy to:
- resolve / download dependencies programmatically,
- launch applications distributed via Maven / Ivy artifacts from the command-line,
- work offline with artifacts,
- sandbox dependency management between projects.
As its [API](#api) illustrates, getting artifacts of dependencies is just a matter of specifying these along
with a few repositories. You can then straightforwardly get the corresponding artifacts, easily getting
precise feedback about what goes on during the resolution.
Launching an application distributed via Maven artifacts is just a command away with the [launcher](#command-line) of coursier.
In most cases, just specifying the corresponding main dependency is enough to launch the corresponding application.
If all your dependencies are in cache, chances are coursier will not even try to connect to remote repositories. This
also applies to snapshot dependencies of course - these are only updated on demand, not getting constantly in your way
like is currently the case by default with SBT.
When using coursier from the command-line or via its SBT plugin, sandboxing is just one command away. Just do
`export COURSIER_CACHE="$(pwd)/.coursier-cache"`, and the cache will become `.coursier-cache` from the current
directory instead of the default global `~/.coursier/cache/v1`. This allows for example to quickly inspect the content
of the cache used by a particular project, in case you have any doubt about what's in it.
## Usage
### SBT plugin
Enable the SBT plugin globally by adding
```scala
addSbtPlugin("com.github.alexarchambault" % "coursier-sbt-plugin" % "1.0.0-M2")
```
to `~/.sbt/0.13/plugins/build.sbt`
To enable it on a per-project basis, add it only to the `project/plugins.sbt` of a SBT project.
The SBT plugin has been tested only with SBT 0.13.8 / 0.13.9.
Once enabled, the `update`, `updateClassifiers`, and `updateSbtClassifiers` commands are taken care of by coursier. These
provide more output about what's going on than their default implementations do.
### Command-line
Download and run its laucher with
```
$ curl -L -o coursier https://git.io/vEpQR && chmod +x coursier && ./coursier --help
```
The launcher itself weights only 8 kB and can be easily embedded as is in other projects.
It downloads the artifacts required to launch coursier on the first run.
```
$ ./coursier --help
```
lists the available coursier commands. The most notable ones are `launch`,
and `fetch`. Type
lists the available coursier commands. The most notable ones are `launch`, and `fetch`. Type
```
$ ./coursier command --help
```
to get a description of the various options the command `command` (replace with one
of the above command) accepts.
### launch
Both command belows can be given repositories with the `-r` or `--repository` option, like
```
-r central
-r https://oss.sonatype.org/content/repositories/snapshots
-r "ivy:https://repo.typesafe.com/typesafe/ivy-releases/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]"
```
`central` and `ivy2local` correspond to Maven Central and `~/.ivy2/local`. These are used by default
if no `-r` or `--repository` option is specified.
As soon as a `-r` or `--repository` option is specified, these default are not used any more - only the
specified repositories are used.
Repositories starting with `ivy:` are assumed to be Ivy repositories, specified with an Ivy pattern. Else,
a Maven repository is assumed.
#### launch
The `launch` command fetches a set of Maven coordinates it is given, along
with their transitive dependencies, then launches the "main `main` class" from
@ -115,10 +294,10 @@ $ ./coursier launch net.sf.proguard:proguard-base:5.2.1 -M proguard.ProGuard
$ ./coursier launch net.sf.proguard:proguard-retrace:5.2.1 -M proguard.retrace.ReTrace
```
### fetch
#### fetch
The `fetch` command simply fetches a set of dependencies, along with their
transitive dependencies, then prints the local paths of all their artefacts.
transitive dependencies, then prints the local paths of all their artifacts.
Example
```
@ -140,23 +319,183 @@ Welcome to the Ammonite Repl 0.5.2
@
```
## API
More explanations to come :-)
## Scala JS demo
### bootstrap
The `bootstrap` generates tiny bootstrap launchers, able to pull their dependencies from
repositories on first launch. For example, the launcher of coursier is [generated](https://github.com/alexarchambault/coursier/blob/master/project/generate-launcher.sh) with a command like
```
$ ./coursier bootstrap \
com.github.alexarchambault:coursier-cli_2.11:1.0.0-M2 \
-b -f -o coursier \
-M coursier.cli.Coursier
```
See `./coursier bootstrap --help` for a list of the available options.
### API
Add to your `build.sbt`
```scala
libraryDependencies ++= Seq(
"com.github.alexarchambault" %% "coursier" % "1.0.0-M2",
"com.github.alexarchambault" %% "coursier-cache" % "1.0.0-M2"
)
```
The first module, `"com.github.alexarchambault" %% "coursier" % "1.0.0-M2"`, mainly depends on
`scalaz-core` (and only it, *not* `scalaz-concurrent` for example). It contains among others,
definitions,
mainly in [`Definitions.scala`](https://github.com/alexarchambault/coursier/blob/master/core/shared/src/main/scala/coursier/core/Definitions.scala),
[`Resolution`](https://github.com/alexarchambault/coursier/blob/master/core/shared/src/main/scala/coursier/core/Resolution.scala), representing a particular state of the resolution,
and [`ResolutionProcess`](https://github.com/alexarchambault/coursier/blob/master/core/shared/src/main/scala/coursier/core/ResolutionProcess.scala),
that expects to be given metadata, wrapped in any `Monad`, then feeds these to `Resolution`, and at the end gives
you the final `Resolution`, wrapped in the same `Monad` it was given input. This final `Resolution` has all the dependencies,
including the transitive ones.
The second module, `"com.github.alexarchambault" %% "coursier-cache" % "1.0.0-M2"`, is precisely in charge of fetching
these input metadata. It uses `scalaz.concurrent.Task` as a `Monad` to wrap them. It also fetches artifacts (JARs, etc.).
It caches all of these (metadata and artifacts) on disk, and validates checksums too.
In the code below, we'll assume some imports are around,
```scala
import coursier._
```
Resolving dependencies involves create an initial resolution state, with all the initial dependencies in it, like
```scala
val start = Resolution(
Set(
Dependency(
Module("org.spire-math", "cats-core_2.11"), "0.3.0"
),
Dependency(
Module("org.scalaz", "scalaz-core_2.11"), "7.2.0"
)
)
)
```
It goes without saying that a `Resolution` is immutable, as are all the classes defined in the core module.
The resolution process will go on by giving successive `Resolution`s, until the final one.
`start` above is only the initial state - it is far from over, as the `isDone` method on it tells,
```scala
scala> start.isDone
res4: Boolean = false
```
In order for the resolution to go on, we'll need things from a few repositories,
```scala
scala> val repositories = Seq(
| Cache.ivy2Local,
| MavenRepository("https://repo1.maven.org/maven2")
| )
repositories: Seq[coursier.core.Repository] = List(IvyRepository(file:/Users/alexandre/.ivy2/local/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext],None,Map(),true,true,true), MavenRepository(https://repo1.maven.org/maven2,None,false))
```
The first one, `Cache.ivy2Local`, is defined in `coursier.Cache`, itself from the `coursier-cache` module that
we added above. As we can see, it is an `IvyRepository`, picking things under `~/.ivy2/local`. An `IvyRepository`
is related to the [Ivy](http://ant.apache.org/ivy/) build tool. This kind of repository involves a so-called [pattern](http://ant.apache.org/ivy/history/2.4.0/concept.html#patterns), with
various properties. These are not of very common use in Scala, although SBT uses them a bit.
The second repository in a `MavenRepository`. These are simpler than the Ivy repositories. They're the ones
we're the most used to in Scala. Common ones like [Central](https://repo1.maven.org/maven2) like here, or the repositories
from [Sonatype](https://oss.sonatype.org/content/repositories/), are Maven repositories. These originate
from the [Maven](https://maven.apache.org/) build tool. Unlike the Ivy repositories which involve customisable patterns to point
to the underlying metadata and artifacts, the paths of these for Maven repositories all look alike,
like for any particular version of the standard library, under paths like
[this one](http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.7/).
Both `IvyRepository` and `MavenRepository` are case classes, so that it's straightforward to specify one's own
repositories.
Now that we have repositories, we're going to mix these with things from the `coursier-cache` module,
for resolution to happen via the cache. We'll create a function
of type `Seq[(Module, String)] => F[Seq[((Module, String), Seq[String] \/ (Artifact.Source, Project))]]`.
Given a sequence of dependencies, designated by their `Module` (organisation and name in most cases)
and version (just a `String`), it gives either errors (`Seq[String]`) or metadata (`(Artifact.Source, Project)`),
wrapping the whole in a monad `F`.
```scala
val fetch = Fetch.from(repositories, Cache.fetch())
```
The monad used by `Fetch.from` is `scalaz.concurrent.Task`, but the resolution process is not tied to a particular
monad - any stack-safe monad would do.
With this `fetch` method, we can now go on with the resolution. Calling `process` on `start` above gives a
[`ResolutionProcess`](https://github.com/alexarchambault/coursier/blob/master/core/shared/src/main/scala/coursier/core/ResolutionProcess.scala),
that drives the resolution. It is loosely inspired by the `Process` of scalaz-stream.
It is an immutable structure, that represents the various states the resolution process can be in.
Its method `current` gives the current `Resolution`. Calling `isDone` on the latter says whether the
resolution is done or not.
The `next` method, that expects a `fetch` method like the one above, gives
the "next" state of the resolution process, wrapped in the monad of the `fetch` method. It allows to do
one resolution step.
Lastly, the `run` method runs the whole resolution until its end. It expects a `fetch` method too,
and will make at most `maxIterations` steps (50 by default), and return the "final" resolution state,
wrapped in the monad of `fetch`. One should check that the `Resolution` it returns is done (`isDone`) -
the contrary means that `maxIterations` were reached, likely signaling an issue, unless the underlying
resolution is particularly complex, in which case `maxIterations` could be increased.
Let's run the whole resolution,
```scala
val resolution = start.process.run(fetch).run
```
To get additional feedback during the resolution, we can give the `Cache.default` method above
a [`Cache.Logger`](https://github.com/alexarchambault/coursier/blob/cf269c6895e19f2d590f08811406724304332950/cache/src/main/scala/coursier/Cache.scala#L484-L490).
By default, downloads happen in a global fixed thread pool (with 6 threads, allowing for 6 parallel downloads), but
you can supply your own thread pool to `Cache.default`.
Now that the resolution is done, we can check for errors in
```scala
val errors: Seq[(Dependency, Seq[String])] = resolution.errors
```
These would mean that the resolution wasn't able to get metadata about some dependencies.
We can also check for version conflicts, in
```scala
val conflicts: Set[Dependency] = resolution.conflicts
```
which are dependencies whose versions could not be unified.
Then, if all went well, we can fetch and get local copies of the artifacts themselves (the JARs) with
```scala
import java.io.File
import scalaz.\/
import scalaz.concurrent.Task
val localArtifacts: Seq[FileError \/ File] = Task.gatherUnordered(
resolution.artifacts.map(Cache.file(_).run)
).run
```
We're using the `Cache.file` method, that can also be given a `Logger` (for more feedback) and a custom thread pool.
### Scala JS demo
*coursier* is also compiled to Scala JS, and can be tested in the browser via its
[demo](http://alexarchambault.github.io/coursier/#demo).
# Contributors
## Contributors
- Your name here :-)
Don't hesitate to pick an issue to contribute, and / or ask for help for how to proceed
on the [Gitter channel](https://gitter.im/alexarchambault/coursier).
# Projects using coursier
## Projects using coursier
- [Lars Hupel](https://github.com/larsrh/)'s [libisabelle](https://github.com/larsrh/libisabelle) fetches
some of its requirements via coursier,

69
USAGE.md
View File

@ -1,69 +0,0 @@
# API
Ensure you have a dependency on its artifact, e.g. add in `build.sbt`,
```scala
resolvers += Resolver.sonatypeRepo("snapshots")
libraryDependencies +=
"com.github.alexarchambault" %% "coursier" % "0.1.0-SNAPSHOT"
```
Then,
```scala
import coursier._
// Cache for metadata can be setup here, see cli/src/main/scala/coursier/Coursier.scala
val repositories = Seq(
Repository.ivy2Local,
Repository.mavenCentral
)
val dependencies = Set(
Dependency(Module("com.lihaoyi", "ammonite-pprint_2.11"), "0.3.2"),
Dependency(Module("org.scala-lang", "scala-reflect"), "2.11.6")
)
val resolution =
Resolution(dependencies)
.process // Process ADT, allows to go from a Resolution to the next
.run(repositories) // Running the process, fetching metadata from these repositories
.run // Run the Task[Resolution], and get the resulting Resolution
// Note that only metadata are downloaded during resolution
// Repositories use scalaz Task-s. Thus the line .run(repositories) implicitily
// runs the resolution process within Task-s. Alternative monads could run the process too.
assert(resolution.isDone) // Check that resolution converged
// Errors in
resolution.errors // Seq[(Dependency, Seq[String])], the Seq[String] contains the errors returned by each repository
// Artifact URLs in
resolution.artifacts // Seq[Artifact]
// Artifact has in particular a field url: String
// Now if we want to download or cache the artifacts, add in build.sbt
// "com.github.alexarchambault" %% "coursier-files" % "0.1.0-SNAPSHOT"
val files = Files(
Seq(
Repository.mavenCentral.fetch.root -> // URL with this prefix cached in directory:
new java.io.File(sys.props("user.home") + "/.coursier/cache/central/files")
),
() => ???, // TODO Tmp directory for URLs with no cache
Some(logger) // Optional, logger: FilesLogger
)
val cachePolicy = Repository.CachePolicy.Default
for (artifact <- artifacts)
files.file(artifact, cachePolicy).run match {
case -\/(err) => // Download failed, err: String
case \/-(file) => // Success, file: java.io.File
}
// Artifacts can be downloaded in parallel thanks to Task.gatherUnordered
// See the example in cli/src/main/scala/coursier/Coursier.scala
```

20
cache/src/main/scala/coursier/Cache.scala vendored
View File

@ -395,8 +395,8 @@ object Cache {
def file(
artifact: Artifact,
cache: Seq[(String, File)],
cachePolicy: CachePolicy,
cache: Seq[(String, File)] = default,
cachePolicy: CachePolicy = CachePolicy.FetchMissing,
checksums: Seq[Option[String]] = Seq(Some("SHA-1")),
logger: Option[Logger] = None,
pool: ExecutorService = defaultPool
@ -446,8 +446,8 @@ object Cache {
}
def fetch(
cache: Seq[(String, File)],
cachePolicy: CachePolicy,
cache: Seq[(String, File)] = default,
cachePolicy: CachePolicy = CachePolicy.FetchMissing,
checksums: Seq[Option[String]] = Seq(Some("SHA-1")),
logger: Option[Logger] = None,
pool: ExecutorService = defaultPool
@ -473,6 +473,18 @@ object Cache {
"[artifact](-[classifier]).[ext]"
)
lazy val defaultBase = new File(
sys.env.getOrElse(
"COURSIER_CACHE",
sys.props("user.home") + "/.coursier/cache/v1"
)
)
lazy val default = Seq(
"http://" -> new File(defaultBase, "http"),
"https://" -> new File(defaultBase, "https")
)
val defaultConcurrentDownloadCount = 6
lazy val defaultPool =

2
cache/src/main/scala/coursier/Platform.scala vendored
View File

@ -56,6 +56,6 @@ object Platform {
implicit def fetch(
repositories: Seq[core.Repository]
): Fetch.Metadata[Task] =
Fetch(repositories, Platform.artifact)
Fetch.from(repositories, Platform.artifact)
}

12
cli/src/main/scala-2.11/coursier/cli/Coursier.scala
View File

@ -45,18 +45,10 @@ case class CommonOptions(
val verbose0 = verbose.length - (if (quiet) 1 else 0)
}
object CacheOptions {
def default =
sys.env.getOrElse(
"COURSIER_CACHE",
sys.props("user.home") + "/.coursier/cache"
)
}
case class CacheOptions(
@HelpMessage("Cache directory (defaults to environment variable COURSIER_CACHE or ~/.coursier/cache)")
@HelpMessage("Cache directory (defaults to environment variable COURSIER_CACHE or ~/.coursier/cache/v1)")
@ExtraName("C")
cache: String = CacheOptions.default
cache: String = Cache.defaultBase.toString
)
sealed trait CoursierCommand extends Command

2
cli/src/main/scala-2.11/coursier/cli/Helper.scala
View File

@ -210,7 +210,7 @@ class Helper(
val fetchs = cachePolicies.map(p =>
Cache.fetch(caches, p, logger = logger, pool = pool)
)
val fetchQuiet = coursier.Fetch(
val fetchQuiet = coursier.Fetch.from(
repositories,
fetchs.head,
fetchs.tail: _*

2
core/shared/src/main/scala/coursier/Fetch.scala
View File

@ -80,7 +80,7 @@ object Fetch {
}
}
def apply[F[_]](
def from[F[_]](
repositories: Seq[core.Repository],
fetch: Content[F],
extra: Content[F]*

2
core/shared/src/main/scala/coursier/core/Repository.scala
View File

@ -8,7 +8,7 @@ import scalaz._
import coursier.core.compatibility.encodeURIComponent
trait Repository {
trait Repository extends Product with Serializable {
def find[F[_]](
module: Module,
version: String,

2
core/shared/src/main/scala/coursier/core/ResolutionProcess.scala
View File

@ -8,7 +8,7 @@ import scala.annotation.tailrec
sealed trait ResolutionProcess {
def run[F[_]](
fetch: Fetch.Metadata[F],
maxIterations: Int = -1
maxIterations: Int = 50
)(implicit
F: Monad[F]
): F[Resolution] = {

425
doc/README.md
View File

@ -2,7 +2,7 @@
*Pure Scala Artifact Fetching*
A pure Scala substitute for [Aether](http://www.eclipse.org/aether/)
A Scala library to fetch dependencies from Maven / Ivy repositories
[![Build Status](https://travis-ci.org/alexarchambault/coursier.svg?branch=master)](https://travis-ci.org/alexarchambault/coursier)
[![Build status (Windows)](https://ci.appveyor.com/api/projects/status/trtum5b7washfbj9?svg=true)](https://ci.appveyor.com/project/alexarchambault/coursier)
@ -12,11 +12,11 @@ A pure Scala substitute for [Aether](http://www.eclipse.org/aether/)
*coursier* is a dependency resolver / fetcher *à la* Maven / Ivy, entirely
rewritten from scratch in Scala. It aims at being fast and easy to embed
in other contexts. Its very core (`core` module) aims at being
extremely pure, and should be approached thinking algebraically.
extremely pure, and only requires to be fed external data (Ivy / Maven metadata) via a monad.
The `files` module handles caching of the metadata and artifacts themselves,
The `cache` module handles caching of the metadata and artifacts themselves,
and is less so pure than the `core` module, in the sense that it happily
does IO as a side-effect (although it naturally favors immutability for all
does IO as a side-effect (always wrapped in `Task`, and naturally favoring immutability for all
that's kept in memory).
It handles fancy Maven features like
@ -26,42 +26,222 @@ It handles fancy Maven features like
* [properties](http://books.sonatype.com/mvnref-book/reference/resource-filtering-sect-properties.html),
* etc.
It happily resolves dependencies involving modules from the Hadoop ecosystem (Spark, Flink, etc.), that
make a heavy use of these.
and is able to fetch metadata and artifacts from both Maven and Ivy repositories.
It can be used either from the command-line, via its API, or from the browser.
Compared to the default dependency resolution of SBT, it adds:
* downloading of artifacts in parallel,
* better offline mode - one can safely work with snapshot dependencies if these are in cache (SBT tends to try to fail if it cannot check for updates),
* non obfuscated cache (cache structure just mimicks the URL it caches).
It downloads the metadata or the artifacts in parallel (usually, 6 parallel
downloads).
From the command-line, it also has:
* a [launcher](#launch), able to launch apps distributed via Maven / Ivy repositories,
* a [bootstrap](#bootstrap) generator, able to generate stripped launchers of these apps.
## Command-line
Lastly, it can be used programmatically via its [API](#api) and has a Scala JS [demo](#scala-js-demo).
## Table of content
1. [Quick start](#quick-start)
2. [Why](#why)
3. [Usage](#usage)
1. [SBT plugin](#sbt-plugin)
2. [Command-line](#command-line)
3. [API](#api)
4. [Scala JS demo](#scala-js-demo)
4. [Contributors](#contributors)
5. [Projects using coursier](#projects-using-coursier)
## Quick start
The default global cache used by coursier is `~/.coursier/cache/v1`. E.g. the artifact at
`https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.7/scala-library-2.11.7.jar`
will land in `~/.coursier/cache/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.7/scala-library-2.11.7.jar`.
From the SBT plugin, the default repositories are the ones provided by SBT (typically Central or JFrog, and `~/.ivy2/local`).
From the CLI tools, these are Central (`https://repo1.maven.org/maven2`) and `~/.ivy2/local`.
From the API, these are specified manually - you are encouraged to use those too.
* SBT plugin
Enable the SBT plugin by adding
```scala
addSbtPlugin("com.github.alexarchambault" % "coursier-sbt-plugin" % "1.0.0-M2")
```
to `~/.sbt/0.13/plugins/build.sbt` (enables it globally), or to the `project/plugins.sbt` file
of a SBT project. Tested with SBT 0.13.8 / 0.13.9.
* CLI
Download and run its laucher with
```
$ curl -L -o coursier https://git.io/vBSmI && chmod +x coursier && ./coursier --help
$ curl -L -o coursier https://git.io/vEpQR && chmod +x coursier && ./coursier --help
```
Note that the launcher itself weights only 8 kB and can be easily
embedded as is in other projects.
The first time it is run, it will download the artifacts required to launch
coursier. You'll be fine the next times :-).
Run an application distributed via artifacts with
```
$ ./coursier launch com.lihaoyi:ammonite-repl_2.11.7:0.5.2
```
The cache of this default launcher defaults to a directory named `.coursier`,
in the same directory as the launcher. This can be changed by manually adjusting
the `COURSIER_CACHE` variable in the first lines of the launcher.
Download and list the classpath of one or several dependencies with
```
$ ./coursier fetch org.apache.spark:spark-sql_2.11:1.5.2 com.twitter:algebird-spark_2.11:0.11.0
Dependencies:
org.apache.spark:spark-sql_2.11:1.5.2
com.twitter:algebird-spark_2.11:0.11.0
Fetching artifacts
/path/to/.coursier/cache/v1/https/repo1.maven.org/maven2/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar
/path/to/.coursier/cache/v1/https/repo1.maven.org/maven2/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar
/path/to/.coursier/cache/v1/https/repo1.maven.org/maven2/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar
/path/to/.coursier/cache/v1/https/repo1.maven.org/maven2/com/typesafe/config/1.2.1/config-1.2.1.jar
...
```
* API
Add to your `build.sbt`
```scala
libraryDependencies ++= Seq(
"com.github.alexarchambault" %% "coursier" % "1.0.0-M2",
"com.github.alexarchambault" %% "coursier-cache" % "1.0.0-M2"
)
```
Add an import for coursier,
```tut:silent
import coursier._
```
To resolve dependencies, first create a `Resolution` case class with your dependencies in it,
```tut:silent
val start = Resolution(
Set(
Dependency(
Module("org.scalaz", "scalaz-core_2.11"), "7.2.0"
),
Dependency(
Module("org.spire-math", "cats-core_2.11"), "0.3.0"
)
)
)
```
Create a fetch function able to get things from a few repositories via a local cache,
```tut:silent
val repositories = Seq(
Cache.ivy2Local,
MavenRepository("https://repo1.maven.org/maven2")
)
val fetch = Fetch.from(repositories, Cache.fetch())
```
Then run the resolution per-se,
```tut:silent
val resolution = start.process.run(fetch).run
```
That will fetch and use metadata.
Check for errors in
```tut:silent
val errors: Seq[(Dependency, Seq[String])] = resolution.errors
```
These would mean that the resolution wasn't able to get metadata about some dependencies.
Then fetch and get local copies of the artifacts themselves (the JARs) with
```tut:silent
import java.io.File
import scalaz.\/
import scalaz.concurrent.Task
val localArtifacts: Seq[FileError \/ File] = Task.gatherUnordered(
resolution.artifacts.map(Cache.file(_).run)
).run
```
## Why
The current state of dependency management in Scala suffers several flaws, that prevent applications to fully
profit from and rely on dependency management. Coursier aims at addressing these by making it easy to:
- resolve / download dependencies programmatically,
- launch applications distributed via Maven / Ivy artifacts from the command-line,
- work offline with artifacts,
- sandbox dependency management between projects.
As its [API](#api) illustrates, getting artifacts of dependencies is just a matter of specifying these along
with a few repositories. You can then straightforwardly get the corresponding artifacts, easily getting
precise feedback about what goes on during the resolution.
Launching an application distributed via Maven artifacts is just a command away with the [launcher](#command-line) of coursier.
In most cases, just specifying the corresponding main dependency is enough to launch the corresponding application.
If all your dependencies are in cache, chances are coursier will not even try to connect to remote repositories. This
also applies to snapshot dependencies of course - these are only updated on demand, not getting constantly in your way
like is currently the case by default with SBT.
When using coursier from the command-line or via its SBT plugin, sandboxing is just one command away. Just do
`export COURSIER_CACHE="$(pwd)/.coursier-cache"`, and the cache will become `.coursier-cache` from the current
directory instead of the default global `~/.coursier/cache/v1`. This allows for example to quickly inspect the content
of the cache used by a particular project, in case you have any doubt about what's in it.
## Usage
### SBT plugin
Enable the SBT plugin globally by adding
```scala
addSbtPlugin("com.github.alexarchambault" % "coursier-sbt-plugin" % "1.0.0-M2")
```
to `~/.sbt/0.13/plugins/build.sbt`
To enable it on a per-project basis, add it only to the `project/plugins.sbt` of a SBT project.
The SBT plugin has been tested only with SBT 0.13.8 / 0.13.9.
Once enabled, the `update`, `updateClassifiers`, and `updateSbtClassifiers` commands are taken care of by coursier. These
provide more output about what's going on than their default implementations do.
```tut:invisible
// TODO Change cache policy, sandboxing, parallel downloads, limitations
```
### Command-line
Download and run its laucher with
```
$ curl -L -o coursier https://git.io/vEpQR && chmod +x coursier && ./coursier --help
```
The launcher itself weights only 8 kB and can be easily embedded as is in other projects.
It downloads the artifacts required to launch coursier on the first run.
```
$ ./coursier --help
```
lists the available coursier commands. The most notable ones are `launch`,
`fetch`, and `classpath`. Type
lists the available coursier commands. The most notable ones are `launch`, and `fetch`. Type
```
$ ./coursier command --help
```
to get a description of the various options the command `command` (replace with one
of the above command) accepts.
### launch
Both command belows can be given repositories with the `-r` or `--repository` option, like
```
-r central
-r https://oss.sonatype.org/content/repositories/snapshots
-r "ivy:https://repo.typesafe.com/typesafe/ivy-releases/[organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]"
```
`central` and `ivy2local` correspond to Maven Central and `~/.ivy2/local`. These are used by default
if no `-r` or `--repository` option is specified.
As soon as a `-r` or `--repository` option is specified, these default are not used any more - only the
specified repositories are used.
Repositories starting with `ivy:` are assumed to be Ivy repositories, specified with an Ivy pattern. Else,
a Maven repository is assumed.
#### launch
The `launch` command fetches a set of Maven coordinates it is given, along
with their transitive dependencies, then launches the "main `main` class" from
@ -72,7 +252,7 @@ For example, it can launch:
* [Ammonite](https://github.com/lihaoyi/Ammonite) (enhanced Scala REPL),
```
$ ./coursier launch com.lihaoyi:ammonite-repl_2.11.7:0.5.0
$ ./coursier launch com.lihaoyi:ammonite-repl_2.11.7:0.5.2
```
along with the REPLs of various JVM languages like
@ -115,10 +295,10 @@ $ ./coursier launch net.sf.proguard:proguard-base:5.2.1 -M proguard.ProGuard
$ ./coursier launch net.sf.proguard:proguard-retrace:5.2.1 -M proguard.retrace.ReTrace
```
### fetch
#### fetch
The `fetch` command simply fetches a set of dependencies, along with their
transitive dependencies, then prints the local paths of all their artefacts.
transitive dependencies, then prints the local paths of all their artifacts.
Example
```
@ -130,52 +310,193 @@ $ ./coursier fetch org.apache.spark:spark-sql_2.11:1.5.2
...
```
### classpath
The `classpath` command transitively fetches a set of dependencies like
`fetch` does, then prints a classpath that can be handed over directly
to `java`, like
By adding the `-p` option, these paths can be handed over directly to
`java -cp`, like
```
$ java -cp "$(./coursier classpath com.lihaoyi:ammonite-repl_2.11.7:0.5.0 | tail -n1)" ammonite.repl.Repl
$ java -cp "$(./coursier fetch -p com.lihaoyi:ammonite-repl_2.11.7:0.5.2)" ammonite.repl.Main
Loading...
Welcome to the Ammonite Repl 0.5.0
(Scala 2.11.7 Java 1.8.0_60)
Welcome to the Ammonite Repl 0.5.2
(Scala 2.11.7 Java 1.8.0_51)
@
```
## API
This [gist](https://gist.github.com/larsrh/42da43aa74dc4e78aa59) by [Lars Hupel](https://github.com/larsrh/)
illustrates how the API of coursier can be used to get transitives dependencies
and fetch the corresponding artefacts.
```tut:invisible
// TODO options: repositories, sources/javadoc, attributes
```
More explanations to come :-)
### bootstrap
## Scala JS demo
The `bootstrap` generates tiny bootstrap launchers, able to pull their dependencies from
repositories on first launch. For example, the launcher of coursier is [generated](https://github.com/alexarchambault/coursier/blob/master/project/generate-launcher.sh) with a command like
```
$ ./coursier bootstrap \
com.github.alexarchambault:coursier-cli_2.11:1.0.0-M2 \
-b -f -o coursier \
-M coursier.cli.Coursier
```
See `./coursier bootstrap --help` for a list of the available options.
### API
Add to your `build.sbt`
```scala
libraryDependencies ++= Seq(
"com.github.alexarchambault" %% "coursier" % "1.0.0-M2",
"com.github.alexarchambault" %% "coursier-cache" % "1.0.0-M2"
)
```
The first module, `"com.github.alexarchambault" %% "coursier" % "1.0.0-M2"`, mainly depends on
`scalaz-core` (and only it, *not* `scalaz-concurrent` for example). It contains among others,
definitions,
mainly in [`Definitions.scala`](https://github.com/alexarchambault/coursier/blob/master/core/shared/src/main/scala/coursier/core/Definitions.scala),
[`Resolution`](https://github.com/alexarchambault/coursier/blob/master/core/shared/src/main/scala/coursier/core/Resolution.scala), representing a particular state of the resolution,
and [`ResolutionProcess`](https://github.com/alexarchambault/coursier/blob/master/core/shared/src/main/scala/coursier/core/ResolutionProcess.scala),
that expects to be given metadata, wrapped in any `Monad`, then feeds these to `Resolution`, and at the end gives
you the final `Resolution`, wrapped in the same `Monad` it was given input. This final `Resolution` has all the dependencies,
including the transitive ones.
The second module, `"com.github.alexarchambault" %% "coursier-cache" % "1.0.0-M2"`, is precisely in charge of fetching
these input metadata. It uses `scalaz.concurrent.Task` as a `Monad` to wrap them. It also fetches artifacts (JARs, etc.).
It caches all of these (metadata and artifacts) on disk, and validates checksums too.
In the code below, we'll assume some imports are around,
```tut:silent
import coursier._
```
Resolving dependencies involves create an initial resolution state, with all the initial dependencies in it, like
```tut:silent
val start = Resolution(
Set(
Dependency(
Module("org.spire-math", "cats-core_2.11"), "0.3.0"
),
Dependency(
Module("org.scalaz", "scalaz-core_2.11"), "7.2.0"
)
)
)
```
It goes without saying that a `Resolution` is immutable, as are all the classes defined in the core module.
The resolution process will go on by giving successive `Resolution`s, until the final one.
`start` above is only the initial state - it is far from over, as the `isDone` method on it tells,
```tut
start.isDone
```
```tut:invisible
assert(!start.isDone)
```
In order for the resolution to go on, we'll need things from a few repositories,
```tut
val repositories = Seq(
Cache.ivy2Local,
MavenRepository("https://repo1.maven.org/maven2")
)
```
The first one, `Cache.ivy2Local`, is defined in `coursier.Cache`, itself from the `coursier-cache` module that
we added above. As we can see, it is an `IvyRepository`, picking things under `~/.ivy2/local`. An `IvyRepository`
is related to the [Ivy](http://ant.apache.org/ivy/) build tool. This kind of repository involves a so-called [pattern](http://ant.apache.org/ivy/history/2.4.0/concept.html#patterns), with
various properties. These are not of very common use in Scala, although SBT uses them a bit.
The second repository in a `MavenRepository`. These are simpler than the Ivy repositories. They're the ones
we're the most used to in Scala. Common ones like [Central](https://repo1.maven.org/maven2) like here, or the repositories
from [Sonatype](https://oss.sonatype.org/content/repositories/), are Maven repositories. These originate
from the [Maven](https://maven.apache.org/) build tool. Unlike the Ivy repositories which involve customisable patterns to point
to the underlying metadata and artifacts, the paths of these for Maven repositories all look alike,
like for any particular version of the standard library, under paths like
[this one](http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.11.7/).
Both `IvyRepository` and `MavenRepository` are case classes, so that it's straightforward to specify one's own
repositories.
Now that we have repositories, we're going to mix these with things from the `coursier-cache` module,
for resolution to happen via the cache. We'll create a function
of type `Seq[(Module, String)] => F[Seq[((Module, String), Seq[String] \/ (Artifact.Source, Project))]]`.
Given a sequence of dependencies, designated by their `Module` (organisation and name in most cases)
and version (just a `String`), it gives either errors (`Seq[String]`) or metadata (`(Artifact.Source, Project)`),
wrapping the whole in a monad `F`.
```tut:silent
val fetch = Fetch.from(repositories, Cache.fetch())
```
The monad used by `Fetch.from` is `scalaz.concurrent.Task`, but the resolution process is not tied to a particular
monad - any stack-safe monad would do.
With this `fetch` method, we can now go on with the resolution. Calling `process` on `start` above gives a
[`ResolutionProcess`](https://github.com/alexarchambault/coursier/blob/master/core/shared/src/main/scala/coursier/core/ResolutionProcess.scala),
that drives the resolution. It is loosely inspired by the `Process` of scalaz-stream.
It is an immutable structure, that represents the various states the resolution process can be in.
Its method `current` gives the current `Resolution`. Calling `isDone` on the latter says whether the
resolution is done or not.
The `next` method, that expects a `fetch` method like the one above, gives
the "next" state of the resolution process, wrapped in the monad of the `fetch` method. It allows to do
one resolution step.
Lastly, the `run` method runs the whole resolution until its end. It expects a `fetch` method too,
and will make at most `maxIterations` steps (50 by default), and return the "final" resolution state,
wrapped in the monad of `fetch`. One should check that the `Resolution` it returns is done (`isDone`) -
the contrary means that `maxIterations` were reached, likely signaling an issue, unless the underlying
resolution is particularly complex, in which case `maxIterations` could be increased.
Let's run the whole resolution,
```tut:silent
val resolution = start.process.run(fetch).run
```
To get additional feedback during the resolution, we can give the `Cache.default` method above
a [`Cache.Logger`](https://github.com/alexarchambault/coursier/blob/cf269c6895e19f2d590f08811406724304332950/cache/src/main/scala/coursier/Cache.scala#L484-L490).
By default, downloads happen in a global fixed thread pool (with 6 threads, allowing for 6 parallel downloads), but
you can supply your own thread pool to `Cache.default`.
Now that the resolution is done, we can check for errors in
```tut:silent
val errors: Seq[(Dependency, Seq[String])] = resolution.errors
```
These would mean that the resolution wasn't able to get metadata about some dependencies.
We can also check for version conflicts, in
```tut:silent
val conflicts: Set[Dependency] = resolution.conflicts
```
which are dependencies whose versions could not be unified.
Then, if all went well, we can fetch and get local copies of the artifacts themselves (the JARs) with
```tut:silent
import java.io.File
import scalaz.\/
import scalaz.concurrent.Task
val localArtifacts: Seq[FileError \/ File] = Task.gatherUnordered(
resolution.artifacts.map(Cache.file(_).run)
).run
```
We're using the `Cache.file` method, that can also be given a `Logger` (for more feedback) and a custom thread pool.
### Scala JS demo
*coursier* is also compiled to Scala JS, and can be tested in the browser via its
[demo](http://alexarchambault.github.io/coursier/#demo).
# To do / missing
- Snapshots metadata / artifacts, once in cache, are not automatically
updated for now. [#41](https://github.com/alexarchambault/coursier/issues/41)
- File locking could be better (none for metadata, no re-attempt if file locked elsewhere for artifacts) [#71](https://github.com/alexarchambault/coursier/issues/71)
- Handle "configurations" like Ivy does, instead of just the standard
(hard-coded) Maven "scopes" [#8](https://github.com/alexarchambault/coursier/issues/8)
- SBT plugin [#52](https://github.com/alexarchambault/coursier/issues/52),
requires Ivy-like configurations [#8](https://github.com/alexarchambault/coursier/issues/8)
See the list of [issues](https://github.com/alexarchambault/coursier/issues).
# Contributors
## Contributors
- Your name here :-)
Don't hesitate to pick an issue to contribute, and / or ask for help for how to proceed
on the [Gitter channel](https://gitter.im/alexarchambault/coursier).
# Projects using coursier
## Projects using coursier
- [Lars Hupel](https://github.com/larsrh/)'s [libisabelle](https://github.com/larsrh/libisabelle) fetches
some of its requirements via coursier,

2
fetch-js/src/main/scala/coursier/Platform.scala
View File

@ -89,7 +89,7 @@ object Platform {
implicit def fetch(
repositories: Seq[core.Repository]
): Fetch.Metadata[Task] =
Fetch(repositories, Platform.artifact)
Fetch.from(repositories, Platform.artifact)
trait Logger {
def fetching(url: String): Unit

2
plugin/src/main/scala-2.10/coursier/CoursierPlugin.scala
View File

@ -39,7 +39,7 @@ object CoursierPlugin extends AutoPlugin {
coursierVerbosity := 1,
coursierResolvers <<= Tasks.coursierResolversTask,
coursierSbtResolvers <<= externalResolvers in updateSbtClassifiers,
coursierCache := new File(sys.props("user.home") + "/.coursier/sbt"),
coursierCache := Cache.defaultBase,
update <<= Tasks.updateTask(withClassifiers = false),
updateClassifiers <<= Tasks.updateTask(withClassifiers = true),
updateSbtClassifiers in Defaults.TaskGlobal <<= Tasks.updateTask(withClassifiers = true, sbtClassifiers = true),

2
plugin/src/main/scala-2.10/coursier/Tasks.scala
View File

@ -240,7 +240,7 @@ object Tasks {
val resLogger = createLogger()
val fetch = coursier.Fetch(
val fetch = Fetch.from(
repositories,
Cache.fetch(caches, CachePolicy.LocalOnly, checksums = checksums, logger = Some(resLogger), pool = pool),
Cache.fetch(caches, cachePolicy, checksums = checksums, logger = Some(resLogger), pool = pool)

2
tests/shared/src/test/scala/coursier/test/ResolutionTests.scala
View File

@ -166,7 +166,7 @@ object ResolutionTests extends TestSuite {
)
val projectsMap = projects.map(p => p.moduleVersion -> p.copy(configurations = MavenRepository.defaultConfigurations)).toMap
val testRepository = new TestRepository(projectsMap)
val testRepository = TestRepository(projectsMap)
val repositories = Seq[Repository](
testRepository

2
tests/shared/src/test/scala/coursier/test/TestRepository.scala
View File

@ -6,7 +6,7 @@ import coursier.core._
import scalaz.{ Monad, EitherT }
import scalaz.Scalaz._
class TestRepository(projects: Map[(Module, String), Project]) extends Repository {
case class TestRepository(projects: Map[(Module, String), Project]) extends Repository {
val source = new core.Artifact.Source {
def artifacts(
dependency: Dependency,