Savant Build Tool: Dependency Management

Brian Pontarelli

When we first started writing Savant, we realized that the first requirement was a dependency management system. This is necessary for nearly every feature in a good build tool. We wanted Savant plugins to be resolved as dependencies and to be loaded into separate classloaders along with their dependencies. We also knew that the Java compiler plugin would need the project’s dependencies for the compiler classpath and the TestNG plugin would need the dependencies to run the tests.

Savant Dependency Management BlocksFor years we had been using a strict versioning system that broke versions into 4 different groups:

  • Major
  • Minor
  • Patch
  • Pre-release

This versioning is expressed as <major>.<minor>.<patch>-<pre-release>

We found that the Semantic Versioning (SemVer) specification was nearly identical to our versioning methodology. Rather than continue to use our own system, we fully embraced SemVer. The problem was that no other dependency management system implemented SemVer.

Maven and Ivy both were good initial solutions for dependency management, however they both lacked any rules with regards to version compatibility or version naming. Therefore, we couldn’t use Maven Central or the Ivy repositories we had previously been using with Gradle.

This was a difficult decision because it meant that we had to start from scratch and were going to have to manually add dependencies to the repository until Savant became more widely adopted.

Even though these tradeoffs were huge, we chose not to compromise on the integrity of the software and more importantly on SemVer. Rather, we implemented a semantic version compliant dependency management solution.

Before we dive into how Savant’s dependency management tool works, there are three terms that require definition:

  • artifact
    • An artifact is file that is produced by a project and available for other projects to use. Examples of artifacts include JAR files, SQL files, ZIPs, RPMs, etc.
  • dependency
    • A dependency is formed when a project depends on an artifact of another project.
  • transitive dependencies
    • Transitive dependencies are indirect dependencies of your project through a direct dependency. For example, if you project depends on library A and library A depends on library B, your project transitively depends on library B.

In Savant, dependencies are identified by an ID and a version. The ID has four parts:

  • Group
  • Project
  • Name
  • Type

The Group is always a reverse DNS name that identifies the organization responsible for the artifact. The Project is the name of the project that produces the artifact. The Name is the name of the artifact and can be different than the Project, but is often the same as the Project. When a project only projects a single artifact, the Project and Name are almost always the same. The Type is the file type of the artifact. This is usually jar for Java artifacts, but can be any type. For example, Inversoft often uses tar.gz for our bundle artifacts.

In addition to the ID, the dependency also requires a version. There are a couple shorthand notations for Savant dependencies:

  • <group>:<name>:<version>
    • i.e. org.apache.commons:commons-collections:3.1.0
  • <group>:<name>:<version>:<type>
    • i.e. com.mycompany:database:1.0.1:sql
  • <group>:<project>:<name>:<version>:<type>
    • i.e. com.mycompany:database-project:mysql:1.0.1:sql

Dependencies are defined inside the project build file using the dependencies definition. Dependencies are broken down into named dependency groups. Here is an example:

project(...) {
  dependencies {
    group(name: "provided") {
      dependency(id: "javax.servlet:servlet-api:3.1.0", skipCompatibilityCheck: true)
    }
    group(name: "compile") {
     dependency(id: "com.fasterxml.jackson.core:jackson-annotations:2.3.0")
      dependency(id: "com.fasterxml.jackson.core:jackson-core:2.3.0")
      dependency(id: "com.fasterxml.jackson.core:jackson-databind:2.3.0")
      dependency(id: "com.google.inject:guice:4.0-beta.4")
      dependency(id: "com.jolbox:bonecp:0.8.0")
      dependency(id: "javax.inject:javax.inject:1")
      dependency(id: "javax.mail:mail:1.4.4")
      dependency(id: "net.sf.trove4j:trove4j:3.0.3")
      dependency(id: "org.apache.commons:commons-io:2.1")
      dependency(id: "org.apache.commons:commons-lang3:3.1")
      dependency(id: "org.mybatis:mybatis:3.2.7")
      dependency(id: "org.mybatis:mybatis-guice:3.6")
      dependency(id: "org.primeframework:prime-email:0.10.0")
      dependency(id: "org.primeframework:prime-mvc:0.32.0")
      dependency(id: "org.primeframework:prime-transformer:1.0.3")
      dependency(id: "org.slf4j:slf4j-api:1.7.7")
      dependency(id: "org.supercsv:supercsv:1.52")
    }
    group(name: "test-compile", export: false) {
      dependency(id: "org.easymock:easymock:3.2.0")
     dependency(id: "org.primeframework:prime-mock:0.5.0")
      dependency(id: "org.testng:testng:6.8.7")
    }
    group(name: "test-runtime", export: false) {
      dependency(id: "com.mysql:mysql-connector-java:5.1.32")
      dependency(id: "org.postgresql:postgresql:9.3.1102+jdbc41")
    }
    group(name: "database", export: false) {
      dependency(id: “com.mycompany:database:mysql:1.0.1:sql”)
   }
   group(name: "stopwords", export: false) {
     dependency(id: “com.mycompany:stopwords:stopwords:3.0.1:txt”)
    }
  }
}

As you can see, some of the dependency groups look familiar such as compile and test-runtime. However, the groups are completely free-form and as you can see we have defined two additional groups named database and stopwords. This project might use those artifacts to create the database and populate a table with some stop words.

Next, Savant needs to know how to download the project dependencies. Savant downloads dependencies using a workflow. The workflow is also defined in the project definition. It is broken down into two sections: fetch and publish. The fetch section is how Savant locates and downloads the dependencies initially and the publish section is used to store the dependencies locally so they aren’t downloaded during each build. Each section contains 1 or more processes that define how the artifacts are fetched and stored.

Here is a common workflow definition:

project(...) {
  workflow {
    fetch {
      cache()
      url(url: "http://savant.mycompany.com")
    }
    publish {
      cache()
    }
  }
}

This tells Savant to first check the local cache to see if the dependency already exists. If it doesn't, it attempts to download it using HTTP from the repository http://savant.mycompany.com. If it is found at that location, it is stored in the local cache.

Savant does not provide a default workflow or attempt to download artifacts from Savant Central (currently http://savant.inversoft.org). We feel that defaults like this rarely work in secure corporate environments and therefore decided not to include a default workflow. This requires a bit of additional work when creating your build file, but it adds clarity and control.

Finally, we want to mention that by default, certain Savant plugins use specific dependency groups in specific ways. However, these settings are configurable, so most projects should not need to change these settings. Here are the dependency groups and how they are used:

  • compile
    • These are artifacts your project depends on in order to compile your source code.
    • The Java and Groovy plugins use this group during compilation. However, transitive dependencies are NOT included. If your project needs an artifact at compile time it must define that as a direct dependency.
  • provided
    • These are artifacts your project depends on in order to compile your source code, but are provided to you at runtime. The classic example of this is the servlet-api JAR.
    • The Java and Groovy plugins use this group during compilation. However, transitive dependencies are NOT included.
  • runtime
    • These are artifacts your project depends on at runtime but not compile time. Usually these are implementations of an API that your project compiles against.
    • The Java TestNG and Groovy TestNG plugin include these dependencies and all their transitive dependencies when running the tests.
    • The Webapp plugin places all of these dependencies in the WEB-INF/lib folder.
  • test-compile
    • These are artifacts your project’s tests depend on in order to be compiled.
    • The Java and Groovy plugins use this group during compilation. However, transitive dependencies are NOT included.
  • test-runtime
    • These are artifacts your project depends on in order to run the tests.
    • The Java TestNG and Groovy TestNG plugin include these dependencies and all their transitive dependencies when running the tests.

This has been a brief overview of the Savant dependency management system. We have covered how dependencies are defined, downloaded, and cached by Savant. We have also briefly covered the different dependency groups and the plugins that use them. Savant has additional dependency management features and other plugins that can assist you in downloading and using dependency artifacts. You can learn more about the various features and plugins via the Savant wiki here:

http://github.com/inversoft/savant-core/wiki

Stay tuned for our next blog post about Savant version compatibility and management.

 


For more about Inversoft Open Source projects click on the image below.

Inversoft Open Source Projects

Tags:
Savant