skycracking

Continuous Integration using TFS, NuGet, and Artifactory

This blog shows how Artifactory, a binary repository manager, can be used a) as the storage location for remotely located build references, b) as a drop site for locally built CI artifacts, c)and in a future blog how it can also function as a storage and of all these binaries.

For this demo we will use the MyLogger solution.MyLogger.sln consumes third-party references during its build process and produces a library exposing a simple logging method. This library which will later be consumed from Artifactory by a second solution in a follow-up blog. The third party remotely located resources used by MyLogger will be cached in Artifactory.

Artifactory, combined with NuGet, allows TFS users to retrieve the latest binaries from their local CI builds while automatically updating referenced binaries coming from multiple remote repositories. Using Artifactory in this manner will limit the downtime of CI builds due to limited internet access.

First, Let’s Download the Source Code Example

You can clone, fork and clone, or use the ?download? tab to get a zip file of the code. After downloading the source code, create a TFS project (TFVC or GIT) using the Default Scrum Template in TFS 2013. Add the source to the TFS project.

Your source code tree should look like this:

source code tree


WARNING: You should wait before opening the solution in Visual Studio. NuGet and Artifactory needs to be installed first or there may be some attempts to find the packages from other sources.

Second, Let?s Setup of an Artifactory Pro Server

Standard Setup:

For integration with NuGet, the Pro version will be needed. The Java SDK 1.7+ is a requirement for running Artifactory. The Artifactory Pro Server may be located on one?s local workstation, the TFS host, an unrelated machine, or even exist in a cloud. For our purpose here our local TFS server will host the Artifactory Pro services.

  1. Download and install Java SDK 1.7+ (create an environment variable for $JAVA_HOME)http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
  2. Download and install an evaluation copy of Artifactory Pro to run as service. For this purpose we recommend running as a service. In the \bin folder use the artifactory-service.exe

Customizations:

After the server is installed with a valid Pro temporary license (license will be emailed from JFrog), and before starting the Artifactory services replace the following two files with the one?s provided in the source code download from Codeplex. These two files create the repositories and setup the user accounts.

  • security.xml = $(Artifactory-Home)\etc\security.import.xml
  • artifactory.config.xml = $(Artifactory-Home)\etc\artifactory.config.import.html

To manage TFS related remote and locally produced binaries in Artifactory, we will need some custom repositories (see below – these were automatically created when you replace the artifactory.config.xml file above). A reference of instructions can be found at: http://www.jfrog.com/confluence/display/RTF/NuGet+Repositories
custom repositories

nuget-staging-local — This repository will be the storage location for the local built libraries prior to QA testing and approval. The latest libraries here can be used for integration into the latest development code for testing.

nuget-release-local — Repository to store the ?release? versions of all packages. This repository will be the final storage for the locally built and QA tested/approved for consumption or production.

nuget-symbols-local — Repository for storing the symbols from the CI build. This repository will be the location of the symbols that match the builds found in the nuget-staging-local repository. These symbols will be pulled and deployed for debugging purposes by the developers.

jcenter-cache — Local repository to ?proxy? the BinTray/JCenter gallery. This repository will cache the libraries that will be consumed in the builds from jcenter.bintray.com. This provides a local source for these libraries and maybe be useful for use with TeamCity plugins and the Artifactory Remote Search.

nuget-gallery-cache — Local repository to ?proxy? the nuget gallery. This repository will cache the libraries that will be consumed in the builds from nuget.org. This provides a local source for these libraries.

Now, Create a Build Definitions in Visual Studio

After the source code for the MyLogger solution has been downloaded and is in a TFS project, we need to create a build definition.  This build definition is based on the default process template and will use default settings with the exceptions highlighted and described below.

MyLogger Build Definition

MyLogger Build Definition
\NuGetExample\MyLogger\MyLogger.sln is the solution that will build and store the MyLogger.dll

/p:DistribPackages=true — This is a custom MSBuild task created inside the Nuget.targets file. The DistribPackages task is mean to “push” the Nuget packages from the build site to the local Artifactory repository.

When NuGet Management is added to a solution in TFS it adds a NuGet.exe, a NuGet.config, and a NuGet.targets file to the Solution File. In this project we check in the NuGet.config and the Nuget.targets file, but not the NuGet.exe.  The NuGet.targets file is customized to contain the MSBuild targets needed to deploy to Artifactory.

\\tfs\builds\symbols — creates a drop site for the symbols that are generated for this build. In this example we do not capture all the generated symbols and test results.

$(BuildDefinitionName)_2.0.1$(Rev:.r) – This will give allow the user to set the Major.Minor.Build number. This revision number allows TFS to increment the Revision number used in the “get last revision” of MyLogger when building the Multi-Example project.

Time to Integrate NuGet with Visual Studio

After the build definition has been created, we need to integrate the ?NuGet Package Manager? with Visual Studio solutions. Install instructions follow below:

  • In Visual Studio, go to Tools –> NuGet Package Manager –> Package Manager Settings
  • General –> Package Restore – Turn ON “Automatically check for missing packages during build in Visual Studio”

Package Restore

Package Sources – Make sure only the local Artifactory is being used for package sources. We want everything used in the build to be cached inside of the local Artifactory server.  Changes made here will be reflected in the %APPDATA%\Roaming\NuGet\NuGet.config.

Package Sources

By creating the integration of the Nuget Management Packages add-in to a solution within Visual Studio, the follow files have been modified as described below and will need to be checked in. The project and solution files should also be checked in after the integration.

NuGet.config file

The Artifactory server path will be added. Make sure the Artifactory key is pointing to the correct location and only one exists.

     <packageSources>
        <add key=”Artifactory” value=”http://tfs:8081/artifactory/api/nuget/staging” />
     </packageSources>

Nuget.targets file

The nuget.targets file is where the majority of custom changes were made to implement the integration of Artifactory, NuGet and TFS. These commands and targets can be viewed in the nuget.targets file.

  • New Command UpdateCommand
  • New Command DistribCommand
  • Set Build to Depend on DistribPackages
  • New Target UpdatePackages ? updates the packages folder from the packages.config if needed.
  • New Target DistribPackages ? runs the NuGet push command to Artifactory
  • New Target TFSEnvVarTest ? get the path to msbuild.exe on the build server
  • New Task TFSEnvVar

*.sln file

The nuget project will be added to the solution file.

*.csproj files

During the VS?s GUI integration of NuGet to the solution a NuGet project will be imported and a new target will be created in each .csproj file found in the solution:


       <Import Project=”$(SolutionDir)\.nuget\NuGet.targets”

<Target Name=”EnsureNuGetPackageBuildImports”

Next, Build MyLogger and Deploy to Artifactory

Verify that the third party reference is coming from Artifactory and is managed by NuGet.
To verify the NuGet packages are actually coming from Artifactory, view the Artifactory \logs\request.log. Since we are doing a clean build the \packages folder will be empty and should now be populated only from Artifactory.

nuget gallery cache

Execute the build definition. If the build definition has been correctly setup, after queuing, we should see a BuildDefinition name followed by the Major, Minor, Build and Revision. After the build completes we will need to review the MSBuild log to verify that the version of Newtonsoft.Json file was correctly updated during the build.

BuildDefinition name
The TFS drop site, if it was setup in the build definition, will contain the logs, the libraries, and the NuGet packages. This drop site has been created for reference. All the files found here will also be found in the nuget package found in the Artifactory nuget-staging-local repository.

TFS drop site


In the MSBuild log find a RestorePackages section. References to NuGet packages, regardless of where they originate, are stored in the projects packages.config file parallel to each .csproj file.


Here we are reading the package.config file of each project within the solution and executing a NuGet update command through MSBuild. If packages are missing or a newer version is available, a MsBuild task in the nuget.targets file will download these files to the Packages folder.


During the build of the multiple-example project, we use a pre-build powershell script to execute MSBuld prior to the Try, Compile, so that the packages are updated prior to the compile of the code. Otherwise, the packages.config would already be loaded and will not get updated prior to the build.

powershell script
The BuildPackage section, in the MSBuild log, will show which NuGet packages have been created during the build and where the references to the dependencies packages are derived from.

BuildPackage section
The DistribPackages section in the MsBuild log file shows where the NuGet packages were “pushed” to by the NuGet application.

DistribPackages section
The nuget-staging-local repository is the location where we ?drop? the contents of the build after they have been zipped up into a NuGet package. From this local repository we can reference specific versions to include in other builds, we can pull specific versions to be deployed to an environment for testing, and we can promote from here to another repository such as nuget-release-local for later use in production or to be consumed by other developer teams. These concepts will be demonstrated in a follow-up blog.

nuget-staging-local
This concludes the first section of this blog.

A future continuation blog will cover the promotion of this package to a “release-local” repository, security and access, and the consumption of this package by other builds to produce new packages which will be delivered into a CI production pipeline.

Leave a Reply

Please type the code shown*


Can't read the image? click here to refresh

It's Time to Trust Your Software! Artifactory Bintray JFrog Mission Control Xray
Popular Posts