Setup an Automated Build using Jenkins and Artifactory

Description

Such environment setup facilitates continuous integration (CI) and continuous delivery (CD), which improves productivity across your development ecosystem, by automatically:

  • building modules when source code changes

  • saving build results

  • reproducing builds

  • archiving binary modules

This guide should take 1 hour to complete.

Intended Audience

The audience for this document is engineers who are in charge of integrating MicroEJ Module Manager (MMM) to their continuous integration environment.

In addition, this document should be of interest to all developers wishing to understand how MicroEJ works with headless module builds.

For those who are only interested in command line module build, consider using the MMM Command Line Interface.

Prerequisites

This training was tested with Jenkins 2.426.1, Artifactory 7.71.5 and Gitea 1.21.1.

Note

For SDK versions before 5.4.0, please refer to this MicroEJ Documentation Archive.

Introduction

The overall build and deployment flow of a module can be summarized as follows:

  1. Some event triggers the build process (i.e module source changed, user action, scheduled routine, etc.)

  2. The module source code is retrieved from the Source Control System

  3. The module dependencies are imported from the Repository Manager

  4. The Automation Server then proceeds to building the module

  5. If the build is successful, the module binary is deployed to the Repository Manager

../_images/tuto_microej_cli_flow.PNG

Overview

The next sections describe step by step how to setup the build environment and build your first MicroEJ module.

The steps to follow are:

  1. Run and setup Jenkins, Artifactory and Gitea

  2. Create a simple MicroEJ module (Hello World)

  3. Create a new Jenkins job for the Hello World module

  4. Build the module

In order to simplify the steps, this training will be performed locally on a single machine.

Artifactory will host MicroEJ modules in 3 repositories:

  • microej-module-repository: repository initialized with pre-built MicroEJ modules, a mirror of the Central Repository

  • custom-modules-snapshot: repository where custom snapshot modules will be published

  • custom-modules-release: repository where custom release modules will be published

Prepare your Docker environment

This section assumes the prerequisites have been properly installed.

  1. Create a new directory, inside create a file named docker-compose.yaml and copy this content:

    version: '3'
    services:
      artifactory:
        image: releases-docker.jfrog.io/jfrog/artifactory-oss:7.71.5
        container_name: artifactory
        environment:
          - JF_ROUTER_ENTRYPOINTS_EXTERNALPORT=8082
        ports:
          - 8082:8082 # for router communication
          - 8081:8081 # for artifactory communication
          - 8085:8085 # for artifactory federation communication
        volumes:
          - artifactory:/var/opt/jfrog/artifactory
          - /etc/localtime:/etc/localtime:ro
        restart: always
        logging:
          driver: json-file
          options:
            max-size: "50m"
            max-file: "10"
        ulimits:
          nproc: 65535
          nofile:
            soft: 32000
            hard: 40000
    
      gitea:
        image: gitea/gitea:1.21.1
        container_name: gitea
        environment:
          - USER_UID=1000
          - USER_GID=1000
        restart: always
        volumes:
          - gitea:/data
          - /etc/timezone:/etc/timezone:ro
          - /etc/localtime:/etc/localtime:ro
        ports:
          - "3000:3000"
          - "222:22"
    
      jenkins:
        image: jenkins_master
        container_name: jenkins
        build:
          dockerfile: Dockerfile
        restart: always
        ports:
          - 50000:50000
          - 8080:8080
        volumes:
          - jenkins:/var/jenkins_home
          - /var/run/docker.sock:/var/run/docker.sock
        links:
          - gitea
          - artifactory
    
    volumes:
      gitea:
      artifactory:
      jenkins:
    
  2. Create another file named Dockerfile and copy this content:

    FROM jenkins/jenkins:2.426.1-lts
    USER root
    RUN apt-get update -qq \
        && apt-get install -qqy apt-transport-https ca-certificates curl gnupg2 software-properties-common
    RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
    RUN add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/debian \
       $(lsb_release -cs) \
       stable"
    RUN apt-get update  -qq \
        && apt-get -y install docker-ce
    RUN usermod -aG docker jenkins
    
  3. In this directory, launch the command docker compose up -d. After a few moments you should have three running containers (named jenkins, gitea and artifactory).

Using docker compose ps will show if containers started properly. Logs can be viewed with docker compose logs.

Get a Module Repository

A Module Repository is a portable ZIP file that bundles a set of modules for extending the MicroEJ development environment. Please consult the Module Repository section for more information.

This training uses the MicroEJ Central Repository, which is the Module Repository used by MICROEJ SDK to fetch dependencies when starting an empty workspace. It bundles Foundation Library APIs and numerous Add-On Libraries.

Next step is to download a local copy of this repository:

  1. Visit the Central Repository on the MicroEJ Developer website.

  2. Navigate to the Production Setup section.

  3. Click on the offline repository link. This will download the Central Repository as a ZIP file.

Setup Artifactory

Configure Artifactory

For demonstration purposes we will allow anonymous users to deploy modules in the repositories:

  1. Once Artifactory container is started, go to http://localhost:8082/.

  2. Login to Artifactory for the first time using the default admin account (Username: admin, Password: password).

  3. Skip the installation wizard if it appears.

  4. Go to Administration > User Management > Settings.

  5. In the User Security Configuration section, check Allow Anonymous Access.

  6. Click on Save.

  7. Go to Administration > User Management > Permissions.

  8. Click on Anything entry (do not check the line), then go to Users tab

  9. Click on anonymous and check Deploy/Cache permission in the Selected Users Repositories category.

  10. Click on Save.

Next steps will involve uploading large files, so we have to increase the file upload maximum size accordingly:

  1. Go to Administration > Artifactory > General > Settings.

  2. In the General Settings section, change the value of File Upload In UI Max Size (MB) to 1024 then click on Save.

Create Repositories

We will now create and configure the repositories. Let’s start with the repository for the future built snapshot modules:

  1. Go to Administration > Repositories > Repositories in the left menu.

  2. Click on Add Repositories > Local Repository

  3. Select Maven.

  4. Set Repository Key field to custom-modules-snapshot and click on Create Local Repository.

Repeat the same steps for the other repositories with the Repository Key field set to custom-modules-release and microej-module-repository.

Import MicroEJ Repositories

In this section, we will import MicroEJ repositories into Artifactory repositories to make them available to the build server.

  1. Go to Administration > Artifactory > Import & Export > Repositories.

  2. Scroll to the Import Repository from Zip section.

  3. As Target Local Repository, select microej-module-repository in the list.

  4. Click on Select file and select the MicroEJ module repository zip file (central-repository-[version].zip) that you downloaded earlier (please refer to section Get a Module Repository).

  5. Click Upload. At the end of upload, click on Import. Upload and import may take some time.

Artifactory is now hosting all required MicroEJ modules. Go to Application > Artifactory > Artifacts and check that the repository microej-module-repository does contain modules as shown in the figure below.

../_images/tuto_microej_cli_artifactory_module_preview.PNG

Setup Gitea

Install Gitea

  1. Once the Gitea container is started, go to http://localhost:3000/.

  2. Don’t change anything on the Initial Configuration, click on Install Gitea

  3. Click on Register account and create one. The first created user become the administrator.

Configure Gitea

  1. At the top right click on the arrow then New Repository

  2. As Repository Name set helloworld, leave the other options as default.

  3. Click Create Repository.

Setup Jenkins

Install Jenkins

  1. Once Jenkins container is started, go to http://localhost:8080/.

  2. To unlock Jenkins, copy/paste the generated password that has been written in the container log. Click on Continue.

  3. Select option Install suggested plugins and wait for plugins installation.

  4. Fill in the Create First Admin User form. Click Save and continue.

  5. Click on Save and finish, then on Start using Jenkins.

Configure Jenkins

  1. Go to Manage Jenkins > Plugins.

  2. Add Docker Pipeline plugin:
    1. Go to Available plugins section.

    2. Search Docker Pipeline.

    3. Install it and restart Jenkins

Build a new Module using Jenkins

Since your environment is now setup, it is time to build your first module from Jenkins and check it has been published to Artifactory. Let’s build an “Hello World” Sandboxed Application project.

Create a new MicroEJ Module

In this example, we will create a very simple module using the Sandbox Application buildtype (build-application) that we’ll push to a Git repository.

Note

For demonstration purposes, we’ll create a new project and share it on a local Git bare repository. You can adapt the following sections to use an existing MicroEJ project and your own Git repository.

  1. Start MICROEJ SDK.

  2. Go to File > New > Sandboxed Application Project.

  3. Fill in the template fields, set Project name to com.example.hello-world.

    ../_images/tuto_microej_cli_module_creation.PNG
  4. Click Finish. This will create the project files and structure.

  5. Right-click on source folder src/main/java and select New > Package. Set a name to the package and click Finish.

  6. Right-click on the new package and select New > Class. Set Main as name for the class and check public static void main(String[] args), then click Finish.

  7. Add the line System.out.println("Hello World!"); to the method and save it.

    ../_images/tuto_microej_cli_module_files.PNG
  8. Locate the project files
    1. In the Package Explorer view, right-click on the project then click on Properties.

    2. Select Resource menu.

    3. Click on the arrow button on line Location to show the project in the system explorer.

    ../_images/tuto_microej_cli_module_location.PNG

Note

For more details about Applications development, refer to the Application Developer Guide.

Upload to your Git repository

Note

We need the IP address of the Docker Bridge Network, here we consider that it’s 172.17.0.1 but you can check with the command ip addr show docker0 on the Docker host.

  1. Open the project directory, create a file named Jenkinsfile and copy this content inside:

    pipeline {
        agent {
            docker { 
                image 'microej/sdk:5.8.1' 
                args '-e ACCEPT_MICROEJ_SDK_EULA_V3_1B=YES'
            }
        }
        stages {
            stage('Publish') {
                steps {
                    sh 'mmm publish shared -r ivy/ivysettings-artifactory.xml'
                }
            }
        }
    }
    
  2. Create a directory named ivy, create a file named ivysettings-artifactory.xml and copy this content inside:

    <?xml version="1.0" encoding="UTF-8"?>
    <ivy-settings>
      
    	<property name="artifactory.repository.url" value="http://172.17.0.1:8082/artifactory" override="false"/>
    	<property name="local.repository.dir" value="${user.home}/.ivy2/repository/" override="false"/>
      
    	<!--
    		Map MMM resolvers (*.resolver) to custom resolver
    		
    		Kinds of repositories:
    		- release: used when publishing a released module.
    		- snapshot: used when publishing a snapshot module.
    		- local: used when publishing a snapshot module locally.
    	-->
    	<property name="release.resolver" value="modulesReleaseRepository" override="false"/>
    	<property name="shared.resolver" value="modulesSnapshotRepository" override="false"/>
    	<property name="local.resolver" value="localRepository" override="false"/>
     
    	<property name="modules.resolver" value="fetchAll" override="false" />
    	<property name="request.cache.dir" value="${user.home}/.ivy2/cache" override="false"/>
    	<property name="default.conflict.manager" value="latest-compatible" override="false"/>
      
    	<settings defaultResolver="${modules.resolver}" defaultConflictManager="${default.conflict.manager}" defaultResolveMode="dynamic"/>
    	<caches defaultCacheDir="${request.cache.dir}"/>
      
      	<resolvers>
    	 
    		<url name="modulesReleaseRepository" m2compatible="true">
    			<artifact pattern="${artifactory.repository.url}/custom-modules-release/[organization]/[module]/[branch]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
    			<ivy pattern="${artifactory.repository.url}/custom-modules-release/[organization]/[module]/[branch]/[revision]/ivy-[revision].xml" />
    		</url>
    		
    		<url name="modulesSnapshotRepository" m2compatible="true" checkmodified="true">
    			<artifact pattern="${artifactory.repository.url}/custom-modules-snapshot/[organization]/[module]/[branch]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
    			<ivy pattern="${artifactory.repository.url}/custom-modules-snapshot/[organization]/[module]/[branch]/[revision]/ivy-[revision].xml" />
    		</url>
    		
    		<url name="microejModulesRepository" m2compatible="true">
    			<artifact pattern="${artifactory.repository.url}/microej-module-repository/[organization]/[module]/[branch]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
    			<ivy pattern="${artifactory.repository.url}/microej-module-repository/[organization]/[module]/[branch]/[revision]/ivy-[revision].xml" />
    		</url>
    		
    		<filesystem name="localRepository" m2compatible="true" checkmodified="true">
    			<artifact pattern="${local.repository.dir}/[organization]/[module]/[branch]/[revision]/[artifact]-[revision](-[classifier]).[ext]" />
    			<ivy pattern="${local.repository.dir}/[organization]/[module]/[branch]/[revision]/ivy-[revision].xml" />
    		</filesystem>
    		
    		<chain name="fetchRelease">
    			<resolver ref="modulesReleaseRepository"/>
    			<resolver ref="microejModulesRepository"/>
    		</chain>
    		
    		<chain name="fetchSnapshot">
    			<resolver ref="modulesSnapshotRepository"/>
    			<resolver ref="fetchRelease"/>
    		</chain>
    
    		<chain name="fetchLocal">
    			<resolver ref="localRepository"/> 
    			<resolver ref="fetchSnapshot"/> 
    		</chain>
    
    		<chain name="fetchAll">
    			<resolver ref="fetchLocal"/>
    		</chain>
    	
    	</resolvers>
    </ivy-settings>
    

This file configures the MicroEJ Module Manager to import and publish modules from the Artifactory repositories described in this training. Please refer to the Settings File section for more details.

Note

At this point, the content of the directory com.example.hello-world should look like the following:

com.example.hello-world
├── bin
│   └── ...
├── ivy
│   └── ivysettings-artifactory.xml
├── src
│   └── ...
├── src-adpgenerated/
│   └── ...
├── CHANGELOG.md
├── Jenkinsfile
├── LICENSE.txt
├── README.md
└── module.ivy
  1. Open a terminal from the directory com.example.hello-world and type the following commands:

    git init
    git checkout -b main
    git add *
    git commit -m "Add Hello World application"
    git remote add origin http://localhost:3000/<admin_user>/helloworld.git
    git push -u origin main
    

Create a New Jenkins Job

Start by creating a new job for building our application.

  1. Go to Jenkins dashboard.

  2. Click on New Item.

  3. Set item name to Hello World.

  4. Select Multibranch Pipeline.

  5. Validate with Ok button.

  6. In General tab set Display Name to Hello World

  7. In Branch Sources, click on Add Source > Git.

  8. Add Project Repository http://172.17.0.1:3000/<admin_user>/helloworld.git

    ../_images/tuto_microej_cli_jenkins_git_hello.PNG
  9. Click on Save.

Build the “Hello World” Application

Let’s run the job!

In Jenkins Hello World dashboard, click on main branch, then click on Build Now.

Note

You can check the build progress by clicking on the build progress bar and showing the Console Output.

At the end of the build, the module is published to http://localhost:8082/artifactory/list/custom-modules-snapshot/com/example/hello-world/.

Congratulations!

At this point of the training:

  • Artifactory is hosting your module builds and MicroEJ modules.

  • Jenkins automates the build process using MicroEJ Module Manager.

The next recommended step is to adapt MMM/Jenkins/Artifactory configuration to your ecosystem and development flow.

Appendix

This section discusses some of the customization options.

Customize Jenkins

Jenkins jobs are highly configurable, following options and values are recommended by MicroEJ, but they can be customized at your convenience.

In General tab:

  1. Check Discard old builds and set Max # of builds to keep value to 15.

  2. Click on Advanced button, and check Block build when upstream project is building.

In Build triggers tab:

  1. Check Poll SCM, and set a CRON-like value (for example H/30 * * * * to poll SCM for changes every 30 minutes).

In Post-build actions tab:

  1. Add post-build action Publish JUnit test result report:

  2. Set Test report XMLs to **/target~/test/xml/**/test-report.xml, **/target~/test/xml/**/*Test.xml.

Note

The error message ‘**/target~/test/xml/**/test-report.xml’ doesn’t match anything: ‘**’ exists but not ‘**/target~/test/xml/**/test-report.xml’ will be displayed since no build has been executed yet. These folders will be generated during the build.

  1. Check Retain long standard output/error.

  2. Check Do not fail the build on empty test results

Customize target~ path

Some systems and toolchains don’t handle long path properly. A workaround for this issue is to move the build directory (that is, the target~ directory) closer to the root directory.

To change the target~ directory path, set the build option target.

In Advanced, expand Properties text field and set the target property to the path of your choice. For example:

target=C:/tmp/