How can Jenkins jobs be automated

Jenkins tutorial: programming a seed job

In part one of this three-part series of articles I showed you how to install and configure Jenkins - and how to create build and release jobs for Maven projects via the Jenkins user interface. In part two we created the same jobs programmatically with the Jenkins Job DSL and made the job list clearer with views.

In this final, third part, I will explain to you how you can extract duplicated Groovy code in utility classes and how you can create fully automatic Jenkins jobs for new Java projects in Git-Monorepo, or all of them in the case of changes in the job DSL code Customize Jenkins jobs at the same time.

You can find the code created in the complete series of articles in my GitLab repository https://gitlab.com/SvenWoltmann/jenkins-tutorial-demo.

Jenkins security system: Script Security

Before we move on to automation, it is important to familiarize yourself with Jenkins' scripted security system. So far we have been able to run all scripts without any problems because we created them as an administrator. Jenkins maintains a whitelist of permitted scripts, to which all scripts that an admin creates are automatically added. These scripts can later also be executed by regular users.

However, this only applies to scripts that an admin copies directly into a job. In the course of further automation, we want to configure Jenkins so that it loads scripts from the Git repository. Downloaded scripts cannot be executed by regular users. This also includes the SYSTEM-User, with whose rights a job is executed by default - even if an admin starts it. Jenkins' security system protects us from potentially dangerous Groovy code that could be introduced by downloading it from the Git repository.

How can we make downloaded scripts executable?

There are several ways to make downloaded Groovy scripts executable:

  • Scripts that cannot be executed automatically end up in a validation queue. An admin can do this under "Manage Jenkins" → "In-process Script Approval" release manually. However, he would have to do this again every time he changes the Groovy code. Depending on the composition of the team, this could prove to be impractical.
  • Configure the security system so that the seed job is always executed as admin (by default it is executed by the SYSTEM user). This is unsafe as it could introduce potentially dangerous code. In addition, with this variant it is not possible to store duplicated code in utility classes.
  • Deactivate Script Security completely ("Manage Jenkins""Configure Global Security" Checkbox "Enable script security for Job DSL scripts" deactivate). This is even more unsafe as it could allow any user to run potentially dangerous scripts in any job. In the previous option, this risk was at least reduced to Groovy files downloaded by the seed job.
  • Run the code in the Groovy Sandbox. In principle, the code can be executed in the sandbox, but it is limited to a fixed set of operations. This option only works together with the execution as an admin user, otherwise the scripts would have to be activated again via the validation queue. In this case, the sandbox prevents the execution of malicious code. This is the only option that allows code to be outsourced in utility classes.

In my opinion, the "Groovy Sandbox" in combination with running the job as an admin user is the most sensible option. In the following I will show you how to configure the security system accordingly.

Global configuration of Jenkins script security

First we have to activate the option to configure the execution rights for individual jobs. To do this, we open the global security settings via "Manage Jenkins" → "Configure Global Security". There we have to add the "Per-project configurable Build Authorization" under "Access Control for Builds":

The "Run as User who Triggered Build" strategy should already be activated in the checkboxes that now open. Here we also have to allow the option "Run as a specific user" in order to later be able to configure a job so that it is always executed by an admin user.

We save the changes with the "Save" button. We will make all other settings later directly in the Jenkins job.

Jenkins script security system error messages

In the following I list the error messages that you might get displayed by the security system. I also state the respective causes and possible solutions. You can skip this section for now. You can come back here if security-related error messages occur later in the tutorial.

ERROR: script not yet approved for use

Jenkins displays this error message when a job tries to run a script that was not whitelisted by an admin (either explicitly or in which the admin manually copied the script into a job).

Possible solutions:

  • Jenkins automatically places the script in the validation queue. An admin can do it under "Manage Jenkins""In-process Script Approval" release manually.
  • Run the job as an admin user: Job page → "Authorization""Configure Build Authorization" activate → "Authorize Strategy" on "Run as Specific User" or "Run as User who Triggered Build" and set an admin user accordingly or start the job as an admin user.

ERROR: startup failed: [...] unable to resolve class [...]

You get this error message if you try to access another Groovy class on the file system from a Groovy script that is not running in the Groovy sandbox.

Solution:

  • The only way to access other classes is to run the code in the Groovy sandbox: Job Configuration → "Process Job DSLs""Use Groovy Sandbox" activate.

ERROR: You must configure the DSL job to run as a specific user in order to use the Groovy sandbox

This error message appears if you have activated the Groovy Sandbox for a job, but have not configured the security system in such a way that the job is executed as a specific user. By default, a job is always carried out by the "SYSTEM" user.

Solution:

  • Execute the job as a specific user: Job page → "Authorization""Configure Build Authorization" activate → "Authorize Strategy" on "Run as Specific User" or "Run as User who Triggered Build" set and deposit a user accordingly. The selected user should again have admin rights, otherwise you will get the error message "script not yet approved for use" as long as the script has not been explicitly activated by an admin.

ERROR: Scripts not permitted to use method [...] / staticMethod [...]

This error message tells us that calling the corresponding method is not permitted in the Groovy sandbox.

Solution:

  • Jenkins automatically places the method signature in the validation queue. An admin has to take them under "Manage Jenkins""In-process Script Approval" release manually.

ERROR: […] No signature of method: […] is applicable for argument types: […]

This error occurs when trying to configure the sparse checkout within the Groovy sandbox using the method shown above.

Solution:

  • Instead of using the method, the extensions (as shown above) must be configured using the.

Automation of job creation

In the previous article we wrote Jenkins jobs as code. This is in the files,,, in the directory of our monorepo. (If you don't work with a monorepo, you can also use a separate repository for the Jenkins jobs.)

To import the jobs into Jenkins, we manually created creator jobs, added a "Process Job DSLs" build step to each, copied the Groovy code into it and executed the job. If we want to change something in a job, we have to copy the code again into the respective creator job. This is still feasible with four jobs - with a few hundred it is no longer practicable. The previous method also has the disadvantage that we had to duplicate the code we needed several times instead of being able to outsource it to helper classes.

Load the Jenkins Job DSL code from the project repository

First of all, we don't want to manually copy the code into the creator jobs, but rather let so-called loader jobs load the code from the Git repository. It's pretty easy to set up.

Using the example of the "Jenkins Tutorial Demo - Library 1" job, I will show you how to do this. The procedure can be adopted one-to-one for the other three jobs.

We create a new job on the Jenkins start page via "New Item". We assign "Jenkins Tutorial Demo - Library 1 (DSL) Loader" as the name, and select "Freestyle project" as the type. Under "Build" we add a "Process Job DSLs" step. So far the process is identical to the creation of the Creator jobs (therefore I have not used screenshots until now).

Instead of selecting "Use the provided DSL script", we leave it with the standard selection "Look on Filesystem". In order to make the job file available in the file system, we first have to check it out from GitLab. So we scroll up again to "Source Code Management", select "Git" and enter the repository URL "[email protected]: SvenWoltmann / jenkins-tutorial-demo.git":

Under "Additional Behaviors" we add "Sparse Checkout Paths" and enter it there so that only this directory is checked out. (If you have the Jenkins jobs in a separate repository, you can skip this step.)

Now we scroll down to "Build" again and enter the path of the Groovy file,, under "DSL Scripts". We also activate the "Use Groovy Sandbox" option (as explained in the previous chapter):

We save the job with "Save". Before we can run it, we have to specify that the job is to be run as an admin user (also explained in the previous chapter). To do this, we click on "Authorization" on the job page. (If you do not see this point, please check whether you have carried out the steps in the section Global Configuration of Jenkins Script Security correctly.)

On the "Authorization" page we activate "Configure Build Authorization" and select either "Run as User who Triggered Build" or "Run as Specific User" as the strategy. The difference is as follows:

  • Run as User who Triggered Build: The job must ultimately be started by an admin user in order to be able to execute the downloaded Groovy code (unless this has already been whitelisted by an admin).
  • Run as Specific User: With this option you can specify a user with whose rights - regardless of the logged in user - the job will be carried out. If you enter an admin user here, every logged-in user can start the job.

Since I am the only user on my Jenkins server, I choose the option "Run as User who Triggered Build":

We save the change with "Save" and start the job with "Build Now". Despite all the security-relevant settings that we have made, the job initially aborts with an error message:

We cannot avoid this error message. It says that calling the specified method is not permitted within the Groovy sandbox. At the same time, Jenkins wrote the method signature in the validation queue, in which we release it as follows. We reach the approval form via "Manage Jenkins""In-process Script Approval":

We click on "Approve" and then see the method signature in the list of approved signatures.

We have to repeat the steps "Execute job" and "Approve method signature" three more times so that the list of approved signatures finally contains four entries:

We start the job a fifth time. Now it runs successfully through:

Even if we disregard the activation of the method signatures (we don't have to repeat this), it was still a fairly high manual effort. If we had to do that for the release jobs and the "application1" project, we wouldn't have gained much compared to the previous, completely manual solution. I will show you how to do it more easily in the next section.

Create multiple Jenkins jobs with a single seed job

In order to generate the three remaining jobs (ie the build job for the "application1" project and the two release jobs) and the view, we do not need to create any additional loader jobs. Instead, we can simply add the four other Groovy files to the previously created Loader job under "DSL Scripts":

Since the loader job no longer only creates the build job for the "library1" project, we can rename it from "Jenkins Tutorial Demo - Library 1 (DSL) Loader" to "Jenkins Tutorial Demo - Seed Job". We run it again and see the following status message:

Four Jenkins jobs and one view were successfully created with a single job. It was really fast now!

What actually happens to jobs that already exist? If nothing has changed in the Groovy code, the job also remains unchanged. If changes are made in the Groovy code, the job is adapted. The workspace and the build history of the job are retained.

Extract shared code

In the last interim conclusion, I criticized the fact that we had duplicated some of the code (e.g. the code block for the sparse checkout). When we manually imported the code into Jenkins in the previous article, we had no way of outsourcing common functionality. Now that the code is in the filesystem, we can very well extract repetitive code by creating utility classes that will be used by all jobs.

We create three classes in the subdirectory, and, and extract a large part of the duplicated code there. We leave the previous Groovy files unchanged and also create the following four job files and one view file:

jenkins-jobs / src / jobs / BuildJobLibrary1WithUtils.groovy jenkins-jobs / src / jobs / BuildJobApplication1WithUtils.groovy jenkins-jobs / src / jobs / ReleaseJobLibrary1WithUtils.groovy jgroithy-jobs / jobs / srobc / jobs / Release jobs / ViewDSLJobsWithUtils.groovy

You can find the new job / view files as well as the utility classes in the GitLab repository. Listing them here would inflate the article unnecessarily. I have given the complete paths above so that we can copy this list directly into the seed job and start it. The job runs successfully and has created four new jobs and a new view:

We can now create build and release jobs for a new Java project fairly quickly and import them into Jenkins via the seed job. We can now adjust a large part of the functionality centrally in the utility classes. After a change we only have to start the seed job.

Extension of the seed job

When we add a new project, however, we still have to a) copy and adapt two Groovy files, b) enter them into the seed job and c) start it.

Points b) and c) can be changed quite easily: instead of listing concrete Groovy files in the seed job, we can also use wildcards; and in order not to have to start the seed job manually, we can define a time-based trigger, for example.

First we replace the list of specific file names with the following three wildcard entries:

jenkins-jobs / src / jobs / BuildJob *.groovy jenkins-jobs / src / jobs / ReleaseJob *.groovy jenkins-jobs / src / jobs / View *.groovy

We could also specify here, but the example file is also in the directory and I would like to keep a certain amount of control over the files to be executed.

To run the seed job automatically, let's add a trigger that checks the Git repository for changes every 15 minutes:

In order for the trigger to work, we have to change the "Authorize Strategy" for the seed job. This is currently set to "Run as User who Triggered Build". The build trigger executes the job as the "SYSTEM" user. This user does not have the authorization to execute code in the Groovy sandbox. Therefore we have to change the strategy to "Run as Specific User" and enter a user with admin rights. For the sake of simplicity, I store myself here, but you can also create a separate user with admin rights for this.

The only thing we have to do now to create a new job is to create the corresponding Groovy files in the Git repository. Everything else then happens automatically. We can also automate this last manual step. I will show you how to do this in the next - and at the same time last - chapter of this series of articles.

Fully automatic generation of build and release jobs for new projects

Nowhere is it written that Groovy files have to be mapped one-to-one to Jenkins jobs. Ultimately, a Groovy file contains an executable program that calls the method once in the previous examples. We can extend such a program - like any other - with loops and branches and thus implement any complex logic.

In the following we create two generic job generators: one for build jobs and one for release jobs. The generators search for files in the Monorepo and create a Jenkins job for each (unless it is a module of a multi-module project). This possibility is a great advantage of the monorepo. If we wanted to implement this logic for projects that are distributed over several repositories, we would have to read and check out the existing repositories via an API of the repository server.

The generic build job in the file:

The code first calls the function and passes the file name of the currently executed script, which is provided by Jenkins in the variable. The method returns a list of the directories of all Maven projects in the repository. This list is iterated over and a Jenkins job is created for each project. The code inside the loop is the same as the code we created in the Extracting Shared Code section (and), with the only difference that the job name and description are generated from the project path.

The generic release job looks like this. Again, the code within the loop corresponds to the code previously created in and.

The class with the methods and, like the other utility classes, can be found in the GitLab repository. The methods are documented so that how they work should be easy to understand.

In addition, we create a file that creates a view for all generically created jobs:

Before you can call up the seed job, you have to deactivate the "Sparse Checkout" again so that the method can find the and directories. (If you have the Jenkins jobs in a separate repository, you must instead also check out the repository - or the repositories - with the Java code).

If you run the seed job, you will initially get error messages again because the code uses methods that are not yet activated for the Groovy sandbox. You have to go through these again - one after the other "Manage Jenkins" → "In-process Script Approval" release. When you're done, the following method signatures should be on the whitelist:

The seed job should now have created the "Part III - Generic" view with the following four jobs:

Our Groovy code has now automatically created build and release jobs for all Java projects in the Monorepo. In the end, try the following: create a new, simple Maven project in the monorepo. For example, copies the "library1" project into a "library2" project. Thereupon, build and release jobs should automatically appear in Jenkins for this.

Conclusion

In this series of articles I have shown you some of the most important functions of Jenkins with build and release jobs for Maven projects, as well as with the Jenkins Job DSL and the so-called "Seed Jobs". There are countless expansion options, here are a few examples:

  • You can have the seed job automatically create build jobs for feature branches.
  • You can build your release jobs Docker containers and have them pushed into a Docker registry (with the Fabric8 Maven plugin).
  • You can deploy web applications via blue-green deployment.
  • You can deploy microservices in a Kubernetes cluster (also with the Fabric8 Maven plugin).

Ultimately, you can implement any complex build and deploy logic using shell scripts in the various build steps and an extensive selection of plugins.

Do you have problems following the individual steps? Are you stuck somewhere? Then write a comment under the article; I'll do my best to help you.

I would also be very happy if you shared the article using one of the following share buttons.