A Review of Jenkins Groovy Pipelines

Jenkins implemented a form of job that can be written in Groovy a couple of years ago. They call them Groovy Pipelines, and they have two major types:

• Declarative pipelines

  • Have a strongly opinionated format for syntax and semantics in writing jobs
  • Slightly modified example from a "build and deploy" job of mine:

  pipeline {
      agent { label 'deployer' }
  
      stages{
          stage('prep-environment'){
              steps {
                  // Ensure network mapping exists
                  bat 'net use Y: /delete /y & exit 0'
                  bat 'net use X: /delete /y & exit 0'
                  bat 'net use Y: "\\\\sv-build-file\\Published Repositories" "example@password" /USER:justin.local\\jenkins.image.samba & exit 0'
                  bat 'net use X: "\\\\sv-build-file\\3rd Party" "example@password" /USER:justin.local\\jenkins.image.samba & exit 0'
  
                  // Clear original base jar properties file
                  bat 'del /F /Q "%TC_SLAVE_HOME%\\workspace\\publish\\jenkins.properties" & exit 0'
  
                  script{ mh.cloneRepo(PROJECT) }
              }
          }
          stage('extract-plugin') {
              steps {
                  script{ bh.unzipPlugin("Windows", BUILD_TAG, PLUGIN_ZIP, WORKSPACE, TC_SLAVE_HOME, PARSE_PLUGIN_FROM_TAG) }
              }
          }
          stage('publish-jar') {
              steps {
                  bat 'ant publish-jar'
              }
          }
      }
  }
  

• Scripted pipelines

  • Have a weakly opinionated in syntax and semantics in writing jobs
  • Slightly modified example from a job to trigger an Ansible playbook:

  node('master'){
      sh 'cd /opt/ansible/playbooks; hg pull -u;' + 
         'ansible-playbook --extra-vars "DATASET=' + DATASET + ' ci=' + CI_RUN + '"' + ' -i hosts process-test-logs.yml'
  }
  

The major difference between the two is that scripted pipelines have no predefined syntax that you must follow, whereas declarative have predefined pipeline, stage, and step blocks. This makes the declarative pipelines sometimes easier to read, but leaves most of the power user functionality to the much more relaxed and capable scripted pipelines. I recently upgraded all of the jobs running my performance tests at work to Groovy pipelines, both declarative and scripted depending on which I felt suited the purpose better. Here's some warnings, hurdles, and tripwires to watch out for if you do so yourself.

If you have a bunch of conditional logic, avoid declarative pipelines

In scripted pipelines, you have at your disposal all of the normal tricks in the programmer's toolbox:

• Variables (including Groovy built-ins like the regex Pattern type)
• Conditionals (if, switch, etc)
• Loops (for, while, etc)
• Classes
• Method / function calls

This makes working with complex scripted pipelines very simple. Say you need to calculate the number of nodes to deploy to per machine based on the total number of nodes; easy. You can calculate that with a quick loop and some modulos before you start deploying. Say you want to sanitize and verify your inputs before handing them off to various shell scripts and build actions; easy. Just throw some if statements and checks at the top of the script before you pass off your inputs. Declarative pipelines, on the other hand, have some troubles with these types of usecases. You see, you cannot use variables, conditionals, loops, or really any of the above natively in declarative pipelines. If you want to, you need to wrap your statements in script blocks, which reduces readability and pushes your code even further to the right. Declarative pipelines already have enough problems with code indentation insanity (just like JavaScript) due to the requirement of steps being in stages being in pipelines. So if you have complex conditions or operations you need to perform in your pipeline, definitely pick the scripted pipeline over declarative.

Watch out for CPS and your unserializable types

Jenkins pipelines restrict all variables to the Serializable type. This is so that pipelines can survive unexpected reboots and failures, such that if your Jenkins master randomly crashes it knows how and where to start after coming back online. Jenkins’ version of Groovy, however, is missing serializable implementations of quite a few common types. You'll have troubles working with JSON, configuration files, properties files, and much more programmatically in Groovy, because they don't have serializable versions of those yet. However, you can get around this in a couple of ways:

• Nullify unserializable variables before they would leave scope
  • Jenkins' Groovy will only notice that you have unserializable variables if they would leave scope and need to be stored, so set them to null first!
• Wrap your logic in a function and decorate it with the @NonCPS annotation

So, if you ever see a "class not serializable" error coming from your build scripts, you should know how to handle the problem.

Options tend to be easier to handle in declarative pipelines

When you swap over to Jenkins Groovy pipelines, you're going to lose a lot of the options you used to have as checkboxes in the regular job config. The most annoying ones that I miss in my scripted pipelines are:

• Timestamps
  • Right now you would need to concat a date to the front of all your output manually in a scripted pipeline
• Retries
  • You would also need to define your own logic to catch exceptions and retry after build failures
• Timeouts
  • Again, this will be all on you to figure out programmatically in Groovy

However, this is one of the few cases where declarative pipelines have an advantage. Declarative pipelines have options specifier blocks at the top of each pipeline and stage. This makes it very easy to get timestamps beside all of your output, to specify retries after a failure, timeouts on when to give up, and quite a few other power user features.

In general I tend towards scripted pipelines

And even if scripted pipelines can be a headache for the above problems, I almost always try to start with scripted pipelines over declarative. Why? Because I can just do more with less headache in scripted pipelines. They don't force me into a highly opinionated syntax that creeps more and more to the right of the screen as I flesh out my logic. They do make it easy to set options, and in certain simple cases they are easier to read than scripted. But dollars to donuts, if I had to pick just one to use for the rest of my career, I'd go the scripted pipeline route.