Automation of Delivery for the Jenkins Pipeline Code

Pipeline as Code

https://www.jenkins.io/doc/book/pipeline-as-code/

Pipeline as Code describes a set of features that allow Jenkins users to define pipelined job processes with code, stored and versioned in a source repository. These features allow Jenkins to discover, manage, and run jobs for multiple source repositories and branches?—?eliminating the need for manual job creation and management.

To use Pipeline as Code, projects must contain a file named Jenkinsfile in the repository root, which contains a "Pipeline script."

Additionally, one of the enabling jobs needs to be configured in Jenkins:

Multibranch Pipeline: build multiple branches of a single repository automatically

Organization Folders: scan a GitHub Organization or Bitbucket Team to discover an organization’s repositories, automatically creating managed Multibranch Pipeline jobs for them

Pipeline: Regular Pipeline jobs have an option when specifying the pipeline to "Use SCM".

Jenkins 设计 Pipeline as Code 是面向具体的项目，

即项目的CI代码和项目的业务代码应该是一个整体，存储在同一个repo中。

对应的是在repo中，必须定义一个Jenkinsfile，在此文件中定义 Pipeline的逻辑。

PREJECT REPO = CI CODE + SERVICE CODE

另外对于两个扩展形式是：

Multibranch pipeline --- 包含多个分支，每个分支都是完整的 CI + SERVICE代码

PROJECT REPO[MULTIBRANCH] = BRANCH1 + BRANCH2

　　　　BRANCH1 = CI CODE + SERVICE CODE

　　　　BRANCH2 = CI CODE + SERVICE CODE

Orgnization pipeline --- 包含多个组织, 每个组织有多个项目，每个项目有多个分支

　　ORGANIZATION REPO = ORGANIAZATION1 + ORGANIZATION2

　　　　　　ORGANIZATION1 = PROJECT REPO[MULTIBRANCH]1 + PROJECT REPO[MULTIBRANCH]2

Example Repository Structure

+--- GitHub Organization
    +--- Project 1
        +--- master
        +--- feature-branch-a
        +--- feature-branch-b
    +--- Project 2
        +--- master
        +--- pull-request-1
        +--- etc...

Single source of truth

https://en.wikipedia.org/wiki/Single_source_of_truth

Jenkins这种设计是符合真理的单一来源规则, ENTITY === MODEL + DATA。

MODEL 对应CI， DATA对应业务代码。

In information systems design and theory, single source of truth (SSOT) is the practice of structuring information models and associated data schema such that every data element is mastered (or edited) in only one place.

Jenkinsfile使用

https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#_footnotedef_2

问题提出

不管是哪一种类型的 Pipeline（纯Pipeline，多分支Pipeline，组织级Pipeline）

都需要维护人员手动建立对应的CI实例，有以下缺点

消耗人力，
不利于迁移到新的Jenkins服务器上（例如做负载均衡或者是容灾热备份）

另外此中结构下，CI代码和业务代码是存在一个库中， CI入口文件只对应一个文件 Jenkinsfile，有一个缺点

对于非业务源码绑定的CI场景不适用，例如CI需求多样，非对应单个Project REPO。
一个REPO一个分支，只对应一个Jenkinsfile，即不能建立多个Jenkins Jobs，对单REPO多Jobs需求也不满足。

对于更高级的CI系统，我们期望：

CI REPO 可以只负责CI代码。
支持多个Pipeline Jobs。
Pipeline Jobs建立的工作，也使用代码管理。（Pipeline as Code使得Jobs可以通过代码管理，这里更近一步）
支持在任意Jenkins Server上自动化Jobs的建立。（这就是CI Jobs持续发布功能。）

https://www.jenkins.io/doc/book/pipeline/getting-started/#through-the-classic-ui

Pipeline SCM配置

https://devopscube.com/jenkins-pipeline-as-code/

Multibranch 配置

https://devopscube.com/jenkins-multibranch-pipeline-tutorial/

解决方案

http://hiristic.se/

此组织提供了完美的方案。

Automate your delivery pipeline

We focus on delivering solutions and supporting organization to more efficiently produce and deliver code. We provide complete support in choosing the right platform and technologies all the way from development till deployment.

Jenkins REPO

此库负责 Jenkins服务器工作： docker镜像构建、配置管理、Jobs建立。

https://github.com/Hiristic/JDasC

Collection of Init scripts, configurations and executable Jenkinsfiles loaded automatically into a dockerized Jenkins.

How it works

A Docker image is built according to the steps in Dockerfile. Docker compose will build and start the container with arguments located in docker-compose.yaml file.

Steps are:

A docker image containing Jenkins is pulled from Docker hub.

All plugins are downladed and installed to the jenkins instance.

Jenkins2.sh will apply basic configuration and call jenkins.sh script to apply aditional configurations to Jenkins processes.

Docker compose will map in resources needed for jenkins to boot.

Jenkins Groovy init scripts will be executed and create initial jobs on the Jenkins server if they are not already created and mapped.

Jenkins configuration as code will read jenkins.yaml configuration file and apply all configuration to both Jenkins and all the plugins.

Jenkins process will finish its boot procedure and is ready to for use.

Jenkins Pipeline Global Groovy Library

此库专门负责 Jenkinsfiles管理，以及公共逻辑存放（为shared library使用），建立Jobs的引导Job。

https://github.com/Hiristic/Jenkins-global-lib

Collection of Groovy scripts, Jenkins global variables and classes, shell scripts, gdsl file for IDEs and configurations.

此处有很多参考代码。

https://github.com/Hiristic/Jenkins-global-lib/tree/master/jenkinsfiles/Demo

引导Job

https://github.com/Hiristic/Jenkins-global-lib/blob/master/jenkinsfiles/Dev/Generate_jobs_from_code_dev.groovy

import jenkins.model.Jenkins;
import static groovy.io.FileType.FILES
import java.io.File
import org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition
import org.jenkinsci.plugins.workflow.job.WorkflowJob

@NonCPS
def getAllJenkinsfiles() {
  def jenkinsfiles = []
    new File(WORKSPACE, "jenkinsfiles").traverse(type: FILES, nameFilter: ~/.*\.groovy/) {
        jenkinsfiles.add(it)
    }
    return jenkinsfiles
}

pipeline {
  agent any
  parameters {
    string(
      description: 'Select repository to generate jobs from',
      name: 'repo',
      defaultValue: 'https://github.com/Hiristic/Jenkins-global-lib.git',
      trim: false
    )
  }
  stages {
    stage('Generate jobs') {
      steps {
        script {
          echo "Job URL: "+currentBuild.absoluteUrl
          git params.repo
          
          def scripts = []
          scripts = getAllJenkinsfiles()
          if (scripts!=null) {
            // sort to run them in an alphabetic order
            scripts.sort().each {
              createJob(it)
            }
          }
        }
      }
    }
  }
}

def createJob(File jenkinsfile){
  jobName = jenkinsfile.name.minus(".groovy")
  WorkflowJob project = Jenkins.instance.getItem(jobName)
  if (project == null) {
    project = Jenkins.instance.createProject(WorkflowJob.class, jobName)
    echo "Generated new job: "+jobName
  }
  project.setDefinition(new CpsFlowDefinition(jenkinsfile.text, false))
  echo "Applied config to job: "+jobName
}

相关知识点

Jenkins启动后Hook

https://www.jenkins.io/doc/book/managing/groovy-hook-scripts/

Post initialization script (init hook)

You can create a Groovy script file $JENKINS_HOME/init.groovy, or any .groovy file in the directory $JENKINS_HOME/init.groovy.d/, to run some additional things right after Jenkins starts up. The groovy scripts are executed at the end of Jenkins initialization. This script can access classes in Jenkins and all the plugins. So for example, you can write something like:
import jenkins.model.Jenkins;

// start in the state that doesn't do any build.
Jenkins.instance.doQuietDown();

启动后创建Job

https://coderedirect.com/questions/521912/initializing-jenkins-2-0-with-pipeline-in-init-groovy-d-script

import jenkins.model.Jenkins
import org.jenkinsci.plugins.workflow.job.WorkflowJob

WorkflowJob job = Jenkins.instance.createProject(WorkflowJob, 'my-pipeline')

https://github.com/Hiristic/JDasC/tree/238a15993d8861e86f0421f8a12535f0180098ae/init_scripts/src/main/groovy

WorkflowJob project = Jenkins.instance.createProject(WorkflowJob.class, generator_job)
project.setDescription("This job will generate jobs for each jenkinsfile in repository")
File job_generator = new File(Jenkins.instance.rootDir, "init.groovy.d/scripts/config/Generate_jobs_from_repo.groovy")
project.setDefinition(new CpsFlowDefinition(job_generator.text, false))

https://stackoverflow.com/questions/16963309/how-create-and-configure-a-new-jenkins-job-using-groovy

def jobDSL="""
node {
  stage("test"){
   echo 'Hello World'
  }
}

""";
//http://javadoc.jenkins.io/plugin/workflow-cps/index.html?org/jenkinsci/plugins/workflow/cps/CpsFlowDefinition.html
def flowDefinition = new org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition(jobDSL, true);
//http://javadoc.jenkins.io/jenkins/model/Jenkins.html
def parent = Jenkins.instance;
//parent=Jenkins.instance.getItemByFullName("parentFolder/subFolder")
//http://javadoc.jenkins.io/plugin/workflow-job/org/jenkinsci/plugins/workflow/job/WorkflowJob.html
def job = new org.jenkinsci.plugins.workflow.job.WorkflowJob(parent, "testJob")
job.definition = flowDefinition

job.setConcurrentBuild(false);

//http://javadoc.jenkins.io/plugin/branch-api/jenkins/branch/RateLimitBranchProperty.html
job.addProperty( new jenkins.branch.RateLimitBranchProperty.JobPropertyImpl
    (new jenkins.branch.RateLimitBranchProperty.Throttle (60,"hours")));
def spec = "H 0 1 * *";
hudson.triggers.TimerTrigger newCron = new hudson.triggers.TimerTrigger(spec);
newCron.start(job, true);
job.addTrigger(newCron);
job.save();


Jenkins.instance.reload()

Jenkins API

https://javadoc.jenkins-ci.org/jenkins/model/jenkins.html

例如创建Job的接口

T createProject(Class type, String name) Creates a new job.

TopLevelItem createProject(TopLevelItemDescriptor type, String name)

TopLevelItem createProject(TopLevelItemDescriptor type, String name, boolean notify) Creates a new job.

Jenkins Plugins

Groovy

https://plugins.jenkins.io/workflow-cps/

Pipeline

https://plugins.jenkins.io/workflow-aggregator/

DevOps Jenkins

Automation of Delivery for the Jenkins Pipeline Code

Pipeline as Code

Single source of truth

Jenkinsfile使用

问题提出

Pipeline SCM配置

Multibranch 配置

解决方案

Automate your delivery pipeline

Jenkins REPO

How it works

Jenkins Pipeline Global Groovy Library

相关知识点

Jenkins启动后Hook

Post initialization script (init hook)

Jenkins API

Jenkins Plugins

Groovy

Pipeline

相关

一步一步配置docker（tomcat+jenkins+phpmyadmin+nginx）

jenkins的任务卡住

jenkins

Jenkins performance插件生成性能测试报告【待完成】

在 k8s 中的 jenkins 集成 sonarqube 实现代码质量检查

jmeter+jenkins+ant自动化集成

Jenkins 中定时任务构建

Windows 搭建JenKins自动化测试集成环境

敏捷开发专家一席谈：云原生技术下的华为云DevOps实践之路

jenkins&gitlab自动发布

【Jenkins基础教程】Jenkins2.303.3的基本使用

Jenkins配置基于角色的项目权限管理

标签

`T`	`createProject(Class type, String name)` Creates a new job.
`TopLevelItem`	`createProject(TopLevelItemDescriptor type, String name)`
`TopLevelItem`	`createProject(TopLevelItemDescriptor type, String name, boolean notify)` Creates a new job.