背景

https://www.cnblogs.com/lightsong/p/15782809.html

书接前章，前面研究了使用K8S部署Jenkins master服务器的方法，

当时希望slave能力也能够利用K8S集群的来动态调度。

如下图：

将slave资源，以POD方式封装，动态调度到K8S集群中。

kubernetes plugin for Jenkins 就是为了满足这个需求。

Kubernetes plugin for Jenkins

https://plugins.jenkins.io/kubernetes/

利用k8s集群的海量计算处理能力，以动态化agent为手段， 增加jenkins的agent处理能力。

具有灵活的架构，资源用完释放，自动实现JOB的复杂均衡能力。

Jenkins plugin to run dynamic agents in a Kubernetes cluster.

Based on the Scaling Docker with Kubernetes article, automates the scaling of Jenkins agents running in Kubernetes.

The plugin creates a Kubernetes Pod for each agent started, and stops it after each build.

Agents are launched as inbound agents, so it is expected that the container connects automatically to the Jenkins controller. For that some environment variables are automatically injected:

• JENKINS_URL : Jenkins web interface url
• JENKINS_SECRET : the secret key for authentication
• JENKINS_AGENT_NAME : the name of the Jenkins agent
• JENKINS_NAME : the name of the Jenkins agent (Deprecated. Only here for backwards compatibility)

Tested with jenkins/inbound-agent, see the Docker image source code.

It is not required to run the Jenkins controller inside Kubernetes.

How to Setup Jenkins Build Agents on Kubernetes Pods

https://devopscube.com/jenkins-build-agents-kubernetes/

原理

Jenkins使用HTTP请求，跟API SERVER发起POD请求；

K8S 调度器选择 NODE ，将POD调度到其上；

POD在其上运行， 启动Jenkins slave容器；

Jenkins slave容器通过 JNLP 将运行过程日志返回给 Jenkins master；

Job运行完毕， Jenkins slave POD销毁， 释放资源。

How Does Jenkins Kubernetes Pod Agents Work?

Before getting into the implementation, let’s understand how this setup works.

 

Whenever you trigger a Jenkins job, the Jenkins Kubernetes plugin will make an API call to create a Kubernetes agent pod. Then, the Jenkins agent pod gets deployed in the kubernetes with few environment variables containing the Jenkins server details and secrets.

When the agent pod comes up, it used the details in its environment variables and talks back to Jenkins using the JNLP method. The following images show the environment variables of the agent pod.

All the build steps from the Jenkinsfile run on that pod. Once the build is complete, the pod will get terminated automatically. There are also options to retain the build pod.

The Jenkins Kubernetes plugin takes care of all the communication from Jenkins to the Kubernetes cluster.

Also, as long as your Kubernetes cluster scales, you can scale your Jenkins build agents without any issues.

Setting Up Jenkins Build Pods On Kubernetes

Setting Up Kubernetes Namespace & Service Account

service account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins
  namespace: default
  labels:
    "app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-role-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: default

Jenkins Master Setup in Kubernetes

jenkins deployment

# Persistent Volume Claim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

# Deployment Config
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccountName: jenkins-admin
      securityContext:
            fsGroup: 1000 
            runAsUser: 1000
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home         
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pv-claim

# Service Config
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector: 
    app: jenkins
  type: NodePort  
  ports:
    - name: httpport
      port: 8080
      targetPort: 8080
      nodePort: 32000
    - name: jnlpport
      port: 50000
      targetPort: 50000
  

Jenkins Kubernetes Plugin Configuration

Jenkinsfile With Pod Template

podTemplate(containers: [
    containerTemplate(
        name: 'maven', 
        image: 'maven:3.8.1-jdk-8', 
        command: 'sleep', 
        args: '30d'
        ),
    containerTemplate(
        name: 'python', 
        image: 'python:latest', 
        command: 'sleep', 
        args: '30d')
  ]) {

    node(POD_LABEL) {
        stage('Get a Maven project') {
            git 'https://github.com/spring-projects/spring-petclinic.git'
            container('maven') {
                stage('Build a Maven project') {
                    sh '''
                    echo "maven build"
                    '''
                }
            }
        }

        stage('Get a Python Project') {
            git url: 'https://github.com/hashicorp/terraform.git', branch: 'main'
            container('python') {
                stage('Build a Go project') {
                    sh '''
                    echo "Go Build"
                    '''
                }
            }
        }

    }
}

Using Shared Persistent Volumes With Jenkins Docker Agent Pods

podTemplate(containers: [
  containerTemplate(
      name: 'maven', 
      image: 'maven:latest', 
      command: 'sleep', 
      args: '99d'
      )
  ], 
  
  volumes: [
  persistentVolumeClaim(
      mountPath: '/root/.m2/repository', 
      claimName: 'maven-repo-storage', 
      readOnly: false
      )
  ]) 

{
  node(POD_LABEL) {
    stage('Build Petclinic Java App') {
      git url: 'https://github.com/spring-projects/spring-petclinic.git', branch: 'main'
      container('maven') {
        sh 'mvn -B -ntp clean package -DskipTests'
      }
    }
  }
}