巧用Prometheus来扩展kubernetes调度器
Overview
本文将深入讲解 如何扩展 Kubernetes scheduler 中各个扩展点如何使用，与扩展scheduler的原理，这些是作为扩展 scheduler 的所需的知识点。最后会完成一个实验，记录网络流量的调度器。
kubernetes调度配置
kubernetes集群中允许运行多个不同的 scheduler ，也可以为Pod指定不同的调度器进行调度。在一般的Kubernetes调度教程中并没有提到这点，这也就是说，对于亲和性，污点等策略实际上并没有完全的使用kubernetes调度功能，在之前的文章中提到的一些调度插件，如基于端口占用的调度 NodePorts 等策略一般情况下是没有使用到的，本章节就是对这部分内容进行讲解，这也是作为扩展调度器的一个基础。
Scheduler Configuration 卷. 扩展点：preFilter, filter, reserve, preBind, score.
VolumeRestrictions：检查安装在节点中的卷是否满足特定于卷提供程序的限制。扩展点：filter.
VolumeZone：检查请求的卷是否满足它们可能具有的任何区域要求。扩展点：filter.
InterPodAffinity： 实现Pod 间的亲和性与反亲和性的功能。扩展点：preFilter, filter, preScore, score.
PrioritySort：提供基于默认优先级的排序。扩展点：queueSort.
对于更多配置文件使用案例可以参考官方给出的文档
如何扩展kube-scheduler NewSchedulerCommand，并且实现自己的 plugins 的逻辑即可。
import (
scheduler "k8s.io/kubernetes/cmd/kube-scheduler/app"
)
func main() {
command := scheduler.NewSchedulerCommand(
scheduler.WithPlugin("example-plugin1", ExamplePlugin1),
scheduler.WithPlugin("example-plugin2", ExamplePlugin2))
if err := command.Execute(); err != nil {
fmt.Fprintf(os.Stderr, "%v\n", err)
os.Exit(1)
}
}
而 NewSchedulerCommand 允许注入 out of tree plugins，也就是注入外部的自定义 plugins，这种情况下就无需通过修改源码方式去定义一个调度器，而仅仅通过自行实现即可完成一个自定义调度器。
// WithPlugin 用于注入out of tree plugins 因此scheduler代码中没有其引用。
func WithPlugin(name string, factory runtime.PluginFactory) Option {
return func(registry runtime.Registry) error {
return registry.Register(name, factory)
}
}
插件实现
对于插件的实现仅仅需要实现对应的扩展点接口。下面通过内置插件进行分析
对于内置插件 NodeAffinity ,我们通过观察他的结构可以发现，实现插件就是实现对应的扩展点抽象 interface 即可。
定义插件结构体
其中 framework.FrameworkHandle 是提供了Kubernetes API与 scheduler 之间调用使用的，通过结构可以看出包含 lister，informer等等，这个参数也是必须要实现的。
type NodeAffinity struct {
handle framework.FrameworkHandle
}
实现对应的扩展点
func (pl *NodeAffinity) Score(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) {
nodeInfo, err := pl.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)
if err != nil {
return 0, framework.NewStatus(framework.Error, fmt.Sprintf("getting node %q from Snapshot: %v", nodeName, err))
}
node := nodeInfo.Node()
if node == nil {
return 0, framework.NewStatus(framework.Error, fmt.Sprintf("getting node %q from Snapshot: %v", nodeName, err))
}
affinity := pod.Spec.Affinity
var count int64
// A nil element of PreferredDuringSchedulingIgnoredDuringExecution matches no objects.
// An element of PreferredDuringSchedulingIgnoredDuringExecution that refers to an
// empty PreferredSchedulingTerm matches all objects.
if affinity != nil && affinity.NodeAffinity != nil && affinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution != nil {
// Match PreferredDuringSchedulingIgnoredDuringExecution term by term.
for i := range affinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution {
preferredSchedulingTerm := &affinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution[i]
if preferredSchedulingTerm.Weight == 0 {
continue
}
// TODO: Avoid computing it for all nodes if this becomes a performance problem.
nodeSelector, err := v1helper.NodeSelectorRequirementsAsSelector(preferredSchedulingTerm.Preference.MatchExpressions)
if err != nil {
return 0, framework.NewStatus(framework.Error, err.Error())
}
if nodeSelector.Matches(labels.Set(node.Labels)) {
count += int64(preferredSchedulingTerm.Weight)
}
}
}
return count, nil
}
最后在通过实现一个 New 函数来提供注册这个扩展的方法。通过这个 New 函数可以在 main.go 中将其作为 out of tree plugins 注入到 scheduler 中即可
// New initializes a new plugin and returns it.
func New(_ runtime.Object, h framework.FrameworkHandle) (framework.Plugin, error) {
return &NodeAffinity{handle: h}, nil
}
实验：基于网络流量的调度 promQL 与 client_golang 有所了解
实验大致分为以下几个步骤：
- 定义插件API
- 插件命名为
NetworkTraffic
- 定义扩展点
- 这里使用了 Score 扩展点，并且定义评分的算法
- 定义分数获取途径（从prometheus指标中拿到对应的数据）
- 定义对自定义调度器的参数传入
- 将项目部署到集群中（集群内部署与集群外部署）
- 实验的结果验证
实验将仿照内置插件 nodeaffinity 完成代码编写，为什么选择这个插件，只是因为这个插件相对比较简单，并且与我们实验目的基本相同，其实其他插件也是同样的效果。
整个实验的代码上传至 github.com/CylonChau/customScheduler
实验开始
错误处理
在初始化项目时，go mod tidy 等操作时，会遇到大量下面的错误
go: github.com/GoogleCloudPlatform/spark-on-k8s-operator@v0.0.0-20210307184338-1947244ce5f4 requires
k8s.io/apiextensions-apiserver@v0.0.0: reading k8s.io/apiextensions-apiserver/go.mod at revision v0.0.0: unknown revision v0.0.0
kubernetes issue #79384 framework.DecodeInto 函数可以做这个操作
func New(plArgs *runtime.Unknown, handle framework.FrameworkHandle) (framework.Plugin, error) {
args := Args{}
if err := framework.DecodeInto(plArgs, &args); err != nil {
return nil, err
}
...
}
另外一种方式是必须实现对应的深拷贝方法，例如 NodeLabel 中的
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// NodeLabelArgs holds arguments used to configure the NodeLabel plugin.
type NodeLabelArgs struct {
metav1.TypeMeta
// PresentLabels should be present for the node to be considered a fit for hosting the pod
PresentLabels []string
// AbsentLabels should be absent for the node to be considered a fit for hosting the pod
AbsentLabels []string
// Nodes that have labels in the list will get a higher score.
PresentLabelsPreference []string
// Nodes that don't have labels in the list will get a higher score.
AbsentLabelsPreference []string
}
最后将其注册到register中，整个行为与扩展APIServer是类似的
// addKnownTypes registers known types to the given scheme
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&KubeSchedulerConfiguration{},
&Policy{},
&InterPodAffinityArgs{},
&NodeLabelArgs{},
&NodeResourcesFitArgs{},
&PodTopologySpreadArgs{},
&RequestedToCapacityRatioArgs{},
&ServiceAffinityArgs{},
&VolumeBindingArgs{},
&NodeResourcesLeastAllocatedArgs{},
&NodeResourcesMostAllocatedArgs{},
)
scheme.AddKnownTypes(schema.GroupVersion{Group: "", Version: runtime.APIVersionInternal}, &Policy{})
return nil
}
Notes：对于生成深拷贝函数及其他文件，可以使用 kubernetes 代码库中的脚本 kubernetes/hack/update-codegen.sh
这里为了方便使用了 framework.DecodeInto 的方式。
项目部署
准备 scheduler 的 profile，可以看到，我们自定义的参数，就可以被识别为 KubeSchedulerConfiguration 的资源类型了。
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: /mnt/d/src/go_work/customScheduler/scheduler.conf
profiles:
- schedulerName: custom-scheduler
plugins:
score:
enabled:
- name: "NetworkTraffic"
disabled:
- name: "*"
pluginConfig:
- name: "NetworkTraffic"
args:
ip: "http://10.0.0.4:9090"
deviceName: "eth0"
timeRange: 60
如果需要部署到集群内部，可以打包成镜像
FROM golang:alpine AS builder
MAINTAINER cylon
WORKDIR /scheduler
COPY ./ /scheduler
ENV GOPROXY https://goproxy.cn,direct
RUN \
sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
apk add upx && \
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w" -o scheduler main.go && \
upx -1 scheduler && \
chmod +x scheduler
FROM alpine AS runner
WORKDIR /go/scheduler
COPY --from=builder /scheduler/scheduler .
COPY --from=builder /scheduler/scheduler.yaml /etc/
VOLUME ["./scheduler"]
部署在集群内部所需的资源清单
apiVersion: v1
kind: ServiceAccount
metadata:
name: scheduler-sa
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: scheduler
subjects:
- kind: ServiceAccount
name: scheduler-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: system:kube-scheduler
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: custom-scheduler
namespace: kube-system
labels:
component: custom-scheduler
spec:
selector:
matchLabels:
component: custom-scheduler
template:
metadata:
labels:
component: custom-scheduler
spec:
serviceAccountName: scheduler-sa
priorityClassName: system-cluster-critical
containers:
- name: scheduler
image: cylonchau/custom-scheduler:v0.0.1
imagePullPolicy: IfNotPresent
command:
- ./scheduler
- --config=/etc/scheduler.yaml
- --v=3
livenessProbe:
httpGet:
path: /healthz
port: 10251
initialDelaySeconds: 15
readinessProbe:
httpGet:
path: /healthz
port: 10251
启动自定义 scheduler，这里通过简单的二进制方式启动，所以需要一个kubeconfig做认证文件
./main --logtostderr=true \
--address=127.0.0.1 \
--v=3 \
--config=`pwd`/scheduler.yaml \
--kubeconfig=`pwd`/scheduler.conf
启动后为了验证方便性，关闭了原来的 kube-scheduler 服务，因为原来的 kube-scheduler 已经作为HA中的master，所以不会使用自定义的 scheduler 导致pod pending。
验证结果
准备一个需要部署的Pod，指定使用的调度器名称
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
schedulerName: custom-scheduler
这里实验环境为2个节点的kubernetes集群，master与node01，因为master的服务比node01要多，这种情况下不管怎样，调度结果永远会被调度到node01上。
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-69f76b454c-lpwbl 1/1 Running 0 43s 192.168.0.17 node01
nginx-deployment-69f76b454c-vsb7k 1/1 Running 0 43s 192.168.0.16 node01
而调度器的日志如下
I0808 01:56:31.098189 27131 networktraffic.go:83] [NetworkTraffic] node 'node01' bandwidth: %!s(int64=12541068340)
I0808 01:56:31.098461 27131 networktraffic.go:70] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 12541068340}]
I0808 01:56:31.098651 27131 networktraffic.go:70] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 71}]
I0808 01:56:31.098911 27131 networktraffic.go:73] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 71}]
I0808 01:56:31.099275 27131 default_binder.go:51] Attempting to bind default/nginx-deployment-69f76b454c-vsb7k to node01
I0808 01:56:31.101414 27131 eventhandlers.go:225] add event for scheduled pod default/nginx-deployment-69f76b454c-lpwbl
I0808 01:56:31.101414 27131 eventhandlers.go:205] delete event for unscheduled pod default/nginx-deployment-69f76b454c-lpwbl
I0808 01:56:31.103604 27131 scheduler.go:609] "Successfully bound pod to node" pod="default/nginx-deployment-69f76b454c-lpwbl" node="no
de01" evaluatedNodes=2 feasibleNodes=2
I0808 01:56:31.104540 27131 scheduler.go:609] "Successfully bound pod to node" pod="default/nginx-deployment-69f76b454c-vsb7k" node="no
de01" evaluatedNodes=2 feasibleNodes=2
Reference
[1] scheduling config
[2] kube-scheduler
[3] scheduling-plugins
[4] custom scheduler plugins
[5] ssues #79384
[6] scheduler perf tuning
[7] creating a kube-scheduler plugin
filter.filter.preFilter, filter, preScore, score.queueSort.import (
scheduler "k8s.io/kubernetes/cmd/kube-scheduler/app"
)
func main() {
command := scheduler.NewSchedulerCommand(
scheduler.WithPlugin("example-plugin1", ExamplePlugin1),
scheduler.WithPlugin("example-plugin2", ExamplePlugin2))
if err := command.Execute(); err != nil {
fmt.Fprintf(os.Stderr, "%v\n", err)
os.Exit(1)
}
}
而 NewSchedulerCommand 允许注入 out of tree plugins，也就是注入外部的自定义 plugins，这种情况下就无需通过修改源码方式去定义一个调度器，而仅仅通过自行实现即可完成一个自定义调度器。
// WithPlugin 用于注入out of tree plugins 因此scheduler代码中没有其引用。
func WithPlugin(name string, factory runtime.PluginFactory) Option {
return func(registry runtime.Registry) error {
return registry.Register(name, factory)
}
}
插件实现
对于插件的实现仅仅需要实现对应的扩展点接口。下面通过内置插件进行分析
对于内置插件 NodeAffinity ,我们通过观察他的结构可以发现，实现插件就是实现对应的扩展点抽象 interface 即可。
定义插件结构体
其中 framework.FrameworkHandle 是提供了Kubernetes API与 scheduler 之间调用使用的，通过结构可以看出包含 lister，informer等等，这个参数也是必须要实现的。
type NodeAffinity struct {
handle framework.FrameworkHandle
}
实现对应的扩展点
func (pl *NodeAffinity) Score(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) {
nodeInfo, err := pl.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)
if err != nil {
return 0, framework.NewStatus(framework.Error, fmt.Sprintf("getting node %q from Snapshot: %v", nodeName, err))
}
node := nodeInfo.Node()
if node == nil {
return 0, framework.NewStatus(framework.Error, fmt.Sprintf("getting node %q from Snapshot: %v", nodeName, err))
}
affinity := pod.Spec.Affinity
var count int64
// A nil element of PreferredDuringSchedulingIgnoredDuringExecution matches no objects.
// An element of PreferredDuringSchedulingIgnoredDuringExecution that refers to an
// empty PreferredSchedulingTerm matches all objects.
if affinity != nil && affinity.NodeAffinity != nil && affinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution != nil {
// Match PreferredDuringSchedulingIgnoredDuringExecution term by term.
for i := range affinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution {
preferredSchedulingTerm := &affinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution[i]
if preferredSchedulingTerm.Weight == 0 {
continue
}
// TODO: Avoid computing it for all nodes if this becomes a performance problem.
nodeSelector, err := v1helper.NodeSelectorRequirementsAsSelector(preferredSchedulingTerm.Preference.MatchExpressions)
if err != nil {
return 0, framework.NewStatus(framework.Error, err.Error())
}
if nodeSelector.Matches(labels.Set(node.Labels)) {
count += int64(preferredSchedulingTerm.Weight)
}
}
}
return count, nil
}
最后在通过实现一个 New 函数来提供注册这个扩展的方法。通过这个 New 函数可以在 main.go 中将其作为 out of tree plugins 注入到 scheduler 中即可
// New initializes a new plugin and returns it.
func New(_ runtime.Object, h framework.FrameworkHandle) (framework.Plugin, error) {
return &NodeAffinity{handle: h}, nil
}
实验：基于网络流量的调度 promQL 与 client_golang 有所了解
实验大致分为以下几个步骤：
- 定义插件API
- 插件命名为
NetworkTraffic
- 定义扩展点
- 这里使用了 Score 扩展点，并且定义评分的算法
- 定义分数获取途径（从prometheus指标中拿到对应的数据）
- 定义对自定义调度器的参数传入
- 将项目部署到集群中（集群内部署与集群外部署）
- 实验的结果验证
实验将仿照内置插件 nodeaffinity 完成代码编写，为什么选择这个插件，只是因为这个插件相对比较简单，并且与我们实验目的基本相同，其实其他插件也是同样的效果。
整个实验的代码上传至 github.com/CylonChau/customScheduler
实验开始
错误处理
在初始化项目时，go mod tidy 等操作时，会遇到大量下面的错误
go: github.com/GoogleCloudPlatform/spark-on-k8s-operator@v0.0.0-20210307184338-1947244ce5f4 requires
k8s.io/apiextensions-apiserver@v0.0.0: reading k8s.io/apiextensions-apiserver/go.mod at revision v0.0.0: unknown revision v0.0.0
kubernetes issue #79384 framework.DecodeInto 函数可以做这个操作
func New(plArgs *runtime.Unknown, handle framework.FrameworkHandle) (framework.Plugin, error) {
args := Args{}
if err := framework.DecodeInto(plArgs, &args); err != nil {
return nil, err
}
...
}
另外一种方式是必须实现对应的深拷贝方法，例如 NodeLabel 中的
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// NodeLabelArgs holds arguments used to configure the NodeLabel plugin.
type NodeLabelArgs struct {
metav1.TypeMeta
// PresentLabels should be present for the node to be considered a fit for hosting the pod
PresentLabels []string
// AbsentLabels should be absent for the node to be considered a fit for hosting the pod
AbsentLabels []string
// Nodes that have labels in the list will get a higher score.
PresentLabelsPreference []string
// Nodes that don't have labels in the list will get a higher score.
AbsentLabelsPreference []string
}
最后将其注册到register中，整个行为与扩展APIServer是类似的
// addKnownTypes registers known types to the given scheme
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&KubeSchedulerConfiguration{},
&Policy{},
&InterPodAffinityArgs{},
&NodeLabelArgs{},
&NodeResourcesFitArgs{},
&PodTopologySpreadArgs{},
&RequestedToCapacityRatioArgs{},
&ServiceAffinityArgs{},
&VolumeBindingArgs{},
&NodeResourcesLeastAllocatedArgs{},
&NodeResourcesMostAllocatedArgs{},
)
scheme.AddKnownTypes(schema.GroupVersion{Group: "", Version: runtime.APIVersionInternal}, &Policy{})
return nil
}
Notes：对于生成深拷贝函数及其他文件，可以使用 kubernetes 代码库中的脚本 kubernetes/hack/update-codegen.sh
这里为了方便使用了 framework.DecodeInto 的方式。
项目部署
准备 scheduler 的 profile，可以看到，我们自定义的参数，就可以被识别为 KubeSchedulerConfiguration 的资源类型了。
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: /mnt/d/src/go_work/customScheduler/scheduler.conf
profiles:
- schedulerName: custom-scheduler
plugins:
score:
enabled:
- name: "NetworkTraffic"
disabled:
- name: "*"
pluginConfig:
- name: "NetworkTraffic"
args:
ip: "http://10.0.0.4:9090"
deviceName: "eth0"
timeRange: 60
如果需要部署到集群内部，可以打包成镜像
FROM golang:alpine AS builder
MAINTAINER cylon
WORKDIR /scheduler
COPY ./ /scheduler
ENV GOPROXY https://goproxy.cn,direct
RUN \
sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
apk add upx && \
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w" -o scheduler main.go && \
upx -1 scheduler && \
chmod +x scheduler
FROM alpine AS runner
WORKDIR /go/scheduler
COPY --from=builder /scheduler/scheduler .
COPY --from=builder /scheduler/scheduler.yaml /etc/
VOLUME ["./scheduler"]
部署在集群内部所需的资源清单
apiVersion: v1
kind: ServiceAccount
metadata:
name: scheduler-sa
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: scheduler
subjects:
- kind: ServiceAccount
name: scheduler-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: system:kube-scheduler
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: custom-scheduler
namespace: kube-system
labels:
component: custom-scheduler
spec:
selector:
matchLabels:
component: custom-scheduler
template:
metadata:
labels:
component: custom-scheduler
spec:
serviceAccountName: scheduler-sa
priorityClassName: system-cluster-critical
containers:
- name: scheduler
image: cylonchau/custom-scheduler:v0.0.1
imagePullPolicy: IfNotPresent
command:
- ./scheduler
- --config=/etc/scheduler.yaml
- --v=3
livenessProbe:
httpGet:
path: /healthz
port: 10251
initialDelaySeconds: 15
readinessProbe:
httpGet:
path: /healthz
port: 10251
启动自定义 scheduler，这里通过简单的二进制方式启动，所以需要一个kubeconfig做认证文件
./main --logtostderr=true \
--address=127.0.0.1 \
--v=3 \
--config=`pwd`/scheduler.yaml \
--kubeconfig=`pwd`/scheduler.conf
启动后为了验证方便性，关闭了原来的 kube-scheduler 服务，因为原来的 kube-scheduler 已经作为HA中的master，所以不会使用自定义的 scheduler 导致pod pending。
验证结果
准备一个需要部署的Pod，指定使用的调度器名称
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
schedulerName: custom-scheduler
这里实验环境为2个节点的kubernetes集群，master与node01，因为master的服务比node01要多，这种情况下不管怎样，调度结果永远会被调度到node01上。
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-69f76b454c-lpwbl 1/1 Running 0 43s 192.168.0.17 node01
nginx-deployment-69f76b454c-vsb7k 1/1 Running 0 43s 192.168.0.16 node01
而调度器的日志如下
I0808 01:56:31.098189 27131 networktraffic.go:83] [NetworkTraffic] node 'node01' bandwidth: %!s(int64=12541068340)
I0808 01:56:31.098461 27131 networktraffic.go:70] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 12541068340}]
I0808 01:56:31.098651 27131 networktraffic.go:70] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 71}]
I0808 01:56:31.098911 27131 networktraffic.go:73] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 71}]
I0808 01:56:31.099275 27131 default_binder.go:51] Attempting to bind default/nginx-deployment-69f76b454c-vsb7k to node01
I0808 01:56:31.101414 27131 eventhandlers.go:225] add event for scheduled pod default/nginx-deployment-69f76b454c-lpwbl
I0808 01:56:31.101414 27131 eventhandlers.go:205] delete event for unscheduled pod default/nginx-deployment-69f76b454c-lpwbl
I0808 01:56:31.103604 27131 scheduler.go:609] "Successfully bound pod to node" pod="default/nginx-deployment-69f76b454c-lpwbl" node="no
de01" evaluatedNodes=2 feasibleNodes=2
I0808 01:56:31.104540 27131 scheduler.go:609] "Successfully bound pod to node" pod="default/nginx-deployment-69f76b454c-vsb7k" node="no
de01" evaluatedNodes=2 feasibleNodes=2
Reference
[1] scheduling config
[2] kube-scheduler
[3] scheduling-plugins
[4] custom scheduler plugins
[5] ssues #79384
[6] scheduler perf tuning
[7] creating a kube-scheduler plugin
- 插件命名为
NetworkTraffic
- 这里使用了 Score 扩展点，并且定义评分的算法
go mod tidy 等操作时，会遇到大量下面的错误go: github.com/GoogleCloudPlatform/spark-on-k8s-operator@v0.0.0-20210307184338-1947244ce5f4 requires
k8s.io/apiextensions-apiserver@v0.0.0: reading k8s.io/apiextensions-apiserver/go.mod at revision v0.0.0: unknown revision v0.0.0
func New(plArgs *runtime.Unknown, handle framework.FrameworkHandle) (framework.Plugin, error) {
args := Args{}
if err := framework.DecodeInto(plArgs, &args); err != nil {
return nil, err
}
...
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// NodeLabelArgs holds arguments used to configure the NodeLabel plugin.
type NodeLabelArgs struct {
metav1.TypeMeta
// PresentLabels should be present for the node to be considered a fit for hosting the pod
PresentLabels []string
// AbsentLabels should be absent for the node to be considered a fit for hosting the pod
AbsentLabels []string
// Nodes that have labels in the list will get a higher score.
PresentLabelsPreference []string
// Nodes that don't have labels in the list will get a higher score.
AbsentLabelsPreference []string
}
// addKnownTypes registers known types to the given scheme
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&KubeSchedulerConfiguration{},
&Policy{},
&InterPodAffinityArgs{},
&NodeLabelArgs{},
&NodeResourcesFitArgs{},
&PodTopologySpreadArgs{},
&RequestedToCapacityRatioArgs{},
&ServiceAffinityArgs{},
&VolumeBindingArgs{},
&NodeResourcesLeastAllocatedArgs{},
&NodeResourcesMostAllocatedArgs{},
)
scheme.AddKnownTypes(schema.GroupVersion{Group: "", Version: runtime.APIVersionInternal}, &Policy{})
return nil
}
Notes：对于生成深拷贝函数及其他文件，可以使用 kubernetes 代码库中的脚本 kubernetes/hack/update-codegen.sh
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: /mnt/d/src/go_work/customScheduler/scheduler.conf
profiles:
- schedulerName: custom-scheduler
plugins:
score:
enabled:
- name: "NetworkTraffic"
disabled:
- name: "*"
pluginConfig:
- name: "NetworkTraffic"
args:
ip: "http://10.0.0.4:9090"
deviceName: "eth0"
timeRange: 60
FROM golang:alpine AS builder
MAINTAINER cylon
WORKDIR /scheduler
COPY ./ /scheduler
ENV GOPROXY https://goproxy.cn,direct
RUN \
sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
apk add upx && \
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w" -o scheduler main.go && \
upx -1 scheduler && \
chmod +x scheduler
FROM alpine AS runner
WORKDIR /go/scheduler
COPY --from=builder /scheduler/scheduler .
COPY --from=builder /scheduler/scheduler.yaml /etc/
VOLUME ["./scheduler"]
apiVersion: v1
kind: ServiceAccount
metadata:
name: scheduler-sa
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: scheduler
subjects:
- kind: ServiceAccount
name: scheduler-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: system:kube-scheduler
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: custom-scheduler
namespace: kube-system
labels:
component: custom-scheduler
spec:
selector:
matchLabels:
component: custom-scheduler
template:
metadata:
labels:
component: custom-scheduler
spec:
serviceAccountName: scheduler-sa
priorityClassName: system-cluster-critical
containers:
- name: scheduler
image: cylonchau/custom-scheduler:v0.0.1
imagePullPolicy: IfNotPresent
command:
- ./scheduler
- --config=/etc/scheduler.yaml
- --v=3
livenessProbe:
httpGet:
path: /healthz
port: 10251
initialDelaySeconds: 15
readinessProbe:
httpGet:
path: /healthz
port: 10251
./main --logtostderr=true \
--address=127.0.0.1 \
--v=3 \
--config=`pwd`/scheduler.yaml \
--kubeconfig=`pwd`/scheduler.conf
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
schedulerName: custom-scheduler
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-69f76b454c-lpwbl 1/1 Running 0 43s 192.168.0.17 node01
nginx-deployment-69f76b454c-vsb7k 1/1 Running 0 43s 192.168.0.16 node01
I0808 01:56:31.098189 27131 networktraffic.go:83] [NetworkTraffic] node 'node01' bandwidth: %!s(int64=12541068340)
I0808 01:56:31.098461 27131 networktraffic.go:70] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 12541068340}]
I0808 01:56:31.098651 27131 networktraffic.go:70] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 71}]
I0808 01:56:31.098911 27131 networktraffic.go:73] [NetworkTraffic] Nodes final score: [{master-machine 0} {node01 71}]
I0808 01:56:31.099275 27131 default_binder.go:51] Attempting to bind default/nginx-deployment-69f76b454c-vsb7k to node01
I0808 01:56:31.101414 27131 eventhandlers.go:225] add event for scheduled pod default/nginx-deployment-69f76b454c-lpwbl
I0808 01:56:31.101414 27131 eventhandlers.go:205] delete event for unscheduled pod default/nginx-deployment-69f76b454c-lpwbl
I0808 01:56:31.103604 27131 scheduler.go:609] "Successfully bound pod to node" pod="default/nginx-deployment-69f76b454c-lpwbl" node="no
de01" evaluatedNodes=2 feasibleNodes=2
I0808 01:56:31.104540 27131 scheduler.go:609] "Successfully bound pod to node" pod="default/nginx-deployment-69f76b454c-vsb7k" node="no
de01" evaluatedNodes=2 feasibleNodes=2
Reference
[1] scheduling config
[2] kube-scheduler
[3] scheduling-plugins
[4] custom scheduler plugins
[5] ssues #79384
[6] scheduler perf tuning
[7] creating a kube-scheduler plugin