Writing a Kubernetes Custom Scheduler in Go

Table of Contents

  1. Introduction
  2. Prerequisites
  3. Setup
  4. Writing the Custom Scheduler
  5. Testing the Custom Scheduler
  6. Conclusion

Introduction

In this tutorial, we will learn how to write a custom scheduler for Kubernetes using the Go programming language. A scheduler in Kubernetes is responsible for assigning pods to available nodes in the cluster, considering various factors such as resource requirements, node conditions, and custom policies.

By the end of this tutorial, you will have a basic understanding of how the Kubernetes scheduler works and how to implement your own custom logic to schedule pods in a Kubernetes cluster.

Prerequisites

To follow along with this tutorial, you should have the following prerequisites:

  • Basic knowledge of Go programming language
  • Familiarity with Kubernetes concepts like pods and nodes
  • A working Kubernetes cluster

Setup

Before we begin, ensure that you have a working Kubernetes cluster. If you don’t have one already, you can set up a cluster locally using tools like Minikube or use a cloud-based Kubernetes service.

Additionally, make sure you have Go installed on your machine. You can download and install Go from the official Go website (https://golang.org/dl/).

Writing the Custom Scheduler

Step 1: Set up the Go project

Create a new directory for your custom scheduler project and initialize it as a Go module:

$ mkdir custom-scheduler
$ cd custom-scheduler
$ go mod init github.com/your-username/custom-scheduler

Step 2: Import required packages

In your main Go file, import the necessary packages for building a custom scheduler:

package main

import (
	"flag"
	"fmt"
	"os"
	"path/filepath"

	corev1 "k8s.io/api/core/v1"
	utilflag "k8s.io/component-base/cli/flag"
	schedulerconfig "k8s.io/kubernetes/pkg/scheduler/apis/config"
	"k8s.io/kubernetes/pkg/scheduler/framework"
	"k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultbinder"
	"k8s.io/kubernetes/pkg/scheduler/framework/plugins/tainttoleration"
	frameworkcycle "k8s.io/kubernetes/pkg/scheduler/framework/runtime/cycle"
	"k8s.io/kubernetes/pkg/scheduler/framework/runtime/queue"
	"k8s.io/kubernetes/pkg/scheduler/framework/runtime/registry"
	schedulerapp "k8s.io/kubernetes/pkg/scheduler/schedulerapp"
)

In this example, we’re importing various packages required for implementing a custom scheduler, including corev1 for the Kubernetes API, schedulerconfig for scheduler configuration, and framework for the framework plugins.

Step 3: Implement the custom Score and Reserve plugins

To control the scheduling logic, we need to implement custom score and reserve plugins. Create a new file called score.go and add the following code:

package main

import (
	corev1 "k8s.io/api/core/v1"
	framework "k8s.io/kubernetes/pkg/scheduler/framework"
	schedulernode "k8s.io/kubernetes/pkg/scheduler/framework/runtime/node"
	"k8s.io/kubernetes/pkg/scheduler/framework/runtime/predicates"
	"k8s.io/kubernetes/pkg/scheduler/nodeinfo"
)

const (
	// PluginName is the name of the custom score plugin.
	PluginName = "custom-score-plugin"
)

// CustomScorePlugin is a custom score plugin for the scheduler.
type CustomScorePlugin struct {
	handle framework.Handle
}

// Name returns the name of the plugin.
func (cs *CustomScorePlugin) Name() string {
	return PluginName
}

// Score computes the scores for the given node and pod.
func (cs *CustomScorePlugin) Score(pc *framework.PluginContext, pod *corev1.Pod, nodeName string) (float64, *framework.Status) {
	// Your custom scoring logic goes here
}

// New initializes a new instance of the custom score plugin.
func New(csArgs runtime.NamedFunc) (runtime.Plugin, error) {
	csConfig := &CustomScorePlugin{}
	c, err := schedulernode.NewScore(PluginName, csConfig.Score, csArgs)
	if err != nil {
		return nil, err
	}

	csConfig.handle = c

	return csConfig
}

Similarly, create a new file called reserve.go and add the following code for implementing the custom reserve plugin:

package main

import (
	corev1 "k8s.io/api/core/v1"
	framework "k8s.io/kubernetes/pkg/scheduler/framework"
	"k8s.io/kubernetes/pkg/scheduler/framework/runtime"
)

const (
	// PluginName is the name of the custom reserve plugin.
	PluginName = "custom-reserve-plugin"
)

// CustomReservePlugin is a custom reserve plugin for the scheduler.
type CustomReservePlugin struct {
	handle framework.Handle
}

// Name returns the name of the plugin.
func (cr *CustomReservePlugin) Name() string {
	return PluginName
}

// Reserve reserves resources for the given pod on the given node.
func (cr *CustomReservePlugin) Reserve(pc *framework.PluginContext, pod *corev1.Pod, nodeName string) *framework.Status {
	// Your custom resource reservation logic goes here
}

// New initializes a new instance of the custom reserve plugin.
func New() (framework.ResourceReservation, error) {
	crConfig := &CustomReservePlugin{}
	c, err := runtime.NewFrameworkResourceReserver(PluginName,
		crConfig.Reserve)
	if err != nil {
		return nil, err
	}

	crConfig.handle = c

	return crConfig, nil
}

In these plugins, we have placeholders for custom scoring logic and resource reservation logic. You can replace these with your own implementation as per your requirements.

Step 4: Implement the Custom Scheduler

Now, let’s implement the main custom scheduler logic. Create a new file called main.go and add the following code:

package main

func main() {
	framework.Handle([]framework.Plugin{
		// Add your custom plugins here
	}, nil, frameworkruntime.WithEventHandlers(
		events.NewEventRecorder("custom-scheduler"),
		&frameworkcycle.DefaultPlugin{},
	))
}

Here, you can add your custom score and reserve plugins along with other framework plugins based on your requirements.

Testing the Custom Scheduler

To test the custom scheduler, you first need to build the Go binary for your custom scheduler:

$ go build -o custom-scheduler

Next, deploy the custom scheduler in your Kubernetes cluster:

$ kubectl create deployment custom-scheduler --image=your-username/custom-scheduler

Finally, check the logs of the custom scheduler to see if it’s running successfully:

$ kubectl logs -l app=custom-scheduler

If everything is set up correctly, you should see the logs indicating that the custom scheduler has started.

Conclusion

In this tutorial, you learned how to write a custom scheduler for Kubernetes using Go. We covered the basic steps of setting up a Go project, importing the required packages, implementing custom score and reserve plugins, and running the custom scheduler in a Kubernetes cluster. Now, you can further explore the Kubernetes scheduler framework and build more advanced scheduling logic based on your specific use cases.

Remember that this tutorial is just scratching the surface of the vast possibilities and capabilities of the Kubernetes scheduling system. Feel free to experiment and explore more features to make your custom scheduler even more powerful and efficient.

Congratulations on completing this tutorial! You should now have a better understanding of how to write a Kubernetes custom scheduler in Go and the broader concepts of scheduling pods in a Kubernetes cluster.

Published:

Related Articles