Developing a CDN Service with Go

Table of Contents

  1. Introduction
  2. Prerequisites
  3. Setting Up the Project
  4. Creating a Basic HTTP Server
  5. Implementing File Upload
  6. Adding File Retrieval and Caching
  7. Load Balancing and Scaling
  8. Conclusion

Introduction

In this tutorial, we will learn how to develop a Content Delivery Network (CDN) service using Go. A CDN is a network of servers distributed geographically that helps deliver web content more efficiently. By the end of this tutorial, you will have a basic understanding of CDN concepts and be able to build your own CDN service using Go.

Prerequisites

Before starting this tutorial, you should have the following prerequisites:

  1. Basic knowledge of the Go programming language.

  2. Go installed on your machine.

  3. Familiarity with HTTP concepts.

Setting Up the Project

To begin, let’s set up the project:

  1. Create a new directory for your project:

    ```shell
mkdir cdn-service
cd cdn-service
```

  2. Initialize a Go module:

    ```shell
go mod init github.com/your-username/cdn-service
```

  3. Create a new Go file named main.go:

    ```shell
touch main.go
```

  4. Open main.go in your preferred code editor.

Creating a Basic HTTP Server

Let’s start by creating a basic HTTP server that listens for incoming requests:

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", handleRequest)

	log.Println("Server started on http://localhost:8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello, CDN!")
}

In the code above, we import the required packages (fmt, log, and net/http) and define a handleRequest function that writes a simple response to the client.

To start the server, run the following command:

go run main.go

Now, if you visit http://localhost:8080 in your browser, you should see the “Hello, CDN!” message displayed.

Implementing File Upload

Now, let’s add functionality to handle file uploads. We’ll use the multipart/form-data content type to support file uploads.

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
	"path/filepath"
)

func main() {
	http.HandleFunc("/", handleRequest)

	log.Println("Server started on http://localhost:8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
	if r.Method == "POST" {
		file, handler, err := r.FormFile("file")
		if err != nil {
			fmt.Println(err)
			return
		}
		defer file.Close()

		// Save the file to disk
		filePath := filepath.Join("uploads", handler.Filename)
		out, err := os.Create(filePath)
		if err != nil {
			fmt.Println(err)
			return
		}
		defer out.Close()

		_, err = io.Copy(out, file)
		if err != nil {
			fmt.Println(err)
			return
		}

		fmt.Fprintf(w, "File uploaded successfully!")
	} else {
		fmt.Fprintf(w, "Upload a file using POST request.")
	}
}

In the updated code, we handle the POST method and save the uploaded file to the uploads directory. We use the FormFile function to retrieve the file from the request and save it using os.Create and io.Copy.

Adding File Retrieval and Caching

To serve the uploaded files to clients, we’ll add a file retrieval functionality. We’ll also implement caching to improve performance.

package main

import (
	"fmt"
	"log"
	"net/http"
	"os"
	"path/filepath"
	"time"

	"github.com/patrickmn/go-cache"
)

var c = cache.New(5*time.Minute, 10*time.Minute)

func main() {
	http.HandleFunc("/", handleRequest)
	http.HandleFunc("/file/", handleFileRequest)

	log.Println("Server started on http://localhost:8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
	if r.Method == "POST" {
		// Handle file upload
		// ...
	} else {
		fmt.Fprintf(w, "Upload a file using POST request.")
	}
}

func handleFileRequest(w http.ResponseWriter, r *http.Request) {
	filePath := filepath.Join("uploads", r.URL.Path[len("/file/"):])
	_, err := os.Stat(filePath)
	if err != nil {
		http.NotFound(w, r)
		return
	}

	// Check if the file is cached
	if data, found := c.Get(filePath); found {
		// Use the cached version
		http.ServeContent(w, r, "", time.Now(), bytes.NewReader(data.([]byte)))
		return
	}

	// Read and cache the file
	file, err := os.Open(filePath)
	if err != nil {
		http.Error(w, "Error reading file", http.StatusInternalServerError)
		return
	}
	defer file.Close()

	fileData, err := ioutil.ReadAll(file)
	if err != nil {
		http.Error(w, "Error reading file", http.StatusInternalServerError)
		return
	}

	// Cache the file data
	c.Set(filePath, fileData, cache.DefaultExpiration)

	// Serve the file
	http.ServeContent(w, r, "", time.Now(), bytes.NewReader(fileData))
}

In the updated code, we use a caching mechanism provided by the go-cache package to store file data. We first check if the file exists in the cache and serve it if found. If the file is not in the cache, we read it from disk, cache it, and then serve it.

Load Balancing and Scaling

To handle increased load and improve availability, we can use load balancing and scaling techniques. We’ll use the Go reverse proxy package (net/http/httputil) to achieve this.

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
	"os"
)

func main() {
	http.HandleFunc("/", handleRequest)
	http.HandleFunc("/file/", handleFileRequest)
	http.HandleFunc("/upload/", handleUploadRequest)

	log.Println("Server started on http://localhost:8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
	if r.URL.Path == "/" {
		fmt.Fprintf(w, "CDN HomePage")
	} else {
		// Proxy the request to the backend servers
		backendURL, _ := url.Parse("http://localhost:8000")
		proxy := httputil.NewSingleHostReverseProxy(backendURL)
		proxy.ServeHTTP(w, r)
	}
}

func handleFileRequest(w http.ResponseWriter, r *http.Request) {
	// Serve the file
	// ...
}

func handleUploadRequest(w http.ResponseWriter, r *http.Request) {
	// Handle the file upload
	// ...
}

In the code above, we update the handleRequest function to proxy requests to backend servers using the ReverseProxy functionality. The backend URL is specified as http://localhost:8000, but you can replace it with your own.

Conclusion

In this tutorial, we learned how to develop a CDN service using Go. We covered the basics of setting up an HTTP server, handling file uploads, implementing file retrieval with caching, and load balancing with reverse proxies. With this knowledge, you can further enhance the CDN service by adding more advanced features and optimizations.

Remember to always consider security, scalability, and performance when building a production-ready CDN service.

Published:

Related Articles