Go Concurrency: Building a Parallel File Hasher

Table of Contents

  1. Introduction
  2. Prerequisites
  3. Setup
  4. Overview
  5. Implementation
  6. Usage

  7. Conclusion

Introduction

In this tutorial, we will explore Go concurrency by building a parallel file hasher. We will leverage the power of Goroutines and channels to process multiple files concurrently, improving the hashing performance. By the end of this tutorial, you will understand how to utilize Go’s concurrency features to optimize file processing tasks.

Prerequisites

To follow along with this tutorial, you should have a basic understanding of the Go programming language syntax and concepts. Familiarity with Goroutines, channels, and file I/O operations in Go will be helpful. Additionally, ensure that Go is installed on your machine.

Setup

Before we begin, let’s set up our Go project by creating a new directory and initializing a Go module.

mkdir file-hasher
cd file-hasher
go mod init github.com/your-username/file-hasher

Overview

Our goal is to build a program that can compute the hash of each file in a given directory concurrently. We will utilize Goroutines to process multiple files simultaneously, and channels to communicate the computed hash results back to the main Goroutine.

The following steps will be covered in this tutorial:

  1. Walk through all files in a directory
  2. Create Goroutines to compute the hash of each file concurrently
  3. Use channels to send the computed hash values to the main Goroutine

  4. Handle errors encountered during the file processing

  5. Measure the performance improvement achieved through concurrency

    Now let’s dive into the implementation details.

Implementation

First, let’s create a new Go file called main.go. We will use the filepath.Walk function to recursively traverse the directory and fetch all files.

package main

import (
	"crypto/md5"
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"path/filepath"
)

func main() {
	if len(os.Args) < 2 {
		log.Fatal("Please provide a directory path")
	}

	dir := os.Args[1]

	err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
		if err != nil {
			log.Printf("Error accessing path: %s\n", path)
			return err
		}

		if !info.IsDir() {
			data, err := ioutil.ReadFile(path)
			if err != nil {
				log.Printf("Error reading file: %s\n", path)
				return nil
			}

			// Compute the hash of the file using md5
			hash := md5.Sum(data)
			fmt.Printf("File: %s, Hash: %x\n", path, hash)
		}

		return nil
	})

	if err != nil {
		log.Fatal(err)
	}
}

The above code walks through each file in the provided directory, reads the file content, and computes its hash using the md5.Sum function. It prints the file path and the computed hash.

Now, let’s introduce concurrency into our implementation.

package main

import (
	"crypto/md5"
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"path/filepath"
	"sync"
)

func main() {
	if len(os.Args) < 2 {
		log.Fatal("Please provide a directory path")
	}

	dir := os.Args[1]

	// Create a WaitGroup to ensure all Goroutines finish before quitting
	var wg sync.WaitGroup

	// Create an unbuffered channel to receive computed hashes
	hashChan := make(chan string)

	err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
		if err != nil {
			log.Printf("Error accessing path: %s\n", path)
			return err
		}

		if !info.IsDir() {
			wg.Add(1)

			go func(path string) {
				defer wg.Done()

				data, err := ioutil.ReadFile(path)
				if err != nil {
					log.Printf("Error reading file: %s\n", path)
					return
				}

				// Compute the hash of the file using md5
				hash := md5.Sum(data)

				// Send the computed hash value to the channel
				hashChan <- fmt.Sprintf("File: %s, Hash: %x\n", path, hash)
			}(path)
		}

		return nil
	})

	if err != nil {
		log.Fatal(err)
	}

	go func() {
		// Wait for all Goroutines to finish
		wg.Wait()
		// Close the hash channel to signal completion
		close(hashChan)
	}()

	// Receive computed hash values from the channel
	for hash := range hashChan {
		fmt.Print(hash)
	}
}

Here, we introduced a WaitGroup wg to track the completion of each Goroutine. For every file encountered, we increase the WaitGroup count, launch a new Goroutine to process the file, and decrease the WaitGroup count once done. This ensures that all Goroutines finish before the program exits.

We also introduced an unbuffered channel hashChan to receive the computed hash values. Each Goroutine sends the hash value to this channel using the <- operator. Additionally, we added a separate Goroutine (anonymous function) to handle the channel’s closing when all computations are completed.

Usage

To use the parallel file hasher, open a terminal and navigate to the project’s directory. Execute the following command:

go run main.go /path/to/directory

Replace /path/to/directory with the path to the directory containing the files whose hashes you want to compute. The program will then display the file path and its corresponding hash.

Conclusion

In this tutorial, we explored Go concurrency by building a parallel file hasher. We leveraged Goroutines and channels to process multiple files concurrently, resulting in improved performance. We covered the steps to traverse a directory, compute the hash of each file using Goroutines, and communicate the results back to the main Goroutine using channels.

By using Go’s concurrency features effectively, we can significantly optimize file processing tasks. It’s important to handle errors and ensure synchronization between Goroutines when working with concurrency in order to avoid race conditions or deadlocks.

Experiment with different file sizes and directories to observe the performance improvement achieved through parallel file hashing. Happy coding with Go!

Published:

Related Articles