On interfaces and composition

Abstractions are great, when done in moderation, with the goal in mind, and using the language features designed to take advantage of it. We’re doing some work around storage and I’ve been thinking about how we might go about implementing it in a way that makes working with code easy to understand and contained to small units that we can swap in and out as needed.

In most applications you will need a way to store things so they can be retrieved at a later date. Whether that’s compiled build artefacts, metrics, logs, comments, blog posts, images, it doesn’t matter. You have a piece of data, and you want to put it somewhere where you can then find it and do something with it 4 weeks later.

The very first iteration of an app is most probably going to have a database, like MySQL, coded into it, as that’s the one that you decided to use and it’s easy and there’s no reason to use indirections and abstractions.

The issues come when you need to replace MySQL with something else for whatever reason. Now you need to rip out half of your codebase because MySQL was hardcoded.

Enter interfaces

Let’s say your app only really cares about two things: users and products. The only thing your app needs to do with regards to those are

• storing a user
• retrieving a user by email address or ID
• storing a product
• retrieving a product by its ID

That’s it.

So given the following pseudocode, the app only cares about the 4 operations above, but it doesn’t really need to know how those are achieved.

package app

type User struct {
    ID int
    Name string,
    Email string,
    Password string,
    Confirmed bool
}

type Product struct {
    ID int
    Title string,
    Author User,
    Description string,
    Price int
}

func (a *App) ProductsForUser(user User) ([]Product, error) {
    // Get products for the user
}

Hence interfaces. They describe a set of behaviours that implementations would need to do. This hypothetical use case would have this interface.

package app

type User struct {
    ID int
    Name string,
    Email string,
    Password string,
    Confirmed bool
}

type Product struct {
    ID int
    Title string,
    Author User,
    Description string,
    Price int
}

func (a *App) ProductsForUser(user User) ([]Product, error) {
    // Get products for the user
}

Tying this up into the app would look like this, without an implementation or constructor yet.

package app

type App struct {
    storage Storage
}

func New(storage Storage) App {
    return App{
        storage: storage,
    }
}

func (a *App) ProductsForUser(user User) ([]Product, error) {
    products, err := a.storage.ListProductsForUser(user)
    if err != nil {
        return nil, errors.Wrap(err, "storage.ListProductsForUser for user %d", user.ID)
    }

    return products, nil
}

The great thing is is that it doesn’t really matter what the implementation of Storage is. Let’s plug in our MySQL implementation here:

package mysql

import (
    "database/sql"

    "app"
    
    _ "github.com/go-sql-driver/mysql"
)

// This is the mysql.DB, the implementation of the Storage interface
type DB struct {
    db dbConnection
}

func New(username, password, host, port, databasename) (DB, error) {
    db, err := sql.Open("mysql", fmt.Sprintf(
        "%s:%s@%s:%s/%s",
        username, password, host, port, databasename,
    ))
    if err != nil {
        return Storage{}, errors.Wrapf(err, "sql.Open: %s:***@%s:%s/%s",
        username, host, port, databasename)
    }

    db.SetConnMaxLifetime(time.Minute * 3)
    db.SetMaxOpenConns(10)
    db.SetMaxIdleConns(10)

    return DB{
        db: db,
    }, nil
}

func (d *DB) ProductsForUser(user app.User) ([]app.Product, error) {
    d.db.Select(query, user.ID)
}

This will get us direct access to the database, there’s nothing inherently clever or complicated here.

However for performance reasons we might want to cache some of this data, especially if the data tends to be static. Details of certain products would not change often, nor details for the users, and it’s usually faster to talk to a Redis backend than an actual database.

That said we also need to think about how to handle situations where some data is not cached. One of the usual ways to deal with this is to check for the data, and if it’s not there, check the database. Pseudo-code wise it would look like this:

cachedData, found, err := redis.GetSomeData("key")
if err != nil {
    // there was an error, handle it
}

if found {
    // superb, cache had the data
    return cachedData 
}

// cache didn't have the data, let's ask mysql
data, err := database.GetSomeData("key")
if err != nil {
    // even database didn't have it, so 404?
}

// found the data in the database, cache it on redis
// and return it
redis.SetSomeData("key", data)
return data

This works, but not really extensible, kind of hard, and looks a lot like spaghetti. There’s a better way.

Turns out I can wrap the MySQL implementation into the Redis implementation. Also pseudocode:

package redis

import "github.com/go-redis/redis/v8"

type Cache struct {
    fallback Storage // the interface
    client   *redis.Client
}

func New(host, port, password string, db int, fallback Storage) Cache {
    rdb := redis.NewClient(&redis.Options{
        Addr:     fmt.Sprintf("%s:%s", host, port),
        Password: password,
        DB:       db,
    })

    return Cache{
        fallback: fallback,
        client:   rdb,
    }
}

func (c Cache) ProductsForUser(user app.User) ([]app.Product, error) {
    data, err := c.client.Get(ctx, user.ID).Result()
    if err == nil {
        // redis had the data
        return data, nil
    }


    if err == redis.Nil {
    	// redis does not have the data, so check the fallback storage
        products, err := c.fallback.ProductsForUser(user)
        if err != nil {
            return nil, errors.Wrap(err, "redis fallback sent back an error")
        }
        
        // fallback storage had data, cache in redis
        err = c.client.Set(ctx, user.ID, products, 0).Err()
        if err != nil {
            // setting the value to redis failed, but we do have the data
            // log and move on
            log.Warn("could not set data to redis", err)
        }

        return products, nil
    }

    // err is not nil, but also not not found from redis
    return nil, errors.Wrap(err, "something went wrong in redis") 
}

The cool thing about this is that the calling code, our App, does not need to know anything about the inner workings of this. The setup code will look like this:

package main

func main() {
    // create mysql connection
    mysqlStorage, err := mysql.New("username", "password", "host", "port", "databasename")
    if err != nil {
        log.Fatal("could not get mysql")
        os.Exit(1)
    }

    // create redis connection and add the mysql connection as a fallback
    // for all the data that's not cached yet
    redisStorage := redis.New("host", "port", "password", 0, mysqlStorage)
    
    // spin up our app with the redis storage
    service := app.New(redisStorage)

    // service will first check redis, then mysql
    products, err := service.ProductsForUser(43)

    // rest of the owl
}

This way the service’s API remains flat, simple, easy to understand. Moreover each layer of storage remains simple because there are at most two layers they need to concern themselves with: their own (Redis does Redis things), and if that didn’t have the wanted effect, ask the fallback.

The fallbacks are interfaces though, so they don’t need to care what the fallback is, just that asking for the same data looks the same. Let that package worry about it.

This makes it super easy to add additional layers. Do you want an in–memory cache in front of the Redis one? Sure:

package main

func main() {
    // create mysql connection
    mysqlStorage, err := mysql.New("username", "password", "host", "port", "databasename")
    if err != nil {
        log.Fatal("could not get mysql")
        os.Exit(1)
    }

    // create redis connection and add the mysql connection as a fallback
    // for all the data that's not cached yet
    redisStorage := redis.New("host", "port", "password", 0, mysqlStorage)
    
    // wrap the redis into the inmemory
    inmemStorage := inmemory.New(redisStorage)

    // spin up our app with the inmemory storage
    service := app.New(inmemStorage)

    // service will first check inmemory, then redis, then mysql
    products, err := service.ProductsForUser(43)

    // rest of the owl
}

More importantly than then above, unit testing the code becomes easy because we no longer need to spin up an actual Redis server, or a MySQL database, or use the in–memory cache, because what we’re testing isn’t those storage solutions, but rather what the service does with data / errors returned from the storage.

Mockery is a great tool to generate mocks based on interfaces, that way our unit tests can look like this (pseudocode):

func TestGetProduct(t *testing.T) {
    // new mock storage with expecter on it.
    storage := new(mocks.Storage)

    // tell the mock that if the argument for the ProductsForUser method is 43,
    // then return that list of products, nil err,
    // and expect it to be called 1 times.
    storage.EXPECT().ProductsForUser(43).Return([]Product{p1, p2}, nil).Times(1)

    // use the mocked storage implementation with expectations on it
    service := app.New(storage)

    // query the code
    products, err := service.ProductsForUser(43)

    // make the assertions
    storage.AssertExpectations(t)
}

Recap

Using interfaces makes it easy to compose different parts of the services we’re building. That way each individual layer can be isolated, tested, validated, and the entire functionality can be mocked out so we can test the overarching application logic given known returns.

Ultimately this results in easier testing requirements, more confidence in the application code, better test coverage, faster test runs, and easier reasoning about code.

Photo credits

1. Blog header Photo by Kelly Sikkema on Unsplash
2. Wraps! background Photo by Matthew Henry on Unsplash
3. Testing / Mocking background Photo by Annie Spratt on Unsplash