Partial Updates in Couchbase using Go Reflect Package, Part 1

| by David Bolshoy (System Architect)

Hey there!

Here’s David, a System Architect at Rounds. We continue our Go series by showing some neat stuff that can be done using the reflection package. We will learn how Go can examine and alter the internal structure of its runtime objects, and how this can be utilized to create useful generic code (even though Go does not support generics yet).

But first, get acquainted with the reflect package and study the Laws of Reflection. Pay special attention to the third law, you need to understand what settability is and why it can be limited.

Couchbase

Couchbase is one of the most popular open source NoSQL document-oriented databases. Couchbase is designed to provide easy-to-scale key-value or document access with low latency and high sustained throughput. There are many Client SDKs for most popular languages, the popular Go SDK being go-couchbase.

Document Traversal

At Rounds we use Couchbase for many of our REST services, and we are quite happy with it. One thing we find limiting is the lack of partial selects and updates. For example, consider the following document:


"user1": {

    "name": "John",

    "birthday": "1990-01-01",

    "address": {

        "city": "New York",

        "street": "Broadway"

    }

}

Let’s say this document can be accessed with our service using the URI http://service/users/user1. What if we need to access only the address object, or only the city name? We want http://service/users/user1/address to return


{
    "city": "New York",
    "street": "Broadway"
}

and http://service/users/user1/address/city to return "New York".

Well, to accomplish this, we have to fetch the full document first, and then extract the internal object corresponding to the partial URI path:


// Split URI into parts.
partialURI := "/user1/address/city"
parts := strings.Split(partialURI, "/")

// Get couchbase bucket.
bucket, err := couchbase.GetBucket(couchURL, "default", "default")
if nil != err {
    return err
}

// Get our document and deserialize into a general map.
var v map[string]interface{}
if err := bucket.Get(parts[1], v); err != nil { // parts[0] is empty string
    return err
}

// Now traverse by path parts as keys into the map.
var ok bool
cur := v
for i = 2; i < len(parts); i++ {
    p := parts[i]
    if i == len(parts) - 1 { // last path element
        fmt.Println(cur[p]) // Got it
	return
    }
    if cur, ok = cur[p].(map[string]interface{}); !ok { // traverse keys
        fmt.Println("key not found: ", p)
        return
    }
}

This works, but forces us to use the type map[string]interface{}, meaning that we will need lots of type assertions if we want the data to be validated properly. Maybe it’s possible to use Go structures and utilize their type checking abilities?

Enter Structures

We go on and create our User type as below:


type Addr struct {
    City   string `json:"city"`
    Street string `json:"street"`
}
type User struct {
    Name     string `json:"name"`
    Birthday string `json:"birthdate"`
    Address  Addr   `json:"address"`
}

Note the usage of json tags, we will need them later. Now we just ask Couchbase to deserialize into it:


var user User
bucket.Get(userKey, &user)

And voilà, the city data can be accessed as user.Address.City. The deserialization (which uses the powerful json.Unmarshal library function) will take care of type assertions and make sure our struct is valid. But wait, how can we traverse into the structure given an array of path elements?

Structure Traversal

Fortunately, this is where Go reflection package comes to rescue. It has many functions that help us examine and alter the internals of our Go structures.
The first step is to use reflect.ValueOf and reflect.TypeOf to get access to the reflection value and type. Then, we can use the Field(i) method to enumerate the structure fields and examine their information. Remember the json tag? We can use it to match the path elements (because the field names will usually be less suitable, having camel case and what not). The resulting recursive structure traversal function will look along the lines of the following:


func getPartialData(dataIn interface{}, path []string) (partOut interface{}, err error) {

    // Return myself if there's no subpath anymore (recursion termination).
    if len(path) == 0 {
        partOut = dataIn
	return
    }

    // There's a subpath, so we need to traverse.
    typ := reflect.TypeOf(user)
    val := reflect.ValueOf(user)

    if typ.Kind() == reflect.Struct {
        for i := 0; i < typ.NumField(); i++ {
            field := typ.Field(i)
            tag := field.Tag.Get("json") // get json tag of this field
            if tag == "" {               // json tag not given, try field name
                tag = field.Name
            }

            if tag == path[0] {          // subpath matched
                return getPartialData(val.Field(i).Interface(), path[1:])
            }
        }
    }
}

Traversal Goes Berserk

But what are our limitations? Should we use only structure types, or can also use maps, arrays, interfaces, pointers to maps, maps of pointers, pointers to map of pointers to arrays of pointers to maps of arrays ….? Can we define the User type as below and be able to access its internal parts, like /user1/friends/user2/blocked?


type Addr struct {
    City   string `json:"city"`
    Street string `json:"street"`
}
type FriendData struct {
    AddTime Timestamp `json:"add_time"`
    Blocked bool      `json:"blocked"`
}
type Asset struct {
    Name string `json:"name"`
    URL  url    `json:"url"`
}
type User struct {
    Name     string                   `json:"name"`
    Birthday string                   `json:"birthdate"`
    Address  Addr                     `json:"address"`
    Friends  map[string]*FriendData   `json:"friends"`
    Assets   []*Asset                 `json:"assets"`
}

Again Go reflection is here to help. It has methods to access and manipulate maps, arrays and slices, pointers and interfaces. One important function worth noting is Elem(). When applied to an interface or pointer value, it will dereference it and get us access to the concrete value behind it. When applied to an element in the map, it will get us the value of the keyed object.

The resulting function is shown in this Gist. The extraction code is cleaner, data structures are type checked and are easier to use:


// Split URI into parts.
partialURI := "/user1/friends/user2/blocked"
parts := strings.Split(partialURI, "/")

// Get couchbase bucket.
bucket, err := couchbase.GetBucket(couchURL, "default", "default")
if nil != err {
    return err
}

// First get a full document from the couchbase and unmarshal it into our User structure.
var user User
if err := bucket.Get(parts[1], &user); err != nil { 
    return err
}

// Now extract a corresponding part.
if partialData, err := SelectData(&user, parts[2:]); err != nil {
   return err
}
fmt.Println(partialData)

In the second part of this article, we will dive deeper into the reflection and implement partial create/update/delete functionality for Couchbase. Meanwhile (or if you have no patience), you are welcome to check this Gist.

Be back with more goodies soon,

David

Category: ,

Have something to say?