Consistent Data Interface (Part II)

As promised in the last installment of this post, I will be providing a demonstration of constraining a problem through generic interfaces. This post will focus upon this in terms of C#, .NET 3.1 and AWS’s DynamoDB.

As I was recently reading through the DynamoDB documentation and working off the examples, it occurred to me that documentation provides the perfect base with which to build generic interfaces. By focusing upon the main attributes of the objects described in the documentation, we can model out a façade to serve as an entry-point for any specific functionality that we would like out of the service.

Before we get to that, we should define what we want out of a database. Now normally, it is CRUD operations.

A rudimentary C# interface for CRUD for operations on a NOSQL db might looks something like:

 public interface IHasCRUD<TableEntry, Key>
    {
        public Key Create(TableEntry instance);
        public TableEntry Read(Key key);
        public List<TableEntry> Read();
        public void Update(TableEntry updatedInstance);
        public void Delete(Key key);
    }

It gets the idea across and accounts for unknown types, but has some immediate shortcomings, the first being synchronicity. Let’s update the interface to account for this. In C#, the async/await interface returns Tasks (as opposed to the Promises of JS).

public interface IHasCRUD<TableEntry, Key>
{
    public Task<Key> Create(TableEntry instance);
    public Task<TableEntry> Read(Key key);
    public Task<List<TableEntry>> Read();
    public Task Update(TableEntry updatedInstance);
    public Task Delete(Key key);
}

Ok, so that might work for a generic DB, but what about DynamoDB? Reading through the docs, we find that the main aspects of an object held within DynamoDB are the partition key and range key. The partition key, being the attribute that is used to separate out different portions of the data and the range key being the attribute used to query or scan the data. Let’s expand our interface.

public interface IHasDynamoCRUD<TableEntry, Partition, Key>
{
    public Task<Key> Create(TableEntry instance);
    public Task<TableEntry> Read(Partition partition, Key key);
    public Task<List<TableEntry>> Read(Partition partition);
    public Task Update(TableEntry updatedInstance);
    public Task Delete(Partition partition, Key key);
}

Ok, so that interface is looking a bit better, but how can we be assured that the data types we are using with this generic interface match up? Of course, we can extend our use of interfaces to ensure that the data types that we use are compatible.

 public interface IHasDynamoKeys<PartitionKey, RangeKey>{ 
    public PartitionKey GetPartitionKey();
    public RangeKey GetRangeKey();
}

Before we specify any of the implementation details of the interface, let’s add one more complication to the mix. Dynamo DB queries can retrieve a maximum of 1MB of data. This either means that the objects that we are saving need to be really small, or we need a targeted way identifying the data that we need.

One solution to this is a two table, index/data based solution. In one of our tables, we hold a reference (pointer if you will) to our data, while the other holds the actual data. The modified interface for this strategy looks like this.

public interface IAmDataTable{ }
public interface IAmIndexTable{ }
public interface IConvertsToIndex<IndexTableEntry>
{
    public IndexTableEntry ToIndex();
}
public interface IHasIndexedDynamoCRUD<IndexTableEntry, DataTableEntry, Partition, Key>
{
    public Task<Key> Create(DataTableEntry instance);
    public Task<DataTableEntry> Read(Partition partition, Key key);
    public Task<List<IndexTableEntry>> Read(Partition partition);
    public Task Update(DataTableEntry updatedInstance);
    public Task Delete(Partition partition, Key key);
}

Now the interface is getting a bit more involved. Read all operations are run through the index table, limiting the size of the query and returned results. Read one and update queries are run through the data table, returning or updating actual entries. Lastly create and delete bring in some complications in terms of state.

In order to maintain data consistency between the index table, we need to ensure that create and delete are transactions or “all or nothing” operations.

With our interfaces defined, we can move on to the implementation phase. Tune in on the next installation of consistent data interfaces for further details!