Introduction

On Monday, we were talking about NoSQL (Not-only SQL) databases.
These are databases designed to horizontally scale better than and to handle semi-structure data better than traditional RDBMS.
The trade-off as we said is that we might have to settle for eventual consistency of our data rather than consistency at all times.
We said there are four kinds of NoSQL databases: Key-Value Stores, Tuple and Document Stores, Column-Oriented Databases, and Graph-based Databases.
We began looking at Key-Values stores and considered Memcached as a simple case.
Memcached leaves it to the memcached client to determine how to partition data amongst memcached servers, we said more robust systems rely on the node in the key-value store figuring out how to partition the data.
We said there were two aspects of this: a membership in the set of nodes protocol and then a request coordinator protocol so that if a client connects to a node it knows which other node is responsible for the data.
We said the membership protocol can be done using multicast on a LAN of nodes, but is more typical done using a gossip algorithm whereby nodes can exchange information about other nodes they know about via gossiping.

Example - Gossiping in Cassandra

Facebook's Cassandra, a column-oriented db implemented over a key-value store, starts with a set of nodes which are configured as seeds.
At bootstrap time, these nodes are allowed to join the Cassandra ring without seeing any other seed nodes, and are each aware of each other.
Every node in Cassandra runs the following Gossip protocol as a task periodically:
1. Update the local node's heartbeat state (the version) and constructs the node's local view of the cluster gossip endpoint state.
2. Pick a random other node in the cluster to exchange gossip endpoint state with.
3. Probabilistically attempt to gossip with any unreachable nodes (if one exists)
4. Gossip with a seed node if that didn't happen in Step 2
Step 2 and Step 3 allow the system to detect nodes that may have failed. Cassandra uses a `Phi`-Failure detection algorithm to determine whether failed or successful gossip attempts should be viewed as a node being UP or DOWN. (`Phi` is a formula based on the log of probability of going a a given time without successfully gossiping according to a probability distribution of failure. If the `Phi` value is below a threshold the node is assumed to have failed and new jobs aren't sent to it.)
Step 4 allows us to discover nodes that have just entered the system and allow it such knows nodes to find out about other ones.

Distributing Keys to Machines

We now look at the problem of distributing key-value pairs among the nodes.
This can be done using a hashing scheme, however, we want a hashing scheme that minimizes the remapping that occurs if a node is added or removed from our collection of nodes.
For this reason, mod `n` is not typically used as it results in about `k-n` keys or about `1-n/k` fraction of the keys being remapped when increasing the number of machines `n`. ( mod `n` and mod `n+1` share the first `n` values, pretty much everything else needs to be remapped. )
Instead, a so-called consistent hashing scheme is usually used.
To see why mod `n` is bad, consider the following table of 10 keys showing how they would need to be remapped in going from 3 to 4 servers.

Consistent Hashing

A consistent hashing scheme can be used to solve this remapping problem.
In such a scheme, we assume the nodes have a ring topology.
Our hash function `h(k)` should output a decimal number between 0 and 1 and output should appear uniform in this range.
I.e, nodes also have IDs some real number mod 1.
Distance between IDs of nodes and hashes of keys is computed by subtracting their values mod 1.
In a simple scheme a key is stored in the node closest to it.
So if a new node is added roughly `k/(n+1)` keys will be closest to the new node and need to be remapped. This is much smaller than `k-n` keys.

In-Class Exercise

Place four nodes at the N,S,E,W points of a circle.
What are the real numbers for these locations?
Use a random generator to put 10 keys into this circle.
Pick a random new node location and add it to the ring.
Which of your keys needed to be remapped?
Post your solution to the Sep 23 In-Class Exercise.

Replication and Redundancy

Two issues with our consistent hashing are:
1. If two servers are close to each other, one of these nodes will often end up with only a small fraction of the keys to store.
2. If a server is added to the ring, all of the keys that end up at the new node originate from a single other node.
To fix this, rather than map a server `s` to a single point in the ring, we map it to `r` points called replicas (or virtual node).
These could be computed by `r` hash functions `h_1(k),..., h_r(k)`.
This both increases the uniformity in which key-values are mapped to servers and reduces the number of keys that need to be moved when a machine is added.
Replicas do not increase redundancy, the number of machines with a given key-value pair.
To enable `m`-wise, redundancy we use `m`-hash functions for the keys and store the key-value pairs on each of the nodes this hashes correspond to.

Eventual Consistency

Notice the gossip set up does not guarantee that every node is aware of every other node at all times.
Instead, it guarantees that up-to-date information will spread through the network over time.
Clients must deal with the fact that the state of the network might not be perfectly consistent at any moment in time, though it will eventually become consistent at some future point in time.
This is an example of what we meant by eventual consistency.
This is in contrast to the ACID (atomicity, consistency, isolation, and durability) properties associated with a traditional RDBMS.
NoSQL databases try to achieve BASE (basically available, soft state. eventually consistent).

BASE

The BASE acronym is due to Eric Brewer.
It came out about the same time as his statement of the CAP Theorem (1998).
The CAP theorem, which was proven by Gilbert and Lynch in 2002, states that a distributed system of the type we have been looking at cannot simultaneously guarantee:
1. Consistency (all nodes see the same data),
2. Availability (every request receives a response indicating a success or failure),
3. and Partition tolerance (the system works even if nodes go down or are added).
Stand-alone RDBMS don't need to handle Partition Tolerance; distributed RDBMS give up availability to enforce consistency; NoSQL databases give up consistency.
In detail the letters of BASE mean:
- Basically Available - NoSQL databases adhere to the availability guarantee of the CAP theorem;
- Soft state - The system can change over time, even without receiving input (since nodes continue to update each other);
- Eventually consistent - The system will become consistent over time but might not be consistent at a particular moment.

Stabilization

The repartition of key-values over nodes if the number of nodes changes is called stabilization.
Ideally, this should be done as efficiently as possible, minimizing network overhead and waiting time.
Using consistent hashing, we minimize the number of fluctuation in the hash-node mapping, which will help with the stabilization period.
If a node is determined to be removed, other nodes will eventually examine their list of keys to see if any need to be remapped.
If a node is no longer responsible for a key it can use the partitioning scheme to determine who is responsible and send them the key-value pair.
After it is determine that the key-value pair is present in the correct node, the value can be deleted.
For this reason, planned removal of a node often takes time.

Integrity Constraints and Querying

Before we conclude our overview of Key-Value Stores, let me say there are a wide range of possible key-value store systems.
Some use RAM to store key-values, others store them in persistent memory.
Some are full-blown DBMSs; whereas, others are more often used as caches in front of an RDBMS.
A typical property is a limited query interface: put(), get() for key value pairs.
APIs might be offered to this interface using SOAP or REST.
A Key-Value Store is usually agnostic about the structural properties of the values, so these cannot be used for search criteria.
Also, no relationships or referential integrity constraints or database schema are usually defined over the elements stored in such a database.

Tuple and Document Stores

We now look at our next category of NoSQL database.
A tuple store is a key-value store in which the value is always a vector of data.

For example,

marc -> ("Marc", "McLast Name", 25, "Germmany")

There is no requirement that each tuple stored in a tuple store have the same length.
Various implementations of tuple stores do permit organizing entries into semantic groups often called collections or tables.

For example,

Person:marc -> ("Marc", "McLast Name", 25, "Germany")
Person:harry -> ("Harry", "Smith", 29, "Belgium")

A document store is a key-value store where the value is supposed to be somme kind of semi-structured data such as XML, JSON, JSONB (BSON) - binary JSON, YAML.

For example,

{
    Title = "Harry Potter"
    ISBN = "111-1111111111"
    Authors = [ "J.K. Rowling" ] 
    Price = 32
    Dimensions = "8.5 x 11.0 x 0.5" 
    PageCount = 234
    Genre = "Fantasy"
}

Items with Keys

Since document stores store collections of attributes, it is possible to include an item's key as one of the attributes.
Most NoSQL stores allow you to store items in tables in a schema-less fashion, but do require you to specify a primary key over each table.
This primary key may be specified over each table by the user, for example, you might choose title as the key for a collection of books in DynamoDB.
Or the system might pre-define an always present attributes as a primary key. For example, the _id attribute in MongoDB.
This key is used in the partitioning scheme of the DBMS.
Next day, we will look at some MongoDB examples.

Key-Value Stores, Start Document Stores

Outline

Introduction

Example - Gossiping in Cassandra

Distributing Keys to Machines

Consistent Hashing

In-Class Exercise

Replication and Redundancy

Eventual Consistency

BASE

Stabilization

Integrity Constraints and Querying

Tuple and Document Stores

Items with Keys