Introduction

On Monday, we were talking inverted index compression techniques.
If we have an increasing sequence of numbers, `x_1 < x_2 < ... < x_n` as typical for schema independent indexes, it can make sense to store the first number, followed by the gaps `x_{i+1} - x_i` for the remainder of the numbers. This is because the gaps tend to be small.
For this to work in for a docid's, it means we cannot use the hash of the url as the docid (random), but need some increasing sequence for docid or some kind of map between hash and sequence.
We looked at nonparametric techniques such as `gamma` codes: store the length of number - 1 in unary O's followed by number in binary.
We looked at global (LLRUN - based on Huffman code) and local parametric (Goloumb and Rice) compression posting list compression techniques.
Finally, we considered byte-aligned (VByte) and word-align codes (Simple-9).
A good, recent survey of inverted index compression techniques can be found in Pibiri and Venturini 2020.
Since we have a lot to cover today, we will leave compression there and now start talking about dynamic inverted indexes.

Dynamic Inverted Indexes

So far we have been assuming that the collection does not change during the process of index creation.
For many applications such as indexing file systems, digital libraries, and the web, the collection does in fact change with time.
A dynamic index needs to support: (1) document insertions, (2) document deletions and (3) document modifications.
Since the semester is winding down and we still have a lot to cover, we will at handle the first two.
Collections that support the first two are sometimes called semi-static.

Batch Updates

For some applications, it might be okay to collect changes to the text collection over a period of time, and then add them in one go in a batch update operation.
This is typically implemented in one of two ways:
- (REBUILD) A new index is built from scratch. When this process is finished, the old index is deleted and replaced with the new one.
- (REMERGE) An index is built from the text found in the newly inserted documents. After this index has been created, it is merged with the previously existing index.
Obviously, REBUILD is easier to implement, but is slower to add new data than REMERGE.
The situation as to which is better in the presence of both insertions and deletions is less clear...

REBUILD versus REMERGE

Suppose the original index `I_(old)` contains postings for `d_(old)` documents and all these documents are about the same size.
Suppose the batch update consists of `d_(\d\e\l\e\te)` deletions and `d_(\i\n\s\e\r\t)` insertions.
The new index, `I_(\n\e\w)` will contain postings for:
`d_(\n\e\w) = d_(old) - d_(\d\e\l\e\te) + d_(\i\n\s\e\r\t)`
documents
The time for REBUILD only depends on `I_(\n\e\w)`. For merge-based index construction, it can build `I_(\n\e\w)` in time
`c cdot d_(\n\e\w) = c cdot (d_(old) - d_(\d\e\l\e\te) + d_(\i\n\s\e\r\t))`
On the other hand, if we do REMERGE instead, the answer depends on the relative performance of indexing and merging in the search engine.
Based on experiments the book's authors carried out with the GOV2 collection, the final merge operation in merge-based index construction takes about 25% of the total time required to build a full-text index.
From this we could estimate the time to REMERGE as:
`c cdot d_(\i\n\s\e\r\t) + frac(c)(4) cdot(d_(old) + d_(\i\n\s\e\r\t))`.
The two methods then exhibit the same performance if:
`d_(\d\e\l\e\te) = frac(3)(4) cdot d_(old) - frac(1)(4) cdot d_(\i\n\s\e\r\t)`.
If there are more deletions than the above it makes sense to REBUILD rather than REMERGE.

Incremental Index Updates

For applications like Internet news search, file system search, e-business search, etc.; it is imperative that the search results always accurately reflect the text collection.
In these cases, batch updates are no longer an appropriate strategy.
We next look at how to do incremental updates in the case where we only add new data and don't perform deletes.
The backbone of the hash-based in-memory index that we discussed earlier this semester, is an in-memory hash-table that maps from term strings to the memory address of the corresponding postings lists
This might be used for both query processing, in addition to index processing.

NO MERGE Index Updates

Suppose that, while the search engine is building an index, say after creating `n` on-disk index partitions, we want it to process a keyword query composed of `m` query terms.
We could repeat the following procedure for each of the query terms:
1. Fetch the terms postings list fragment from each of the `n` posting lists on disk index partitions
2. Use the in-memory hash table to fetch the term's in-memory list fragment.
3. Concatenate all n+1 fragments to form the terms postings list.
This strategy is called the NO MERGE index update strategy.
It tends not to be a very attractive strategy, due to the large number of disk seeks required to process a search query. (one for every query term and index partition).
It is often used as a baseline to which other strategies are compared.

Contiguous Inverted Lists

The most drastic way to guarantee a low degree of fragmentation in the on-disk postings list is to not allow the creation of a new on-disk fragment for a term for which a list fragment already exists in the index.
There are two common ways to achieve this: using an in-place update; using a remerge update strategy.

REMERGE UPDATE

The batch remerge that we discussed earlier can be used not only for batch update operations in the context of semi-static collections, but also as an update policy for incremental collections.
The search engine's indexing module processes incoming documents in the same way as in the static case.
However, whenever the indexer runs out of memory, a fresh on-disk index is created by merging the existing on-disk index with the data accumulated in-memory.

We call this strategy IMMEDIATE MERGE. Below is some pseudo-code:

buildIndex_ImmediateMerge(M)
{
    I[mem] := empty index //init in-memory index
    currentPosting := 1;
    while (there are more tokens to index) {
        T := next token;
        I[mem].addPosting(T, currentPosting);
        currentPosting++;
        if (I[mem] contains more than M-1 tokens) {
            if (I[disk] exists) {
                I[disk] := mergeIndices(I[mem], I[disk]);
            } else {
                I[disk] := I[mem];
            }
            I[mem] = empty index
        }
    }
    return;
}

In terms of the GOV2 collection, if our partition size is 100k documents, then query response time according to the book is about 10 times faster than the NO MERGE case.
However, on the other hand, the total number of token transferred to/from disk with immediate merge is quadratic in the number of tokens in the collections, so is impractical for large indexes. To index GOV2 even with a partition size of 250K is about 6 times slower than NO MERGE.

In-place Index Updates

In-place index update strategies try to overcome IMMEDIATE MERGE's limitations by not requiring the search engine read the entire index from disk every time it runs out of memory.
Instead, whenever a posting list is written to disk during an on-disk update, some free room is left at the end of the list.
When additional postings for the same term have to be transferred to disk at a later point in time, they can simply be appended to the existing list.
If the amount of free space available at the end of the list is too small, then the entire list is relocated to a new position on disk.
Different versions of in-place update, pre-allocate space differently.

Hybrid Index Maintenance

IMMEDIATE MERGE works well when the size of the new postings we are adding is not too much smaller than the entire posting list for the term.
INPLACE works better when the number of disk accesses is kept low.
The book describes a hybrid set-up where we have two indexes a merge index and a inplace index.
Initially, all new terms are placed into the merge index.
When a list size reaches a certain threshold (based on the disk seek time ~ 0.5MB for typical hard drives), it is switched from the merge to the inplace index.
Since most terms never reach this threshold, for the majority of terms we use IMMEDIATE MERGE and this is fast because the posting lists in question are all short.
Since there are only a few longer postings lists, we don't need to perform that many seeks and we achieve the performance benefits of INPLACE for these longer posting lists.
The book shows this approach's time complexity is only slightly non-linear in the number of tokens.

Logarithmic Merging

One drawback with the hybrid approach is that the amount of non-linearity in the run-time is bounded by `Theta(N^(1 + 1/alpha)/M)`, where N is the size of the collection, M is the amount of memory, and `alpha` depends on the Zipf-power of the distribution of the collection.
So the behavior of this algorithm tends to be worse if we don't have much memory.
To get better performance, in lower memory settings, one typically has to settle for non-contiguous indexes of some sort.
One strategy that yields relatively few non-contiguous indexes is to use logarithmic merging...

How Logarithmic Merging Works

To begin, we imagine our index is split into generations `g` which are whole numbers `1, 2 ... `
When we write a partition, the postings go on disk into a generation 1 index.
We then check are there two generation 1 indexes? If yes, we merge them to make a generation 2 index and then delete the generation 1 indexes.
We then repeat the process by checking if we now have two generation 2 indexes. If so, merge them and so on.
At any given time, we will have at most logarithmic in the number of partitions we have written many active generations.
I.e., a small number, so to look up a token we can look it up in each of these indexes and concatenate these posting lists.
The number of postings transferred from/to disk using this strategy is `Theta(N cdot log (N/M))`
In the literature this idea is known as a Log-Structured Merge Tree or LSM-tree. The idea is due to O'Neil, Cheng, Gawlick and O'Neil 1996.

In-Class Exercise

Suppose we can index 200,000 documents before we run out of memory and need to merge with a disk based index that uses logarithmic merging.
We want to index a billion pages, using 4 machines each responsible for having the index for 1/4 of the documents.
What's the largest generation number that will be used (assuming the in memory generation is generation 0)?
What is the most generations that will need to be merged when we have to flush the in memory partition to disk?
Post your answers to the Apr 20 In-Class Exercise Thread.

Finish Compression, Dynamic Inverted Indexes

Outline

Introduction

Dynamic Inverted Indexes

Batch Updates

REBUILD versus REMERGE

Incremental Index Updates

NO MERGE Index Updates

Contiguous Inverted Lists

REMERGE UPDATE

In-place Index Updates

Hybrid Index Maintenance

Logarithmic Merging

How Logarithmic Merging Works

In-Class Exercise