Introduction

We are now going to look at inverted indexes in more detail.
So far we have been assuming that the index completely fits in RAM. We are now going to drop that assumption because it is often the case that an inverted index is too large to economically fit into memory.
We have already mentioned that an inverted index contains two main components: dictionary and the posting lists.
Before we talk about disk-based versions of these we briefly mention that another common component of an inverted index is the document map.
A document map contains, for each document in the index, information such as the document's URL, its length, PageRank, and other data.
We now look at these components in the case of a static inverted index. This is an index built for a never-changing text collection.
The life-cycle of such an index is relatively straightforward: (1) Index construction -- process the data in the collection one token at a time and build the postings lists and dictionary (indexing time); (2) Query Processing -- after the index is built handle queries for documents (query time).

HW Problem

Exercise 3.1. Try the following queries on at least two commercial search engines, then judge the top five results. Which are relevant?

For the in class answer, I will indicate what I was expecting for query (a) of (a) through (g). If you had about this level of detail for each query you should get full marks.

to be or not to be.

Results Bing (4/5 relevant):
en.wikipedia.org/wiki/To_be,_or_not_to_be
www.artofeurope.com/shakespeare/sha8.htm
answers.yahoo.com/question/index?qid=...
benefitsbuzzblog.com/2012/10/03/il-essential-health-benefits  (not relevant)
www.imdb.com/title/tt0035446 (Hamlet movie)

Results Blekko (1/5 relevant):
hotornot.com  (not relevant)
nosweatshakespeare.com/quotes/hamlet-to-be-or-n
tlc.discovery.com/fansites/whatnottowear/whatnottowear. (not relevant)
ripleys.com (not relevant)
melhotornot.com (not relevant)

Bing seems to perform better than blekko in this test.

The Dictionary

The dictionary provides a mapping from the set of index terms to the location of their posting lists.
At query time, one of the first operations you do is use the dictionary to find the postings lists of each term in the query.
At indexing time, dictionary lookup is used to quickly find the end of the postings lists of each incoming terms and append them.
A simple ADT for a search engine dictionary might look like:
1. Insert a new entry for term `T`.
2. Find and return the entry for term `T` (if present).
3. Find and return the entries for all terms that start with a given prefix `P`.
Here (1) and (2) often are used during indexing time and (2) and (3) at query time.
(3) is not strictly necessary but allows the engine to support prefix queries. i.e., queries like "inform*" which would match informal, informational, etc.

Dictionary Types

For a typical natural language text collection, the dictionary is relatively small compared to the total index size... In the book's example, an uncompressed index for GOV2 is 0.6% of the total index size, for Shakespeare they get 7% of the total index size.
This makes sense because in a larger collection we will tend to have seen all of the available words. Still, contrary to what you might think the number of distinct terms is not finite, but empirically grows something like `O(\sqrt(n))` where `n` is the number of documents seen.
The dictionary is small enough though that for moderately large indexes it can often still fit in memory.
In which case, the two most common in-memory dictionaries are:
- A sort-based dictionary, in which all terms that appear in the text collection are arranged in a sorted array or in a search tree. Look up operations are realized through tree traversal.
- A hash-based dictionary, in which each index term has a corresponding entry in a hash table. Collisions in the hash table are resolved by means of chaining.

Storing Dictionary Terms

If a sort-based dictionary is being used it is important that all the entries have the same size.
In GOV2 though the longest alpha-numeric sequence is 74,147 bytes long; the average sequence is 9.2 bytes long.
We could imagine truncating or padding all terms to length 20. Then storing sorted 20 bytes terms together with 8 byte integer offsets into the posting lists.
This approach would waste 10.8 bytes on average/term.
Instead, a dictionary-as-a-string approach is often used. Here we have two arrays: a primary sorted array of integer offsets into a secondary array containing the actual terms followed by their posting list offset.
Two adjacent elements in the primary array suffice to tell us the length of a term in the secondary array so we don't even have to store a null at the end of each string.
This scheme saves 10.8 - 4 (for the primary array) = 6.8 bytes over the original scheme.

Sort-based versus Hash-based dictionaries

For most applications a hash-based approach is faster than a sort-based implementation as a binary search and/or tree traversal is avoided.
This is assuming the collision chains are kept relatively small. i.e., Assuming the hash-table grows linearly with the number of terms.
The book gives a table showing that a properly scaled hash-table is about three times faster than a sort-based dictionary.
Unfortunately, the speed advantage is only for single term look-ups. Prefix lookups require linear scans; whereas, if you are using a sort-based approach they can be done via binary search and so done much more efficiently.
For this reason, many engines actually implement both approaches.

Posting Lists

As opposed to dictionaries, posting lists contain the majority of the information stored in an inverted index and tend to be too large to store in memory.
Lists are transferred into memory on an as need basis.
To make the transfer of postings from disk to memory as efficient as possible, each term's posting list should be stored in a contiguous region of the hard-drive.
For single term queries one typically accesses a posting list in a sequential fashion.
On the other hand, for conjunctive queries we want to do things like galloping and binary search.
We will look at posting list operations more on Monday.

Static Inverted Indices