Introduction

At the end of last day, we had an in-class exercise after talking about language modeling.
Recall we are trying to build an information retrieval system that returns documents to a user in roughly descending order of relevance.
Our system's index will store information about the collection (corpus) of documents we've indexed.
A language model is mapping from the terms in the vocabulary `V` of the corpus to probabilities (real numbers between 0 and 1).
We said a common way to estimate the probability of a term such as "banana" is to count the occurrences in the corpus of the term and divide that by the total number of term occurrences in the corpus.
We are going to use language model to come up with ways to rank documents from the most to least relevant for a query.
Today, we start by considering how to handle terms which we have not seem before...

Unknown Shakespeare

We were interested in the problem: Given an unknown play purportedly by Shakespeare to find out if it was really by him.
In the book's example of texts by Shakespeare, fully a third of the terms in this collection occur only once.
So we would expect any new work by Shakespeare to include words not in the rest of the collection.
To get a language model then that can take this into account, we could extend our vocabulary to `V' = V cup {mbox(UNKNOWN)}`.
We can then extend our model to `M'` by setting:
`M'(mbox(UNKNOWN)) = beta`,
`M'(sigma) = M(sigma) cdot (1-beta)`.
where `M` is the maximum likelihood model and `sigma` is any term in `V`.
To guess a good value of `beta` you might set it to half the probability of a unique term in the existing text:
`beta = 0.5 cdot frac(1)(sum_(sigma' in V) mbox(frequency)(sigma'))`
We can make a vector out of a language model by listing coordinates `langle M'(sigma_1), M'(sigma_2), ... M'(sigma_n) rangle` where `sigma_i`'s range over our vocabulary.
We can use language models for many things in IR. For example, we can view a query as a document and compare its vector to those in a corpus to look for close matches.
We can also use vectors for forensics:
- Given a new purported "Shakespeare" document. We can calculate a vector for the maximum likelihood model of just the new document.
- We can compare the Euclidean distance between this vector and the vector from the MLE model of whole Shakespeare corpus and between this vector and that larger corpus of randomly chosen random documents to guess if this unknown text is really Shakespeare.
- We could then calculate a standard deviation between vectors within the corpus, view some fixed number of deviations from our model as a "ball" of Shakespeareness and see if the unknown text lives in that ball.

Higher-order Models

You can imagine using a language model to generate random text by picking a sequence of words according to the probabilities given by the model.
This would tend to produce very gibberish-like text because words like "the" and "our", which are high probability, would have a high chance of occurring in this random text, but almost never occur next to each other in real English.
Higher-order language models which look at sequences of terms can solve the problem above. MLE can be considered a zero'th order model.
A first-order language model consists of conditional probabilities that depend on the previous symbol. For example: `M_1(sigma_2 | sigma_1) = frac(mbox(frequency)(sigma_1 sigma_2))(sum_(sigma' in V)(mbox(frequency)(sigma_1 sigma')))`
A first-order language model for terms is equivalent to the zero-order model for term bi-grams (pairs) estimated using MLE:
`M_1(sigma_2 | sigma_1) = frac(M_0(sigma_1 sigma_2))(sum_(sigma' in V)(M_0(sigma_1 sigma')))`.
An `n`th order model can be expressed in terms of a zero-order `(n+1)`-gram model:
`M_n(sigma_(n+1) | sigma_1 ... sigma_n) = frac(M_0(sigma_1 ... sigma_(n+1)))(sum_(sigma' in V)(M_0(sigma_1 ... sigma_n sigma')))`.

Example

Consider the phrase "first witch" in the Shakespeare corpus.
This occurs, for instance, to identify which of three witches is speaking in Macbeth.
The phrase appears 23 times; whereas, "first" appears 1,349 times. There are a total of 912,051 bi-grams in Shakespeare.
So we have:
`M_0(mbox("first witch")) = frac(23)(912,051) approx 0.0025%`
`M_1(mbox("witch") | mbox("first")) = frac(23)(1349) approx 1.7%`.
On the other hand, the first order model assigns 0 probability to the phrase "our the", because the phrase never appears in the text.

Smoothing

Notice although "our the" does not occur in usual speech, we have been using it a whole bunch in our discussion.
So maybe in the real word the probability of this bi-gram should be small but non-zero.
Other phrases like "fourth witch" seem plausible, but don't appear as a bi-grams in the Shakespeare corpus, so would also be assigned probability `0` in our first-order model.
One way to modify our first-order model to improve it for these cases is to smooth it with the zero-order model.
We define a smoothed first-order `M_1'` as:
`M_1'(sigma_2 | sigma_1) = gamma cdot M_1(sigma_2 | sigma_1) + (1 - gamma) cdot M_0(sigma_2)`
and
`M_0'(sigma_1 sigma_2) = gamma cdot M_0(sigma_1 sigma_2) + (1 - gamma)cdot M_0(sigma_1) cdot M_0(sigma_2)`,
where `0 le gamma le 1` is a smoothing parameter.
As an example, if `gamma = 0.5`, the zero-order, smoothed bi-gram model gives "first witch" a probability now of 0.0013% -- less than before.
This is because it now assigns bi-grams like "fourth witch" a non-zero probability, in this case 0.00000030%.
Often it is the case that we will smooth a small collection from say an author with a larger collection say a big corpus of English text. This might be done with an equation like:
`M_(S,0)' = gamma cdot M_(S,0) + (1 - gamma) cdot M_(L,0)`.

Quiz

Which of the following is true?

Effectiveness of an IR system is measured in terms of time (seconds per query) and space (bytes per document).
The Probability Ranking Principle says if an IR system's response to each query is a ranking of the documents in the collection in order of decreasing probability of relevance, then the overall effectiveness of the system to its users will be maximized.
Zipf's Law says the the `i`th most common word in English occurs with probably `e^{-i}`.

Markov Models

The above picture is an example of a Markov Model, another important method for representing term distributions.
A Markov Model is essentially a finite-state machine augmented with transition probabilities.
When used as a language model, each transition is labeled with a term, in addition to a probability.
Following a transition corresponds to predicting or generating that term.
Starting in state `1`, the string "to be or not to be" could be generated by following the state sequence:
`1 -> 2 -> 3 -> 4 -> 1 -> 2 -> 3`.
The associated probability would be: `0.40 times 0.30 times 0.20 times 0.10 times 0.40 times 0.30 = 0.029%`.
Missing transitions are equivalent to transitions with `0` probability.
The probability predicted depends on the starting state. If we had started with state `4` rather than `1`, the probability of "to be or not to be" would have been: `0.065`.

More Markov Models

An `n`-state Markov model can be represented by an `n times n` transition matrix `M`, where `M[i][j]` gives the probability of transitioning from state `i` to state `j`.
The transition matrix corresponding to the Markov model of the last slide is:
` ((0.00, 0.40, 0.00, 0.60), (0.70, 0.00, 0.30, 0.00), (0.00, 0.80, 0.00, 0.20), (0.10, 0.90, 0.00, 0.00))`
Given a transition matrix we can compute the outcome of a transition by multiplying the transition matrix by a state vector representing the current state. For example, if we start initially in state `(1, 0, 0, 0)`. After one step, we have:
`(1, 0, 0, 0) cdot ((0.00, 0.40, 0.00, 0.60), (0.70, 0.00, 0.30, 0.00), (0.00, 0.80, 0.00, 0.20), (0.10, 0.90, 0.00, 0.00)) = (0.00, 0.40, 0.00, 0.60)`
Multiplying again by the matrix gives us the state after two steps: `(0.00, 0.40, 0.00, 0.60) cdot ((0.00, 0.40, 0.00, 0.60), (0.70, 0.00, 0.30, 0.00), (0.00, 0.80, 0.00, 0.20), (0.10, 0.90, 0.00, 0.00)) = (0.34, 0.54, 0.12, 0.00)`

Test Collections

Researchers have developed many substantial collections for evaluating IR systems.
Many of these collections have been created as part of TREC, a series of evaluation efforts conducted annually since 1991 by the U.S. National Institute of Standards and Technology (NIST).
TREC provides a forum for researchers to test their IR systems on a broad range of problems. For example, more than 100 groups from universities, industry, and government participated in TREC 2007.
Each year TREC experiments are structured into six or seven tracks, each devoted to a different area of information retrieval.
For example, enterprise search, genomic IR, legal discovery, e-mail spam filtering, blog search, etc.
TREC focuses researchers on particular common problems and it provides a set of reusable tests that researchers can conduct experiments with.
Europe has two similar conferences INEX for XML and CLEF for multi-lingual support; Japan has NTCIR for Asian language IR; and India has something called FIRE.

TREC Tasks

Basic search tasks, in which the IR system returns a ranked list from a static set of documents, are called adhoc tasks in TREC.
Along with a set of documents, a test collection for an adhoc task consists of a set of relevance judgments (known as "qrels files" or "qrels"), indicating documents that are relevant or not to the given topic.

Older TREC tasks, before 2000, were often taken from things like newspaper articles. For example, you might have a collection of documents like:

<DOC>
<DOCNO>LA051990-0141<DOCNO>
<HEADLINE>COUNCIL VOTES TO EDUCATE DOG OWNERS</HEADLINE>
<P>
The City Ccouncil stepped carefully around enforcement of the dog-curbing 
ordinance this week, vetoing the use of police to enforce the law.
</P>
...
</DOC>

Each document is surrounded by DOC tags and DOCNO serves as a unique identifier for the document. This makes it easy to combine collections.
For newer collection adhoc tasks are often taken from the web.
Until 2009, GOV2 which consisted of a collection of 25,000,000 U.S. Government web pages from early 2004 was the largest collection used in TREC. It took about 426GB of storage.
From 2009-2022, the ClueWeb09 data set is the largest used by TREC.
It consists of an open web crawl of 1 billion web pages and take 5TB of compressed disk space, 25 TB of uncompressed space.
There was also the slightly smaller, but less spammy ClueWeb12 data set.
Since 2022, various Clueweb 22 datasets have been used, the largest being a crawl of 10 billion pages based on Bing results.
In a given year each track consist of about 50 new topics. Participants are required to freeze their systems before downloading the topics. They then can create queries from the topics, run these queries against the document set, and return ranked lists of results for NIST evaluation.

Open-Source IR Systems

We next look at some open-source IR systems.
Three well-known one (plus one I and my master's students wrote) are:
- Lucene - this is the indexing library associated with Nutch/Lucence/Solr. It is written in Java and was developed by Doug Cutting beginning in 1997. It is a project now of the Apache Foundation. It is used by Wikipedia.
- Indri - this project was developed ar UMass and is part of the Lemur project of UMass and CMU. Indri is written in C++. It, like Lucene, can handle multiple field per document, such as title, body, anchor text. It supports automatic query expansion by pseudo-relevance feedback. It also uses techniques that prefer more recent documents over less recent documents when scoring search results.
- Wumpus - is an Academic search engine developed at the U Waterloo by one of the book's authors. It is written in C++. Results from it are used throughout the book.
- Yioop! -- an open-source search engine developed by me and students here at SJSU. It is written in PHP.
Next Monday I will go over how to set-up Yioop!, how to crawl with it, and a little about how it is coded.

Inverted Indices

diagram with dictionary and posting list of an inverted index

An inverted index provides a mapping between terms and their locations in a text collection `C`.
The two main components of such an index are: (1) a dictionary which lists terms contained in the vocabulary of the collection and (2) for each term in the dictionary, a posting list which gives the positions in the collection at which the term occurs.
The diagram above is an example of a schema-independent index because it makes no assumption about the structure of the underlying text. In particular, it gives a raw position number rather than assume the collection is split into documents and further positions within documents.
An inverted index can be viewed as an abstract data type with the following four methods:
- first(t) returns the first position at which the term `t` occurs in the collection.
- last(t) returns the last position at which the term `t` occurs in the collection.
- next(t, current) returns the position of the first occurrence of `t` after the position current in the collection.
- prev(t, current) returns the position of the last occurrence of `t` before the position current in the collection.
For example, in the above diagram: first("hurlyburly") = 316669, last("thunder") = 1247139, next("witch", 745451)=745467, prev("witch", 745451) = 745429.
A sequential scan of a posting list might be implemented by applying the first function on the term then repeatedly applying the next function till the end of the posting list.

ADT Example: Phrase Search

As an example of using the primitives of our ADT, consider the problem of phrase search.
A phrase search is a search for an exact match of a phrase in a document. For example when we search on "first witch" in quotes we want back only those documents that have the phrase "first witch" not documents which have both terms but not adjacent and in the given order.

This could be implemented using our ADT with the following pseudo-code:

nextPhrase(t[1],t[2], .., t[n], position)
{
   v:=position
   for i = 1 to n do
     v:= next(t[i], v)
   if v == infty then // infty represents after the end of the posting list
      return [infty, infty]
   u := v
   for i := n-1 downto 1 do
     u := prev(t[i],u)
   if(v-u == n - 1) then
     return [u, v]
   else 
     return nextPhrase(t[1],t[2], .., t[n], u) 
}

Language Modeling, Test Collections, Open-Source IR Systems, Inverted Indexes

Outline