Language Modeling -- Remembering Monday

On Monday, we defined a language model to be a function from a vocabulary of terms to probabilities. i.e., a distribution on terms.
We gave:
`M(sigma) = frac{mbox{frequency}(sigma)}{sum_{sigma' in V}mbox{frequency}(sigma')}`
as an example language model.
This model is called the maximum likelihood model. We showed how to extend probabilities of terms to probabilities of phrases in this model.

Unknown Shakespeare

We were interested in the problem: Given an unknown play purportedly by Shakespeare to find out if it was really by him.
As we said on Monday, in the book's example of texts by Shakespeare, fully a third of the terms in this collection occur only once.
So we would expect any new work by Shakespeare to include words not in the rest of the collection.
To get a language model then that can take this into account, we could extend our vocabulary to `V' = V cup {mbox(UNKNOWN)}`.
We can then extend our model to `M'` by setting:
`M'(mbox(UNKNOWN)) = beta`,
`M'(sigma) = M(sigma) cdot (1-beta)`.
where `M` is the maximum likelihood model and `sigma` is any term in `V`.
To guess a good value of `beta` you might set it to half the probability of a unique term in the existing text:
`beta = 0.5 cdot frac(1)(sum_(sigma' in V) mbox(frequency)(sigma'))
We can make a vector out of a language model by listing coordinates `langle M'(sigma_1), M'(sigma_2), ... M'(sigma_n) rangle` where `sigma_i`'s range over our vocabulary.
We then calculate a vector for the maximum likelihood model of just the new document.
We can compare the Euclidean distance between these two vectors and compare this to that of the distance between our Shakespeare model and that of a fixed set of random documents to guess if this unknown text is really Shakespeare.
Alternatively, we could calculate a standard deviation between vectors within the corpus, view some fixed number of deviations from our model as a "ball" of Shakespeareness and see if the unknown text lives in that ball.

Higher-order Models

You can imagine using a language model to generate random text by picking a sequence of words according to the probabilities given by the model.
This would tend to produce very gibberish-like text because words like "the" and "our", which are high probability, would have a high chance of occurring in this random text, but almost never occur next to each other in real English.
Higher-order language models which look at sequences of terms can solve the problem above. MLE can be considered a zero'th order model.
A first-order language model consists of conditional probabilities that depend on the previous symbol. For example: `M_1(sigma_2 | sigma_1) = frac(mbox(frequency)(sigma_1 sigma_2))(sum_(sigma' in V)(mbox(frequency)(sigma_1 sigma')))`
A first-order language model for terms is equivalent to the zero-order model for term bi-grams (pairs) estimated using MLE:
`M_1(sigma_2 | sigma_1) = frac(M_0(sigma_1 sigma_2))(sum_(sigma' in V)(M_0(sigma_1 sigma')))`.
An `n`th order model can be expressed in terms of a zero-order `(n+1)`-gram model:
`M_1(sigma_(n+1) | sigma_1 ... sigma_n) = frac(M_0(sigma_1 ... sigma_(n+1)))(sum_(sigma' in V)(M_0(sigma_1 ... sigma_n sigma')))`.

Example

Consider the phrase "first witch" in the Shakespeare corpus.
This occurs, for instance, to identify which of three witches is speaking in Macbeth.
The phrase appears 23 times; whereas, "first" appears 1,349 times. There are a total of 912,051 bi-grams in Shakespeare.
So we have:
`M_0(mbox("first witch")) = frac(23)(912,051) approx 0.0025%`
`M_1(mbox("witch") | mbox("first")) = frac(23)(1349) approx 1.7%`.
On the other hand, the first order model assigns 0 probability to the phrase "our the", because the phrase never appears in the text.

Smoothing

Notice although "our the" does not occur in usual speech, we have been using it a whole bunch in our discussion.
So maybe in the real word the probability of this bi-gram should be small but non-zero.
Other phrases like "fourth witch" seem plausible but as they don't appear as a bi-grams in the Shakespeare corpus would be assigned probability `0` in our first-order model.
One way to modify our first-order model to improve it for these cases is to smooth it with the zero-order model.
We define a smoothed first-order `M_1'` as:
`M_1'(sigma_2 | sigma_1) = gamma cdot M_1(sigma_2 | sigma_1) + (1 - gamma) cdot M_0(sigma_2)`
and
`M_0'(sigma_1 sigma_2) = gamma cdot M_0(sigma_1 sigma_2) + (1 - gamma)cdot M_0(sigma_1) cdot M_0(sigma_2)`,
where `0 le gamma le 1` is a smoothing parameter.
As an example, if `gamma = 0.5`, the zero-order, smoothed bi-gram model give "first witch" a probability now of 0.0013%, less than before.
This is because it now assigns bi-grams like "fourth witch" a non-zero probability, in this case 0.00000030%.
Other it is the case that we will smooth a small collection from say an author with a larger collection say a big corpus of English text. This might be done with an equation like:
`M_(S,0)' = gamma cdot M_(S,0) + (1 - gamma) cdot M_(L,0)`.

Markov Models

The above picture is an example of a Markov Model, another important method for representing term distributions.
A Markov Model is essentially a finite-state machine augmented with transition probabilities.
When used as a language model, each transition is labeled with a term, in addition to a probability.
Following a transition corresponds to predicting or generating that term.
Starting in state `1`, the string "to be or not to be" could be generated by following the state sequence:
`1 -> 2 -> 3 -> 4 -> 1 -> 2 -> 3`.
The associated probability would be: `0.40 times 0.30 times 0.20 times 0.10 times 0.40 times 0.30 = 0.029%`.
Missing transitions are equivalent to transitions with `0` probability.
The probability predicted depends on the starting state. If we had started with state `4` rather than `1`, the probability of "to be or not to be" would have been: `0.065`.

More Markov Models

An `n`-state Markov model can be represented by an `n times n` transition matrix `M`, where `M[i][j]` gives the probability of transitioning from state `i` to state `j`.
The transition matrix corresponding to the Markov model of the last slide is:
` ((0.00, 0.40, 0.00, 0.60), (0.70, 0.00, 0.30, 0.00), (0.00, 0.80, 0.00, 0.20), (0.10, 0.90, 0.00, 0.00))`
Given a transition matrix we can computer the outcome of a transition by multiplying the transition matrix by a state vector representing the current state. For example, if we start initially in state `(1, 0, 0, 0)`. After one step, we have:
`(1, 0, 0, 0) cdot ((0.00, 0.40, 0.00, 0.60), (0.70, 0.00, 0.30, 0.00), (0.00, 0.80, 0.30, 0.20), (0.10, 0.90, 0.00, 0.00)) = (0.00, 0.40, 0.00, 0.60)`
Multiplying again by the matrix gives us the state after two steps: `(0.00, 0.40, 0.00, 0.60) cdot ((0.00, 0.40, 0.00, 0.60), (0.70, 0.00, 0.30, 0.00), (0.00, 0.80, 0.30, 0.20), (0.10, 0.90, 0.00, 0.00)) = (0.34, 0.54, 0.12, 0.00)`

Test Collections

Researchers have developed many substantial collections for evaluating IR systems.
Many of these collections have been created as part of TREC, a series of evaluation efforts conducted annually since 1991 by the U.S. National Institute of Standards and Technology (NIST).
TREC provides a forum for researchers to test their IR systems on a broad range of problems. For example, more than 100 groups from universities, industry, and government participated in TREC 2007.
Each year TREC experiments are structured into six or seven tracks, each devoted to a different area of information retrieval.
For example, enterprise search, genomic IR, legal discovery, e-mail spam filtering, blog search, etc.
TREC focuses researchers on particular common problems and it provides a set of reusable tests that researchers can conduct experiments with.
Europe has two similar conferences INEX for XML and CLEF for multi-lingual support; Japan has NTCIR for Asian language IR; and India has something called FIRE.

TREC Tasks

Basic search tasks, in which the IR system returns a ranked list from a static set of documents, are called adhoc tasks in TREC.
Along with a set of documents, a test collection for an adhoc task consists of a set of relevance judgments (known as "qrels files" or "qrels"), indicating documents that are relevant or not to the given topic.

Older TREC tasks, before 2000, were often taken from things like newspaper articles. For example, you might have a collection of documents like:

<DOC>
<DOCNO>LA051990-0141<DOCNO>
<HEADLINE>COUNCIL VOTES TO EDUCATE DOG OWNERS</HEADLINE>
<P>
The City Ccouncil stepped carefully around enforcement of the dog-curbing 
ordinance this week, vetoing the use of police to enforce the law.
</P>
...
</DOC>

Each document is surrounded by DOC tags and DOCNO serves as a unique identifier for the document. This makes it easy to combine collections.
For newer collection adhoc tasks are often taken from the web.
Until 2009, GOV2 which consisted of a collection of 25,000,000 U.S. Government web pages from early 2004 was the largest collection used in TREC. It took about 426GB of storage.
Since 2009, the ClueWeb09 data set is the largest used by TREC.
It consists of an open web crawl of 1 billion web pages and take 5TB of compressed disk space, 25 TB of uncompressed space.
In agiven year each track consist of about 50 new topics. Participants are require to freeze then systems before downloading the topics. They then can create queries from the topics, run these queries against the document set, are return ranked lists of results for NIST evaluation.

Open-Source IR Systems

We next look at some open-source IR systems.
Three well-known one are:
- Lucene - this is the indexing library associated with Nutch/Lucence/Solr. It is written in Java and was developed by Doug Cutting beginning in 1997. It is a project now of the Apache Foundation. It is used by Wikipedia. We will look at installing and using it on Monday.
- Indri - this project was developed ar UMass and is part of the Lemur project of UMass and CMU. Indri is written in C++. It, like Lucene, can handle multiple field per document, such as title, body, anchor text. It supports automatic query expansion by pseudo-relevance feedback. It also uses techniques prefer more recent documents over less recent documents when scoring search results.
- Wumpus - is an Academic search engine developed at the U Waterloo by one of the book's authors. It is written in C++. Results from it are used throughout the book.
For the rest of today, I am going to demo how to download and configure Yioop!, another open-source search engine developed by me with some input by students here at SJSU. It is written in PHP.

Language Modeling, Test Collections, Open-Source IR Systems

Outline

Language Modeling -- Remembering Monday

Unknown Shakespeare

Higher-order Models

Example

Smoothing

Markov Models

More Markov Models

Test Collections

TREC Tasks

Open-Source IR Systems