Outline

• Language Models
• Quiz
• LMJM and LMD
• Kullback-Leibler Divergence
• Start Divergence From Randomness

Introduction

• Last week, we began to exploring the language modeling approach to defining relevance measures.
• Our goal was to derive the divergence-from-randomness relevance measure.
• Today, we will continue to work towards this goal, deriving two relevance measures on our way and one technique for pseudo-relevance feedback. On Wednesday, we will finish our derivation of DFR.
• We started off the with the probability ranking principle and asked the basic question: How do we estimate whether a user will judge a document relevant given the document and the query?
• We reduced this to the question of ranking according to the probability `p(Q|D)`.
• That is, we want to estimate given a document `d` how likely it was someone would enter the query `q` to retrieve that document -- the document is thus treated as providing a model for generating queries.

Languages Models

• The simplest document language model is the maximum likelihood model:
  `M_d^(ml)(t) = f_(t,d)/l_d`
  where `f_(t,d)` is the frequency of the term in the document and `l_d` is the documents length.
• For example,
  `M_(mbox(Hamlet))^(ml)(\l\o\r\d) = 624/43314 = 1.41%`
  `M_(mbox(Macbeth))^(ml)(\l\o\r\d) = 78/26807 = 0.291%`
• One drawback for using this as a model of `p(q|d)` is that `d` is just a single example of a relevant document and may not consist of many words, so it might give a poor estimate.
• It also assigns `0` probability to all terms not appearing in the document, even though it is possible for a person to making a query involving terms not in say Hamlet.

Smoothing

• People usually smooth the maximal likelihood model with a model generated from the collection as a whole:
  `M_C(t) = l_t/l_C`
  Here `l_t` is the number of occurrence of `t` in the whole corpus.
• Two common smoothing techniques are Jelinek Mercer smoothing (linear combination):
  `M_d^lambda = (1 - lambda) cdot M_d^(ml)(t) + lambda cdot M_C(t)`
  and Dirichlet smoothing (imagine increasing the length of `d` by a factor `mu` according to the underlying corpus):
  `M_d^mu(t) = frac(f_(t,d) + mu cdot M_C(t))(l_d + mu)`

Quiz

Which of the following is true?

1. LLRUN is an example local parametric gap compression technique.
2. If one does batch updates and has more delete operations than `frac(3)(4) cdot d_(old) - frac(1)(4) cdot d_(\i\n\s\e\r\t)`, it will be quicker to REBUILD the index than use REMERGE.
3. Pseudo relevance feedback replaces each term in a user's query with a term that is likely to give better search results.

Ranking with Language Models

• Our basic model deals with single terms.
• We could calculate:
  `p(q|d) = p(|q| = n) cdot prod_(i=1)^n p(t_i |d)`.
• People though usually ignore the query length. However, they do allow the same term to appear multiple times in the query, so we get:
  `p(q|d) = prod_(i=1)^n p(t_i |d) = prod_(t in q) p(t |d)^(q_t)`
  where `q_t` is the number of appearances of `t` in the query.
• Rewriting this with one of the language models on the last slide we have either:
  `p(q | d) = prod_(t in q)( (1 - lambda) cdot M_d^(ml)(t) + lambda cdot M_C(t))^(q_t)`
  or
  `p(q | d) = prod_(t in q)(frac(f_(t,d) + mu cdot M_C(t))(l_d + mu))^(q_t)`

Massaging our equations

• Let's take the log of our probability and split our equations into the terms that are in the document and those that are not:
  `log p(q|d) = log prod_(t in q) p(t |d)^(q_t) = sum_(t in q cap d) q_t cdot log p(t|d) + sum_(t in q\\d) q_t cdot log p(t|d)`
• Let `M_d(t)` be either of our models, and suppose we use it to estimate `p(t|d)`. When `t` is not in our document this takes the form `alpha_d M_C(t)`, where `alpha_d` is either `lambda` or `mu/(l_d + mu)`.
• If we substitute this back into our log equation above, then do a half page of grundge, we get:
  `log p(q|d) = sum_(t in q cap d) q_t log frac(M_d(t))(alpha_d M_C(t)) + nlog alpha_d + sum_(t in q) q_t cdot log M_C(t)`.
• The last term is constant for all documents, so can be dropped without affecting rankings and we get:
  `[sum_(t in q cap d) q_t log frac(M_d(t))(alpha_d M_C(t))] + nlog alpha_d `

Substituting in a particular model

• If we substitute in the Jelinek Mercer smoothing model we get:
  `sum_(t in q cap d) q_t cdot log frac((1-lambda)f_(t,d)/l_d + lambda l_t/l_C)(lambda l_t/l_C) + n log lambda`
  as the last term is constant, it can be dropped giving the final equation (after some rearranging):
  `sum_(t in q) q_t log (1 + (1 - lambda)/lambda cdot f_(t,d)/l_d cdot l_C/l_t)`.
• We could then rank documents according to this equation. This is called language modeling with Jelinek-Mercer smoothing (LMJM). People often take `lambda =0.5` in this equation.
• For the Dirichlet Model a similar substitution gives the following final equation: `[sum_(t in q) q_t log(1 + f_(t,d)/mu cdot l_C/l_t)] - n cdot log(1 + l_d/mu)`
  which is called language modeling with Dirichlet Smoothing (LMD).
• A value of `mu = 1000` is often used.

Kullback-Leibler Divergence

• Kullback-Leibler Divergence is another approach to language modeling based on the relative entropy of the language model determined by the query and that determined by the document.
• For two discrete distributions `f` and `g` the KL divergence is given by the equation:
  `sum_x f(x) cdot log frac(f(x))(g(x))`
  The value of this equation is not necessarily symmetric in `f` and `g`.
• Just as we had a language model for the document, we can imagine making a language model for the query. For example, the max likelihood model would be:
  `M_q^(ml)(t) = q_t/n.`
• Substituting in models into the KL divergence gives:
  `sum_(t in V) M_q(t) cdot log frac(M_q(t))(M_d(t)) = sum_(t in V) M_q(t) cdot log M_q(t) - sum_(t in V) M_q(t) cdot log M_d(t)`.
  The first summation is the same for all docs. The second summation without the negative sign increases with decreasing divergence so can be used as a ranking formulas. If we substitute in our max likelihood model, we thus arrive at:
  `sum_(t in V) M_q^(ml)(t) cdot log M_d(t) = 1/n sum_(t in q) q_t cdot log M_d(t)`.
• If we drop the `1/n` as not effecting the ranking this actually reduces to being the log of our starting point a couple of slides back (i.e., the log of the following after substituting the model):
  `p(q|d) = prod_(t in q) p(t |d)^(q_t)`
• People often don't use `M_q^(ml)(t)` as the model of the query, but instead do query expansion first, then continue with the derivation we just did.
• Lafferty and Zhai (2001) do this where they expand the query using a random walk: Pick a random query term according to the query model, then pick randomly doc with the query term, select using this doc's model a term from the document, add the term to the query. At random choose whether to stop or continue.
• Lafferty and Zhai report this method performs better than the non-query expanded model we have presented.

The Basic DFR Formula

• The next approach we are going to consider, divergerce-from-randomness (DFR) explicitly assumes a random process for the distribution of terms in documents, and then ranks documents by considering the probability that the actual term distribution found in a document would occur by chance.
• Unlike BM25 with `k_1` and `b`, LMJM with `lambda`, and LMD with `mu`, DFR does not have arbitrary parameter values that need to be set.
• The basic starting formula for DFR is the following:
  `(1 - P_2)cdot(-log P_1)`.
• `P_1` represents the probability that a random document `d` contains exactly `f_(t,d)` occurrences of `t`.
• The value `- log P_1` may be viewed as the number of bits of information (self information) associated with the knowledge that `d` has exactly `f_(t,d)` occurrences of `t`.
• `P_1` decreases rapidly as `f_(t,d)` increases and so `-log P_1` increases rapidly in `f_(t,d)`.
• The second factor, `P_2` is related to eliteness and compensates for this rapid change. A document is said to be elite for the term `t` when it "about" the topic associated with the term.
• If `d` is elite in `t`, we might assume that the occurrences of `t` that appear within it are not accidental. If we start reading `d` and `d` is elite in `t`, we might well before the end discover `f_(t,d) - 1` occurrences of `t`. We might given this expect to find more occurrences of `t` in `d`. `P_2` represents the probability of finding at least one.
• `P_2` increases as `f_(t,d)` increases. Thus `(1-P_2)` decreases as `f_(t,d)` increases.

More on the Basic DFR Formula

• DFR was first developed by Amati and Rijsbergen (2002). It has performed well in various TREC conferences since.
• To determine the relevance of a document to a query with this model we calculate: `sum_(t in q) q_t(1 - P_(2,t)) cdot(-log P_(1,t))` where `P_(1,t)` and `P_(2,t)` are the `P_1` and `P_2` associated with the particular term `t`. Here `q_t` is the number of occurrences of the term `t` in the query.
• This of course leaves the problem of how to estimate `P_1` and `P_2`.
• Amati and Rijsbergen in their paper presented seven different methods for doing this. They also presented different methods of incorporating document length normalization into DFR.
• Next day, we will now look at one way of estimating each.