Outline

• Language Models
• HW Problem
• KL Divergence

Introduction

• On Monday, we had begun to explore the language modeling approach to coming up with relevance measures.
• One of our goals was to explain the divergence-from-randomness relevance measure.
• We started off the with the probability ranking principle and asked the basic question: how do we estimate whether a user will judge a document relevant given the document and the query.
• We reduced this to the question of ranking according to the probability p(Q|D).
• That is, we want to estimate given a document `d` how likely it was someone would enter the query `q` to retrieve that document -- the document is thus treated as providing a model for generating queries.

Languages Models

• The simplest document language model is the maximum likelihood model:
  `M_d^(ml)(t) = f_(t,d)/l_d`
  where `f_(t,d)` is the frequency of the term in the document and `l_d` is the documents length.
• For example,
  `M_(mbox(Hamlet))^(ml)(\l\o\r\d) = 624/43314 = 1.41%`
  `M_(mbox(Macbeth))^(ml)(\l\o\r\d) = 78/26807 = 0.291%`
• One drawback for using this as a model of `p(q|d)` is that `d` is just a single example of a relevant document and may not consist of many words, so it might give a poor estimate.
• It also assigns `0` probability to all terms not appearing in the document, even though it is possible for a person to making a query involving terms not in say Hamlet.

Smoothing

• People usually smooth the maximal likelihood model with a model generated from the collection as a whole:
  `M_C(t) = l_t/l_C`
  Here `l_t` is the number of occurrence of `t` in the whole corpus.
• Two common smoothing techniques are Jelinek Mercer smoothing (linear combination):
  `M_d^lambda = (1 - lambda) cdot M_d^(ml)(t) + lambda cdot M_C(t)`
  and Dirichlet smoothing (imagine increasing the length of `d` by a factor `mu` according to the underlying corpus):
  `M_d^mu(t) = frac(f_(t,d) + mu cdot M_C(t))(l_d + mu)`

HW Problem

Exercise 7.2. Suppose the search engine employs the INPLACE strategy with proportional pre-allocation (factor `k=2`). Prove that, for any given list, the total number of bytes transferred from/to disk is less than `5 times s`, where `s` is the list's size in bytes. You may assume that all postings have constant size.

Answer. Consider where we are going from a list of size `b` to a list of `k cdot b`. For us to have a list of size `b` we must have written `b` bytes to disk. In order to prepare the list of size `k cdot b` we must read these `b` bytes back in. If `M` was the original default posting list length. Then the total read/write operations to get a list of `s` bytes would be:

`2M + 2cdot 2 cdot M + 2 cdot 2^2 cdot M + cdots + 2 cdot 2^(log(s/M)) cdot M`
`= 2M(2^(log( frac(s)(M) ) + 1) -1) = 2M(2cdot(s/M) - 1) < 2M cdot 2cdot(s/M) < 5s`

Here we are using the formula for the geometric series and we are bounding by 5 rather than 4 to suck up any sloppiness factor caused by not rounding up on the logs and division operations.

Ranking with Language Models

• Our basic model deals with single terms.
• We could calculate:
  `p(q|d) = p(|q| = n) cdot prod_(i=1)^n p(t_i |d)`.
• People though usually ignore the query length. However, they do allow the same term to appear multiple times in the query, so we get:
  `p(q|d) = prod_(i=1)^n p(t_i |d) = prod_(t in q) p(t |d)^(q_t)`
  where `q_t` is the number of appearances of `t` in the query.
• Rewriting this with one of the language models on the last slide we have either:
  `p(q | d) = prod_(t in q)( (1 - lambda) cdot M_d^(ml)(t) + lambda cdot M_C(t))^(q_t)`
  or
  `p(q | d) = prod_(t in q)(frac(f_(t,d) + mu cdot M_C(t))(l_d + mu))^(q_t)`

Massaging our equations

• Let's take the log of our probability and split our equations into the terms that are in the document and those that are not:
  `log p(q|d) = log prod_(t in q) p(t |d)^(q_t) = sum_(t in q cap d) q_t cdot log p(t|d) + sum_(t in q\\d) q_t cdot log p(t|d)`
• Let `M_d(t)` be either of our models, and suppose we use it to estimate `p(t|d)`. When `t` is not in our document this takes the form `alpha_d M_C(t)`, where `alpha_d` is either `lambda` or `mu/(l_d + mu)`.
• If we substitute this back into our log equation above, then do a half page of grundge, we get:
  `log p(q|d) = sum_(t in q cap d) q_t log frac(M_d(t))(alpha_d M_C(t)) + nlog alpha_d + sum_(t in q\\d) q_t cdot log M_C(t)`.
• The last term is constant for all documents, so can be dropped without affecting rankings and we get:
  `sum_(t in q cap d) q_t log frac(M_d(t))(alpha_d M_C(t)) + nlog alpha_d `

Substituting in a particular model

• If we substitute in the Jelinek Mercer smoothing model we get:
  `sum_(t in q cap d) q_t cdot log frac((1-lambda)f_(t,d)/l_d + lambda l_t/l_C)(lambda l_t/l_C) + n log lambda`
  as the last term is constant, it can be dropped giving the final equation (after some rearranging):
  `sum_(t in q) q_t log (1 + (1 - lambda)/lambda cdot f_(t,d)/l_d cdot l_C/l_t)`.
• We could then rank documents according to this equation. This is called language modeling with Jelinek-Mercer smoothing (LMJM). People often take `lambda =0.5` in this equation.
• For the Dirichlet Model a similar substitution gives the following final equation: `sum_(t in q) q_t log(1 + f_(t,d)/mu cdot l_C/l_t) - n cdot log(1 + l_d/mu)`
  which is called language modeling with Dirichlet Smoothing (LMD).
• A value of `mu = 1000` is often used.

Kullback-Leibler Divergence

• Kullback-Leibler Divergence is another approach to language modeling based on the relative entropy of the language model determined by the query and that determined by the document.
• For two discrete distributions `f` and `g` the KL divergence is given by the equation:
  `sum_x f(x) cdot log frac(f(x))(g(x))`
  The value of this equation is not necessarily symmetric in `f` and `g`.
• Just as we had a language model for the document, we can imagine making a language model for the query. For example, the max likelihood model would be:
  `M_q^(ml)(t) = q_t/n.`
• Substituting in models into the KL divergence gives:
  `sum_(t in V) M_q(t) cdot log frac(M_q(t))(M_d(t)) = sum_(t in V) M_q(t) cdot log M_q(t) - sum_(t in V) M_q(t) cdot log M_d(t)`.
  The first summmation is the same for all docs. The second summation without the negative sign increases with decreasing divergerce so can be used as a ranking formulas. If we substitute in our max likelihood model, we thus arrive at:
  `sum_(t in V) M_q^(ml)(t) cdot log M_d(t) = 1/n sum_(t in q) q_t cdot log M_d(t)`.
• If we drop the `1/n` as not effecting the ranking this actually reduces to being the log of our starting point a couple of slides back (i.e., the log of the following after substituting the model):
  `p(q|d) = prod_(t in q) p(t |d)^(q_t)`
• People often don't use `M_q^(ml)(t)` as the model of the query, but instead do query expansion first, then continue with the derivation we just did.
• Lafferty and Zhai (2001) do this where they expand the query using a random walk: Pick a random query term according to the query model, then pick randomly doc with the query term, select using this doc's model a term from the document, add the term to the query. At random choose whether to stop or continue.
• Lafferty and Zhai report this method performs better than the non-query expanded model we have presented.