Outline

• Effectiveness Measures
• Building a Test Collection
• Efficiency Measures
• Quiz
• Punctuation and Capitalization
• Stemming

Measuring Retrieval Effectiveness

• Measuring the effectiveness of a retrieval method depends on human assessments of relevance.
• TREC experiments often use binary relevance judgements by humans. i.e., someone says, document X is or is not relevant to the topic A.
• A document is typically judged relevant if any part of it is relevant.
• For example, for TREC topic 426, a user might formulate the Boolean query:
  (("law" AND "enforcement") OR "police") AND ("dog" OR "dogs").
• Using the TREC45, (disk 4 and disk 5 of a TREC dataset of newspaper articles), this query produces 881 documents of the 500,000 or so in the collection.

Recall and Precision

• In order to determine the effectiveness of a Boolean Query, we compare (1) the set of documents returned by the query, Res, and (2) the set of relevant documents for the topic contained in the collection, Rel.
• From these two sets we can compute two common standard measures of effectiveness, recall and precision.
  `mbox(recall) = frac(|Rel cap Res|)(|Rel|)`
  `mbox(precision) = frac(|Rel cap Res|)(|Res|)`
• Recall indicates the fraction of relevant documents that appear in the result set, precision indicates the fraction of the result set that is relevant.
• According to NIST there are `202` relevant documents for topic 426. The above Boolean query only returns `167` of these. So the precision is `0.190`, and the recall is `0.827`.

F-measures

• Sometimes recall and precision are combined into one number known as an F-measure.
• The simplest F-measure, the `F_1`-measure, takes the harmonic mean of recall and precision: `F_1 = frac(2)(frac(1)(R) + frac(1)(P)) = frac(2 cdot R cdot P)(R+P)`.
  
• Unlike, the arithmetic mean `frac(R+P)(2)`, the harmonic mean enforces a balance between precision and recall.
• For example, we return the whole data set in response to a query.
• The recall will be 1 and for a large data set the precision will be close to 0, so the arithmetic mean will be about 0.5.
• In constrast, the harmonic mean gives a value of `frac(2P)(1+P)` which will be close `0`.
• A common generalization of the `F_1`-measure is the weighted measure:
  `F_(beta) = frac((beta^2 + 1) cdot R cdot P)(beta^2 cdot R + P)`
• Notice if `beta = 1` this corresponds to the harmonic mean, if `beta = 0` one gets just the recall, and if `beta = infty` one gets just the precision.

Measures for the first `k` results

• For an ordered set `X`, we use `X[1..k]` to denote its first `k` elements.
• Often for search, we are interested in the first 10, 20, 50, or 100 results.
• We can modify precision and recall to the first `k` results as:
  `mbox(recall@k) = frac(|Rel cap Res[1..k]|)(|Rel|)`
  `mbox(precision@k) = frac(|Rel cap Res[1..k]|)(|Res[1..k]|)`
• We write `mbox(P@k)` for `mbox(precision@k)`.
• By definition, `mbox(recall@k)` will increase with `k`. On the other hand, `mbox(P@k)` will tend to decrease with `k`.
• An IR experiment typically looks at `k` over a range of values to try to find out how `mbox(recall@k)` and `mbox(P@k)` trade-off against each other. One may then make a recall-precision plot.
• In such a plot, one puts the on x-axis, the recall values between 0 and 100%. Given such a recall value x, we look for the maximum precision achieved at this recall value or higher (this is called interpolated precision).

Average Precision

• It is also possible to compute an average precision across the full range of recall values:
  `frac(1)(|Rel|)cdot sum_(i=1)^(k) (mbox(relevant)(i) times mbox(P@i))`
  Here `mbox(relevant)(i)` is 1 if the `i`th result is relevant and `0` otherwise.
• This roughly represents the area under a non-interpolated recall-precision curve.
• Choosing `k` larger than the index of the last relevant result does not change the value of average precision.
• To get one number for how the IR system performs across a range of topics, one typically takes the arithmetic mean of the average precision of the system for each topic.
• This number is called the mean average precision or MAP.
• The most common measures to give when trying to present the effectiveness of your IR system are to give its average `mbox(P@10)` across topics and to give its MAP score. The former is more intuitive, the latter probably better means the complete performance of the IR system.
• The above images of a table from the book gives some `mbox(P@10)` and MAP scores for the TREC45 and GOV2 data sets for some of the ranking methods we will consider this semester.
• We will talk about BM25, LMD (language modeling), DFR (divergence from randomness) later in the semester.
• Usually one doesn't compute such scores by hand, but instead uses some piece of well-tested software such as trec_eval.

Quiz

Which of the following is true?

1. The IDF score for a term as presented in class depends on the number of documents that contain the term.
2. If we are using VSM for ranking and retrieval, we can use a docid index for our inverted index.
3. The proximity ranking algorithm incorporates the number of documents that contain the query terms directly in the proximity score of a document.

Building a Test Collection

• In a simple set-up, given a topic we go through each document in our collection and either score it as 1 relevant or 0 not.
• At 10 seconds/document, for the 0.5 million document TREC45 collection, this would take about a year for one person to judge one topic.
• For the 25 million document GOV2 collection, someone would have to spend their whole career to just judge one topic.
• TREC instead typically uses a system of pooling:
  • One or more experimental runs from each group participating in an adhoc task is accepted.
  • Each run consists of the top 1000 or 10,000 documents for each topic.
  • The top 100 or so from each run are then pooled for that topic.
  • These are then presented to assessors in random order for judging. This takes about a week.
  • One group might rank a document 50 and another group 150, if its in the top 100 for any group it will be judged. This is called the pool depth.
  • Unjudged documents are treated as not relevant.
  • `mbox(P@10)`, `mbox(recall@k)`, and MAP scores are then calculated.

Efficiency Measures

• In addition to evaluating a system's retrieval effectiveness, one also wants to evaluate how efficiently it returns results.
• That is, things like the cost of running the system, and how quickly it gets results to its users.
• User's are typically interested in response time, the time between entering the query and receiving results.
• Another important measure is the throughput, the average number of queries processed in a given amount of time.
• Typically, response time is measured by taking a measure of the OS time before executing a set of queries, executing those queries, and then measuring the time again. An average is then taken over the number of queries in the set to give the average response time.

Token and Terms

• So far we have viewed tokens as sequences of alphanumeric text separated by nonalphanumeric text.
• We have further case normalized these tokens by converting each of their letters to lower case.
• We have also treated XML tags as tokens.
• In practice, more complex tokenization is usually done.
• For example, under the above scheme "didn't" would be two tokens "didn" and "t" and neither would match "did" "not".
• 10, 000, 000 would be three tokens and would not match 10000000, but might match "10".
• UTF-8 characters of other languages are ignored.
• For the next couple of days, we will examine some of these issues in detail.

Punctuation and Capitalization

• We've already mentioned that apostrophes can cause problems. For contractions it might make sense to tokenize them as separate words. For instance, "did" "not" for "didn't".
• For other uses, of apostrophe such as in possessives it might makes sense to tokenize things like "Bill's" as "bill" and "bill's", the latter with the apostrophe as part of token.
• Periods usually end sentences, and for such occurrences can be safely dropped. However, they also appear in acronyms such as "I.B.M." and somehow this token should map to the same things as "IBM".
• If we case normalize some acronyms such as "US" we lose their meaning.
• Sometimes text is written in ALL CAPS which could further confuse the issue.
• We also don't want to drop the special characters from some terms; otherwise, one has things like "C#" and "C++" become "C".
• Typically, the solution to these problems is a rule based approach, and each rule in the system would be need to be evaluated for its impact on results.

Stemming

• Stemming allows a query term such as "orienteering" to match an occurrence of "orienteers" or "runs" to match "running".
• Stemming thus considers the morphology or internal structure of terms, where the goal for an IR system is to reduce each term to its root form.
• The program/function that does this is called a stemmer.
• A related concept from linguistics is lemmatization.
• This is the process whereby a term is reduced to a lexeme, which roughly corresponds to a word in the sense of a dictionary entry.
• For each lexeme a particular form of the word, or a lemma, is chosen to represent it. For example, "run" for "running" and "ran".
• One common stemmer is the Porter Stemmer developed by Martin Porter in the 1970s. This has been ported to many languages. His website also has stemmers for other languages.

Finishing up Stemming

• The above give an example of text stemmed with this stemmer.
• As you can see, it operates only on suffixes and tends to be slightly aggressive. For instance, it would stem "orienteering" and "orienteers" both down to "orient", conflating the word orient with these terms.
• The stemmer also doesn't handle irregular forms well. For example, "ran" stems to "ran" rather than "run". "mouse" stems to "mous"; whereas, "mice" stems to "mice".
• The Porter Stemmer operates by applying a sequence of rewrites against the current suffix stem. For example,

  sses -> ss
  ies -> i
  ss -> ss //sas both sides the same says stop applying rules if see ss
  s ->
  

  All the rules are applied in sequence, if there was a change they are reapplied until no change. The result is the stemmed term.
• As a rule of thumb, stemming tends to improve recall, but reduce precision.

Stopping

• Function words are words that have no well-defined meanings in and of themselves; rather, they modify words or indicate grammatical relationships.
• In English, these include words which are prepositions, articles, pronouns, and conjunctions.
• Function words are usually among the most frequently occurring words in any language.
• When documents are viewed as unstructured "bags of words", as in the vector space model, the inclusion of function words may be unnecessary.
• Even in the proximity model, the close proximity of a particular function word to other query terms may convey very little information.
• For these reasons, many IR systems define stopwords, which are stripped from the query before doing an index look-up. These stopwords often include function words.
• Stopwords might also include single terms, digits, and other common terms, such as state-of-being verbs.
• Eliminating these terms from the index also has the side benefit that it will often speed up query response times as the index size is reduced.
• Unfortunately, eliminating these stop words can make queries such as "to be or not to be" impossible.