Introduction

Last week, we were considering document quality measures -- overall quality measures for documents that are independent of the query.
Such measures are often an important component of search result scores.
Usually they are computed by viewing the corpus as a graph of documents that are connected with each other by things like hyperlinks, citations, etc.
We weight the edges in the corpus graph, then use the weighted adjacency matrix to assign weights (document quality) to the nodes (i.e., the documents).
We considered the following document quality measures: Traffic Rank, OPIC, Page Rank, HITS, Salsa.
We then looked at how Page Rank, which is used by Google, might be implemented as a map reduce job.
So far, we have not looked at any production systems that can run map reduce jobs.
So we begin today by considering Hadoop, before moving on to our last topic of the semester: document categorization and filtering.

Map Reduce Using Hadoop

Hadoop is an open source re-implementation of the Google File System and Map Reduce created in 2003 by Doug Cutting and Mike Cafarella then at Yahoo.
Currently, it is an Apache project.
To give a flavor of an actually coded Map Reduce algorithm we briefly present how to write the word count example from earlier in terms of Hadoop.
You can use apt, brew, choco to get an installation of Hadoop suitable for single machine development or testing. For example, I did
```
brew install hadoop
```
to install Hadoop on my laptop.
From the command line one runs Hadoop with lines like:
```
hadoop some-command args
```
For example, if some command is jar then it runs the map-reduce job given by the arguments.
Without any arguments hadoop gives the list of possibilities for some-command, one interesting subset of which is fs, which indicates a command for the hadoop distributed file system.
Hadoop has as one of its components YARN (Yet Another Resource Negotiator) which is used to manage and locate resources which might be distributed across several machines. YARN can be run from the command line using the command yarn.

Word Count as a Hadoop Map Reduce job

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount
{
    /* This inner class is used for our Map job
       First two params to Mapper are types of KeyIn ValueIn
       last two are types of KeyOut, ValueOut
     */
    public static class WCMapper extends Mapper<Object,Text,Text,IntWritable>
    {
        public void map(Object key, Text value, Context context)
            throws IOException, InterruptedException
        {
            // normalize document case, get rid of non word chars
            String document = value.toString().toLowerCase()
                .replaceAll("[^a-z\\s]", "");
            String[] words = document.split(" ");
            for (String word : words) {
                Text textWord = new Text(word);
                IntWritable one = new IntWritable(1);
                context.write(textWord, one);
            }
        }
    }
    /* This inner class is used for our Reducer job
       First two params to Reducer are types of KeyIn ValueIn
       last two are types of KeyOut, ValueOut
     */
    public static class WCReducer extends Reducer<Text, IntWritable,
        Text, IntWritable>
    {
        public void reduce(Text key, Iterable<IntWritable> values,
            Context context) throws IOException, InterruptedException
        {
            int sum = 0;
            IntWritable result = new IntWritable();
            for (IntWritable val: values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }
    public static void main (String[] args) throws Exception
    {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "wordcount");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(WCMapper.class);
        job.setReducerClass(WCReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        TextInputFormat.addInputPath(job, new Path(args[0]));
        Path outputDir = new Path(args[1]);
        FileOutputFormat.setOutputPath(job, outputDir);
        FileSystem fs = FileSystem.get(conf);
        fs.delete(outputDir, true);
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

Compiling and Executing WordCount.java on Hadoop

To compile WordCount.java we need to make sure we have all the relevant classpaths for the Hadoop classes we want.

To do this we use YARN:

javac -classpath `yarn classpath` -d . WordCount.java

We then make a jar file out of our compiled program:

jar -cf WordCount.jar WordCount.class 'WordCount$WCMapper.class' 'WordCount$WCReducer.class'

We can then run the program on hadoop:

hadoop jar WordCount.jar WordCount /Users/cpollett/tmp.txt /Users/cpollett/output

Here tmp.txt is the file we want to count the words for, /Users/cpollett/output is where we are outputting the results in HDFS format for the job.
After running the job, we could see what files were in the output folder with:
```
hadoop fs -ls /Users/cpollett/output/
```
Finally, if we want to see the counts, we could type:
```
hadoop fs -cat /Users/cpollett/output/part-r-00000
```

In my case, this outputs:

2018-12-05 11:28:01,201 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
	1
foo	2
la	3

Quiz

Which of the following is true?

In TrafficRank, we give web crawl seeds sites an initial quantity of money, then download those pages and reassign money to their links.
Both PageRank and HITs rely on the power method to compute document scores.
SALSA makes use of a teleporter matrix.

Long-Term Recurring Information Needs

Our next topic involves information needs that are longstanding, recurring, or common for a large population of users.
For example, given a set of text documents determine the language that each is written in?
As another example, given a set of documents, determine which ones are spam?
The information needed in both cases is commonly understood and likely to a rise in many circumstances.
For example, a search engine might need to identify the language to determine a relevant response, a customer support help desk, might need to identify the language of a email to route it to someone capable of reading it.
A spam filter might prevent a user wasting unnecessary time on emails which are not useful. It might also prevent the indexing of web content which prevents the return of more relevant documents.
As yet another example, in handling medical data, we might need to be able to automatically filter current medical information from historical information.

Categorization and Filtering

To be more precise on the problems we'd like to solve we introduce the following terminology:
Categorization is the process of labeling documents to satisfy some information need. For example, label documents ham (relevant) or spam (not relevant).
Filtering is the process of evaluating on an ongoing basis according to satisfy some information need. For example, routing emails according to language or scanning medical documents and routing those related to mental health to mental health care professionals in a newsletter.
The term routing is often used synonymously with filtering but with the added connotation that the documents are delivered to distinct locations.
Another term for filtering is selective dissemination of information (SDI).
In some sense both classes can be thought of as duals to traditional search: In search, we want to find the documents satisfying the information need; whereas, in categorization and filtering, we want to find the information need satisfied by the document.

Approaches to Categorization and Filtering

Categorization and filtering tasks are often specified differently than in a search setting. For example, one does not generally search just for "spam documents" or "Swahili documents" in the search setting.
Also, the longstanding nature of the task (for example, deliver medical documents meeting some criteria) often makes it worthwhile to train the system in more detail to handle the specific need.
One approach (expert systems) to doing this is have an expert manually specify rules in some restricted languages for each category.
Only documents meeting the conditions of each of the rules for the category would belong to the category.
We will focus on automated approaches to categorization based on machine learning. These produce programs called classifiers to perform the categorization task.
Just as a search engine can return either a fixed set of documents (Boolean retrieval) or a ranked list, classifiers may be either hard (belongs to category or not) or soft (confidence score).
Soft classification is often called ranking, and the study of soft classifiers is called learning to rank.
The ranking functions we discussed earlier, such as BM25 or DFR, can be considered as a kind of soft classifier.
We will look at several common filtering and categorization problems, look at ways to solve, and look at ways to evaluate our solutions...

Topic-Oriented Batch Filtering

Consider the health-care task we presented earlier.
At the start, there are no documents. As new documents arrive, we could imagine filtering them.
If we can afford to wait, we can accumulate documents into a corpus, index them, apply a search technique like BM25 to rank them according to the health categories desired, and send them to the appropriate users.
We can then wait for more documents to arrive and repeat the process.
This approach is called batch filtering.
This approach is viable if enough documents arrive within the repetition window, say once a week, once a month, etc.

Example of Topic Oriented Filtering

The above table shows the results of doing batch filtering for our medical task (Topic 283) on the TREC 45 corpus using BM25.
The query used to rank documents for relevance was `langle` "mental", "illness", "drugs" `rangle`.
`N=140,651` Financial Times articles from 1993 onward in TREC45 were used. Of these 67 were relevant to Topic 283.
Four character char grams were used for tokens. (According to the book, stemming and not stemming approaches yielded similar results).
Documents were grouped into batches of fixed size, depending on the experiment. For example for the 1000 batch experiment, the batch size consisted of 141 documents.
Each batch was indexed independently of the rest of the collection. Each batch is presented to the BM25 filter one at a time in chronological order.
To evaluate the results of more than one batch, the table above shows the simplistic approach of evaluating each performance measure separately for each batch, then average these results to compute a summary measure.

Issues With Evaluating the Result

Some problems with evaluating as we did on the last slide are:
- When we divide into more batches, the number of documents/batch goes down, and so the likelihood a batch has no relevant documents increases. (`|Rel| = 0`). `AP` (average precision) will be undefined if any batch has `|Rel|=0`. So any measure that depends on `|Rel|` will not work for small batches.
- Measures like precision at `k` are heavily influenced by `n`, making it impossible to compare results from different batch sizes.
We can see these effect in our tables from the last slide -- the `AP` values for the Second Batch as well as the average are undefined for greater than 10 batches. Both the `AP` and `P`@`10` results also don't seem to be very stable with respect to batch size.

Size-Specific Precision at `k`

One way to solve the issue with `P`@`k` of the last slides is to compute size specific `P`@`k`. I.e., `k` used for evaluating batches sizes of size `n` is set to `\lfloor rho n\rfloor` where `rho` is some constant.
Typically, `rho` is chosen to be `k/N` where `N` is the corpus size and `k` is roughly the total number of documents presented to the user.
The table above shows using this approaches gives more stable results across batch sizes.

Aggregate Precision at `k`

One drawback with size specific precision at `k`, `P`@`k`, is that the number of relevant documents can vary between batches, but we are always using the same value for `k`.
Instead, of fixing `k` across all of our batches of a given size, we can use the fact that we have a scoring function `s` (BM25 in our example) and in a given batch present to the user all documents which have a relevance score above some threshold `t`.
We choose `t` so that `k=rho N` documents will have score `s >t`.
We define aggregate `P`@`k` and `AP` using this thresholding idea.
The table above shows this approach is even more stable than Size-Specific Precision and AP.

Online Filtering

Online filtering is essentially batch filtering where the batch size is 1.
An online filter must act immediately on each message in a sequences rather than on batches of messages.
Messages deemed relevant must be delivered and all others discarded.
Delivery might be an email to an inbox or text message or sending a message to some folder.
If possible we would like to be able to flag messages by priority. I.e., have some kind of primitive importance ranking.That way we don't need to discard messages.
Otherwise, we need to set some relevance threshold so that the rate of delivery `rho` represents a good balance of precision and recall as measure by a score like an F-measure.
Scores like BM25 when used to give a priority score need to be modified so they can work on a per-document basis. For example `N_t` assumes a large collection of documents. To fake this one can just assume the corpus consists of the single current document.
Above shows the aggregate precision scores for online filtering using BM25 and a theshold that gives a given target (10, 20, 100, etc) number of results over the whole collection versus single batch filtering (whole corpus 1 batch).

Historical Collection Statistics

At the time we deploy a filtering system we might not know the rate at which relevant documents arrive, making it harder to choose thresholds, etc.
We next consider how to determine of this kind of information.
If we are doing batch filtering, one way to do this is to take advantage of historical information.
In the batch setting, we can do this is by combining historical documents with a given batch to do ranking, then removing the historical documents when the ranking has been completed leaving a rank and score for the documents in the batch.
This will mean the batch makes use of better collection statistics such as IDF scores and so the relevancy scores determined will be more accurate.
Rather than return a fixed number of documents per batch, we can return only those documents that appear in the top `k' > k` in the combined list
For a suitable `k'` this will return the same number of documents overall while achieving better precision.

Online Filtering with Historical Collection Statistics -- What to Store

When historical documents are used, online filtering effectively reduces to batch filtering.
One difference is that we only need to figure out where a single documents score is relative the list we have already computed.
Hence, we don't need to build a full inverted index apparatus we have previously considered.
All we need to compute BM25 or a similar score is a dictionary with entries indicating `N_t` for each term. For overall ranking, we might also keep an ordered list of historical scores (with respect to all terms, not per term scores).
We can compute relevance using the current document and `N_t`. To compute rank we binary search in the list to insert the score of the current document.
The table above compares online filtering with and without use of historical information.
We haven't explained the last row of the above table yet.

Historical Training Examples

In the filtering setting it is likely that some examples of relevant documents are already known.
For example, before we start filtering medical papers related to cardiac surgery, we probably have some known examples of papers on this subject.
Such examples are known as training examples or labeled data.
In machine learning, building a classifier using labeled data is known as supervised learning.
Using BM25 relevance feedback is an example of how we could build such a classifier.
I.e., we could look at the `r` known relevant documents, select the top `m` term scores with respect to these documents, say `m=20` and then augment our original query with these terms.
This was used to produce the last row of results from the previous slide for the query we had last day using TREC45 data.

Language Categorization - Filtering

Two common tasks associated with documents and the language they are written in are:
- Categorization: Given a document `d` and a set of possible categories, identify the category to which `d` belongs. Related to this is Category Ranking: rank the category according to how likely the document belongs to that category.
- Filtering: Given a category `c` determine which documents belong to this category. Related to this is document ranking list the documents in order of the likelihood of belong to the category.
The book's authors used the random link feature of Wikipedia to collect 8012 articles from each of 60 languages from Wikipedia.
They then extracted the first 50 bytes of text from each article as snippet messages.
They used 4000 snippets for training and 4012 for categorization and filtering.
To do filtering, each language was treated as a separate topic. Snippets were 4-grammed and training snippets grams were used as query terms in BM25 document ranking for test documents. (I.e., from the 4000 snippets take all in a given language and thenall 4-grams thereof - treat that as the query for BM25 ranking).
The graph above shows the results of this document ranking experiment.

Language Categorization - Categorization

To perform categorization we have to rank languages instead of documents.
To do this we use the relevance scores from the document ranking.
Given a document `d` and a language `l`, let `s(d,l)` be the relevance score for `d` in the document ranking for `l`.
Given two language `l_1` and `l_2` we assume if `s(d,l_1) > s(d,l_2)` that `d` is more likely to be written in `l_1` than `l_2`.
The table above shows the complete ordering of BM25 scores for each language for a snippet of German text.
In this setting P@1 is most commonly called accuracy. This indicates the proportion of documents for which the correct category was ranked first.
MAP scores in this setting are equal to Mean Reciprocal Rank (MRR) scores because there is exactly one relevant result per ranking.
For a single topic, reciprocal rank is defined as `\R\R=1/r` where `r` is the rank of the first and only relevant result. In the case `AP = P`@`r =1/r`.
Summary statistics are often expressed as either micro-averages which are computed over all documents without regard to category or macro-averages which are an average of summary measures computed for each category.
For the accuracy and the language categorization tests just considered both of these scores were 0.79. For MRR, the scores were 0.860 and 0.857 respectively.

On-line Adaptive Spam Filtering

Framework for Categorization

One tool for evaluating spam filtering is the TREC 2005 Public Spam Corpus.
This consists of a sequence of 92,180 messages, of which 39,399 are spam and 78,798 are ham.
Rather than splitting the sequence into training and tests sets, on-line feedback is used.
In this setting, an ideal user is assumed to detect and report all errors.
So every message may be used as a historical training example immediately after it is delivered.
The online scenario is implemented using the TREC Spam Filter Evaluation Toolkit.
This toolkit can be used for any binary classification task (we have two exclusive categories), although we are using it for just spam versus ham.
For evaluation, within the toolkit's framework, a filter must support following operations:
- initialize - Create a filter a process a new stream of messages
- classify file - Return a category and priority for the message in file.
- train spam file - Use file as a historical example of spam.
- train ham file - Use file as a historical example of ham.
- finalize - shutdown the filter.

BM25 Applied to On-line Adaptive Email Filtering

We now give an example of using this framework together with BM25 for Spam filtering.
To start we call initialize to create an initial filter which has no information on which to base its decision.
The rate and manner in which this filter improves with training is known as the learning curve.
We apply BM25 with relevance feedback separately to each of the document ranking problems: identifying spam and identifying ham.
This yields two scores: `s(d, spam)` and `s(d, ham)`.
The category ranking for `d` is determined by the difference `s(d) = s(d,spam) - s(d, ham)`.
If `s(d) > 0` the document is classified as spam, otherwise, it is classified as ham.

BM25 Applied to Adaptive Email Filtering -cont'd

In their experiments, when either a train ham or train spam is called, the book's author's filter generates 4-grams from the message and keeps track of the average length of the spam or ham document and `N_t` for that term.
When classify file is called, a BM25 score is computed with respect to the 4-grams in file and these `N_t` and average length values for both spam and for ham. Finally, `s(d)` is calculated.

Evaluation

The above shows the evaluation measures for the primary categorization problem in an on-line spam filter.
For `F_1` score we assume spam constitutes the relevant class for spam
Notice the P@1 score for spam is very high.
Instead, error `1-P`@`1` expressed as a percentage is often used.
So an error rate of .1% is 10 time better than one of 1%.
The above shows that about 1 in 10000 spam messages is delivered to the user's in-box.
On the other hand, the ham error rate is 32.9% so about a third of good messages are quarantined.
This is probably unacceptable, but the reverse situation might not be.
Micro-average and macro-average scores fail to take into consideration the prevalence of spam. So doubling the amount of ham while keeping the amount of spam the same actual would improve both error rates.
To compensate for this, we can use a logit average LAM. Here `LAM(x,y) = logit^{-1}((logit(x) +logit(y))/2)` is often used.

Choosing a threshold

Rather than accept the ham error rate above the book explores setting different values for `t` and then considering `s(d) > t` as spam .
For example, when `t=2156` the spam error is 56.8% but the ham error rate is 0%. On the other hand, when `t=-180` the spam rate is 0 and the ham rate is 34.9%.
One can use this to choose the best compromise that the filter offers for the two errors.
We can also plot the points (spam error, ham error) for fixed `t` as we let `t` vary. This gives the so-called receiver operating characteristic (ROC) curve.
To compare filters one often looks at 1-AUC (area over this curve) expressed as a percentage.
The book shows the score for this filter versus spam assassin are comparable.

Map Reduce on Hadoop - Categorization and Filtering

Outline