Introduction

On Monday, we described various measures for the quality of search results (P@k, MAP, etc), how to develop a test corpus, and measures of query processing efficency.
Today, we begin by talking about how documents are processed into tokens and terms before they are placed into an inverted index.
Then we are going to learn enough PHP to use Yioop as an IR library, so we can see how some of these processes like stemming actually work.

Token and Terms

So far we have viewed tokens as sequences of alphanumeric text separated by nonalphanumeric text.
We have further case normalized these tokens by converting each of their letters to lower case.
We have also treated XML tags as tokens.
In practice, more complex tokenization is usually done.
For example, under the above scheme "didn't" would be two tokens "didn" and "t" and neither would match "did" "not".
10, 000, 000 would be three tokens and would not match 10000000, but might match "10".
UTF-8 characters of other languages are ignored.
For the next couple of days, we will examine some of these issues in detail.

Punctuation and Capitalization

We've already mentioned that apostrophes can cause problems. For contractions it might make sense to tokenize them as separate words. For instance, "did" "not" for "didn't".
For other uses, of apostrophe such as in possessives it might makes sense to tokenize things like "Bill's" as "bill" and "bill's", the latter with the apostrophe as part of token.
Periods usually end sentences, and for such occurrences can be safely dropped. However, they also appear in acronyms such as "I.B.M." and somehow this token should map to the same things as "IBM".
If we case normalize some acronyms such as "US" we lose their meaning.
Sometimes text is written in ALL CAPS which could further confuse the issue.
We also don't want to drop the special characters from some terms; otherwise, one has things like "C#" and "C++" become "C".
Typically, the solution to these problems is a rule based approach, and each rule in the system would be need to be evaluated for its impact on results.

Stemming

Stemming allows a query term such as "orienteering" to match an occurrence of "orienteers" or "runs" to match "running".
Stemming thus considers the morphology or internal structure of terms, where the goal for an IR system is to reduce each term to its root form.
The program/function that does this is called a stemmer.
A related concept from linguistics is lemmatization.
This is the process whereby a term is reduced to a lexeme, which roughly corresponds to a word in the sense of a dictionary entry.
For each lexeme a particular form of the word, or a lemma, is chosen to represent it. For example, "run" for "running" and "ran".
One common stemmer is the Porter Stemmer developed by Martin Porter in the 1970s. This has been ported to many languages. His website also has stemmers for other languages.

Finishing up Stemming

The above give an example of text stemmed with this stemmer.
As you can see, it operates only on suffixes and tends to be slightly aggressive. For instance, it would stem "orienteering" and "orienteers" both down to "orient", conflating the word orient with these terms.
The stemmer also doesn't handle irregular forms well. For example, "ran" stems to "ran" rather than "run". "mouse" stems to "mous"; whereas, "mice" stems to "mice".
The Porter Stemmer operates by applying a sequence of rewrites against the current suffix stem. For example,
```
sses -> ss
ies -> i
ss -> ss //sas both sides the same says stop applying rules if see ss
s ->
```
All the rules are applied in sequence, if there was a change they are reapplied until no change. The result is the stemmed term.
As a rule of thumb, stemming tends to improve recall, but reduce precision.

Stopping

Function words are words that have no well-defined meanings in and of themselves; rather, they modify words or indicate grammatical relationships.
In English, these include words which are prepositions, articles, pronouns, and conjunctions.
Function words are usually among the most frequently occurring words in any language.
When documents are viewed as unstructured "bags of words", as in the vector space model, the inclusion of function words may be unnecessary.
Even in the proximity model, the close proximity of a particular function word to other query terms may convey very little information.
For these reasons, many IR systems define stopwords, which are stripped from the query before doing an index look-up. These stopwords often include function words.
Stopwords might also include single terms, digits, and other common terms, such as state-of-being verbs.
Eliminating these terms from the index also has the side benefit that it will often speed up query response times as the index size is reduced.
Unfortunately, eliminating these stop words can make queries such as "to be or not to be" impossible.

In-Class Exercise

Say how "Kids, puppies, and kisses." would be normalized and stemmed if Porter Stemming were used.
Give a concrete situation in which stemming would increase recall, but decrease precision.
Post your solutions to the Mar 3 In-Class Exercise Thread.

Classes in PHP

We are now going to switch topics back to PHP so we can understand how to use Yioop as an IR library.

Classes in PHP can be defined with a syntax like:

class MyClass
{
   public $field_var1;
   public $field_var2 = 5;
   ...
   const $constant1;
   const $constant2;
   ...

   function __construct($arg1, $arg2) {
     //constructor code
   }

   function myFun1() {
     // code
   }
   ...
}

Traditionally, you put the code for your class into a file and then use a line like:
```
require "MyClass.inc";
//or more likely
require_once "MyClass.inc";
```
We will discuss a more graceful way to load class definitions known as autoloading after we finish talking about classes and namespaces.
The require function gives a fatal error if the class cannot be found. Also you cannot use require to include remote files even if allow_url_fopen is enabled.
require_once will not re-include the file if it has already been included.
PHP comes with several built-in classes and interfaces.
Several common data structures are also implemented in the Standard PHP Library (SPL).

Constructors/Destructors

We can set up an initial value of a field variable of a class when we declare it:
```
public $my_variable = 0;
```
You can also have a functions __construct and __destruct to do initialization and clean-up.
```
function __construct($n=0) 
{
     $this->my_variable = $n;
}
```

Private, Protected, Public

Member variables and member functions can be declared private, protected or public:

public $my_field1;
var $my_field2;
private $my_field3;
protected function myMethod() { /* some code*/}

Methods without any declaration are the same as public. Fields declared with var are also public.
private means only visible within the class.
protected is visible within the class or within subclasses.
public means the variable or method is visible to anyone within the namespace. The PSR-2 coding guidelines prefer using public over var for field declarations.

Static and Const

The static keyword creates one instance of the field or method for the class.
```
class Foo{ static $bob=1;} 
echo "bob: ".Foo::$bob;
```
Within the class use self:: to refer to static members.
The const keyword can be used to define constants for a class:
```
class Goo{ const BLOB=1;}
```
Notice no dollar sign. Can refer to this using
```
Goo::BLOB
```
Values of constants cannot be changed.
You can create constants outside of any class using the define keyword. For example,
```
define("PI", 22/7);
echo PI;
```

Text Preprocessing and More PHP

Outline

Introduction

Token and Terms

Punctuation and Capitalization

Stemming

Finishing up Stemming

Stopping

In-Class Exercise

Classes in PHP

Constructors/Destructors

Private, Protected, Public

Static and Const