Text Formats

Human language data in electronic text format represents the raw material of IR.
To build an IR system it is important to understand the different formats this text can come in.
Example formats include: HTML, Word DOC, Word DOCX, XML, LATeX, RTF, PDF, RSS, etc.
HTML and XML are perhaps the most important of these for writing a search engine.
This is because unlike plain text, these formats allow us to explicitly use links to represent relationships between documents and parts of documents.

Example HTML page
<html>
<head><title>A Line of Hamlet</title></head>
<body>
<h1>Hamlet being serious</h1>
<p>To be, or not to be,--that is the question:--</p>
<p><a 
href="http://www.gutenberg.org/cache/epub/1524/pg1524.html"
>Full text of hamt</a></p>
</body>
</html>

XML

XML also has a notion of link -- XLink.
XML is a language for specify new tag-based languages. These new languages can be geared to the semantic use at hand.
In HTML the tags do give us only a little semantic information: We know an h tag is more likely to be short and important than a p tag.
On the hand, in XML, we can create a language patient with tags like name, age, etc which are very specific in the content they hold.
An IR system should be able to make use of this additional information -- especially, if the system knows the XML language in question.

More Text Formats

Here is a brief description of some other text formats:

Microsoft formats - originally, Word (doc), Excel (xls), Powerpoint (ppt) were all proprietary binary formats. Due to lawsuits, the specs for these formats are now available online. Modern versions of Office use .docx, .xlsx, and .pptx files. These files are all really zip files which contain a directory of xml files. They are thus relatively straightforward to parse.
Open Office formats - nowadays, this is also a zip file of xml documents.
PDF and Postscript - Postscript is essentially a program for how to render a document on a printer using a version of the language Forth. In PDF, a document consists of a sequence of objects from one or more container types. The easiest of the object types to extract data from is essentially ZIP compressed text.
Javascript or Flash - These are programming language are often used to write web pages, and as they are full-fledged programming languages, it often make difficult to figure out the text or meaning they encapsulate.

Quiz

Which of the following is true?

The word document in an information retrieval setting always means web page.
The inverted index for a document collection is typically much smaller than the collection itself.
Question answering is an example application of information retrieval.

Tokenization of English Text

To process a document to build an inverted index for it, we need to convert that document into a sequence of tokens.
As a preliminary step, each of the formats discussed before must be converted to some kind of raw text. This often involves discarding low level things such as formatting tags (for example, br tags), fonts information, etc.
As part of this step we might convert the text to a standard character encoding. For example, we might convert both ASCII and Big-5 to UTF-8.
For English XML, we might process this raw text, treating each successive sequence of consecutive alphanumeric characters as a token.
We might convert tag tokens to upper-case and words tokens to lower-case letters.
Later, we might also run a stemmer on each word.
The set of distinct tokens, or symbols, gotten by the above process is called the vocabulary, `V`, of the document or document collection.
For the works of Shakespeare, `V` might look like:
`V= \{`a, aaron, abaissiez, ..., zounds, zwaggered, ...,<PLAY> ...`\}`
We will often make things like a term-frequency table listing for each term the number of times it occurred.
We will use the word occurence to refer to a particular time a term occurred. So to spell out an occurrence you need to say which document and where within the document the term happens.

Term Distributions

Rank Frequency Token
1     107,833  <LINE>
2     107,833  </LINE>
3      31,081  <SPEAKER>
4      31,081  </SPEAKER>
5      31,028  <SPEECH>
6      31,028  <SPEECH>
7      28,317  the
8      26,022  and
9      22,639  i
10     19,898  to

The above represents a term-frequency table for the 10 most common terms in an XML collection of Shakespeare's plays.
As you can see 'the' is the most common word, followed by other pronouns, conjunctions, prepositions, etc.
8,336 terms, of the 22,943 words and 43 tags in this collection, occur only once. For example, zwaggered occurs once.
The relative frequency of tags is hard to predict, but for English words -- this is true of other languages as well -- there is a predicatble relationship between the frequency of a term and its rank called Zipf's Law.
It says that the frequency of the `i`th most common term, `F_i`, will be roughly `1/i^{alpha}` for some constant `alpha`.
Knowing this law allows us to figure out things like the most important terms in a query. It also can be used to develop a model for the relevance of a term to a document.

Language Modeling

Imagine that a previously unknown Shakespearean play was discovered. Can we predict anything about the content of this play given the contents of the plays we do know?
One technique to make predictions about an unknown text is to use a special kind of probability distribution known as a language model.
A language model, `M`, is a probability distribution over the terms of a vocabulary. That is,
`M: mbox{term} -> mbox{probability of that term}`
So we have: `sum_(sigma in V) M(sigma) = 1`.
For an existing corpus (collection of documents), we can make a simple language model by defining:
`M(sigma) = frac{mbox{frequency}(sigma)}{sum_{sigma' in V}mbox{frequency}(sigma')}`
We can use our model to determine things like the probability of a phrase like "to be or not to be".
To do this we could take the product of the probabilities of each of the terms in the phrase:
`2.18% times 0.76% times 0.27% times 0.93% times 2.18% times 0.76% = 0.000000000069% `
Such an estimate of the probability of a phrase is called a maximum likelihood estimate (MLE).
On Wednesday, we will look at other kinds of language models as well as give some demos of different IR systems. Please bring your laptops!

Text Formats, Tokenization, Term Distribution, Language Models

Outline

Text Formats

XML

More Text Formats

Quiz

Tokenization of English Text

Term Distributions

Language Modeling