Outline

• Introduction
• Overall Architecture
• Crawling
• Indexing
• Serving Results

Introduction

• You are bored with what you've been eating lately and don't know what to cook tonight.
• So you try to look online for something exotic...
• On a whim you type "platypus recipes" in your favorite search engine...
• You get a bunch of results back from the search engine.
• How did it find these results?
• What is it doing?
• Today's lecture is going to try to answer that question.

What search engines do not do...

• Search engines do not make use of leprechauns or any other magic.
• To download a single page from the web in your browser can involve a number of steps:
  • Find a numerical ip address of a machine associated with a particular site. For example,

    http://www.cs.sjsu.edu/
    

    corresponds to the IP address

    130.65.86.56
    

  • Make a request for that page.
  • Make a request for Javascripts, Flash objects, stylesheets, etc. on that page.
• If after you typed your query into the search engine, the search engine had to go out to the web to try to find pages related to what you typed, it would be way too slow to get results.
• Instead, the search engine already needs to know about everything that exists on the web first and then use that to try to give you search results.

Steps a search engine typically does

1. Downloads as much of the web as it can, before ever serving the results of any search query.
2. Extracts words from the text on the pages it has downloaded.
3. Creates a big index associating each word found with a list of documents containing that word.

At this point, the search engine is ready to handle queries. To handle a query it might:

1. Look up each word in the query in the word-document index
2. Intersect the list of documents found for each word to produce a list of documents each of which has all of the words.
3. Group related documents.
4. Try to order the documents by how relevant they seem to be to the query.

A Diagram of Search Engine Parts

Downloading the Web

• The process of trying to download as many web pages as you can from the web is called crawling.
• The GoogleBot is Google's crawler, YSlurp is Yahoo!'s crawler, MSNBot is used by Bing.
• The process of crawling is typically split among two kinds of modules: fetchers and a queue server. And maybe also indexers.
• The queue server keeps a list of all the urls it knows about but has not downloaded.
• This list begins with a set of seed sites.
• It periodically generates schedules.
• A fetcher who is looking for work, contacts the queue server to get a schedule and then start downloading all the pages listed in the schedule.
• After downloading a page, the fetcher extracts words and links from it. It then might permenantly store the page on disk and send a summary of the page contents back to the queue server.
• The queue server looks at the links in the summary and tries to see if the links have already been crawled.
• If not it adds the links to the queue as something to crawl in the future.
• This process continues for as long as the person managing these machines has patience for.
• Internet Archive did a crawl of the web in 2008. It took about 6 months to download 2 billion pages.
• One of the fastest university crawlers, IRLbot, downloaded aver a 1 gigabit/s connection 6 billion pages in 43 days, not counting downtime for crashes and maintenance.

Maintaining what to crawl next.

• The queue server maintains a priority queue of urls it is aware of, but for which it hasn't downloaded the page yet.
• For the Yioop! search engine this priority queue is implemented as a heap with an auxiliary hash table.
• To add a url, one computes a 8 byte hash of it and stores the url into a file of urls, obtaining a single int into this file.
• One then stores the hash of the url into the priority queue with a weight used to determine when it will be crawled.
• Storing this hash in the queue returns an offset into the priority queue, the hash of the url, the offset into the file of urls, and the offset into the priority queue are then stored in a hash table.
• This set-up means we can adjust the weights of urls if needed without needing to keep the full url in memory (which might be long).

How to decide what to crawl next.

• One way to weight things is to weigh a url as more more than the weight of the page it was found on.
• This would have the effect of crawling the web in breadth-first order: You first crawl the seed sites, then you crawl the sites which are one link away from seed sites, then you crawl sites which are two links away from seed sites and so on.
• Another technique is to use a priority queue and something known as the online page importance algorithm (OPIC).
• Here we imagine that each seed site is initially given some amount of money, say 10 dollars. When a fetcher downloads the pages and extracts the links it distributes the money equally amongst all of the links for that page. So if there were 10 links on the page the fetcher would give them each a dollar and then send the summary, now including dollar figures, to the queue server.
• It is completely possible for more than one page to point to a given other page. For instance, many pages have links to yahoo.com.
• If the queue server gets from a fetcher a link that is already in its queue, it adds the money that the fetcher gave to the url to the value the queue server has for that url.
• When the queue server wants to decide what to crawl next it picks the page that has the most money.
• Having a lot of money is a measure of how important the page was, so this crawls roughly according to page importance.

Making sure we don't crawl what we're not supposed to

• Some sites might not want you to crawl every page they have.
• Other sites might want you to crawl, but not too fast.
• One way a site can tell your spider about such information is have a robots.txt file in the root directory of that site or to include meta tags in their html documents
• Here is an example of a portion of the robots.txt file for www.physics.sjsu.edu:

  User-agent: *
  Crawl-delay: 10
  # Directories
  Disallow: /includes/
  Disallow: /misc/
  
  

• This Disallow lines say what directories not to crawl. Crawl-delay gives a delay in seconds between requests.
• Some sites have hidden links on pages in forbidden directories. If a user-agent follows the link, it is considered a spider and its IP address is blocked. This is called a spider trap.
• To keep your spider on friendly terms with websites. You need to download the robots page before crawling any pages on a site.
• Implementing crawl delay is tricky if you have multiple fetchers: the queue-server typically needs to remember crawl-delayed sites and make sure only one fetcher at a time has a batch with urls for that sites. The order of the urls to crawl for that fetcher must have url's for the crawl delayed site appropriately spaced.

How to keep track of what we've already seen.

• Some web-crawling strategies like Opic actually recrawl already seen pages with a frequency depending on how important the page was.
• This gives a policy for incrementally recrawling the web, but it means you need some facility for aggregating crawls of a given page.
• Another simpler strategy is to never crawl the same page twice, but we need to keep track of all urls we've seen
• At the very least, we don't want to keep requesting the same robot file over and over for every page on a site.
• Still other sites might be poorly constructed and have so called recursive urls
• We want to keep a complete list of robots paths for all sites we've crawled and maybe a list of all sites we've already crawled in memory (RAM) so that its fast. Even keeping the 8 byte id's in a big hash table means we are taking a lot of space.
• One strategy for doing this (used by Yioop! and Heritrix) is to use a Bloom filter. We allocate a fixed amount of memory (M bits) to hold all the urls and paths (roughly 3 bits per item, we need to store).
• To add an item to the filter, we have a set of distinct hash function say 1,..,k, which return values between 0 and M-1. We compute the hash of the url_id with each hash function in turn and set the corresponding bit in the filter to one.
• To see an item is already in the filter, we compute its hash with respect to each hash function and verify whether the corresponding bits are all one.

Preprocessing and Indexing

• Before indexing begins you usually need to do some pre-processing.
• For example, it can be the case that multiple urls points to essentially identical pages.
• For this reason, a step called deduplication is done to eliminate duplicate pages. One way to do this is to check if two documents viewed as strings of characters have the same hash value according to some hash function.
• To build an index, we imagine each document has an id (usually a hash of its url). Similarly, each word has an id (a hash of characters). This allows us to use the same number of bytes (say 8 bytes) as an identifier for each document and word.
• We then cycle over each document we downloaded
• We extract the words from the documents.
• For each word found, we compute a score for how important that word seems to be for that document compared to how common a word it is in general (BM25).
• We add the document id, the documents money rank, and the word score to the list associated with the word id in the index. This list is called a posting list.
• Once we have done this, we sort each word list from highest to smallest as far as some function of money rank and word score.
• We then compress the index so that it can be mostly stored in ram on a machine with a sufficient large memory and export it to our search engine servers.

Example Index Structure Components

• Databases are not typically used to store the results of a large scale web crawl. This is because even a small crawl of a million pages might involve more than 100 million total rows for the word occurrences. Each row in a database might have a 1Kb size even for just a single column. Also, the way databases do indexing is typically also too slow as the row size gets this big to keep up with data coming from the fetchers.
• What follows is an example of the components and how a word-doc (inverted) index might be structured:
  • Global Dictionary. This consists a prefix index with entries (word_id_prefix, offset to where that word_id_prefix first occurs) followed by a sequence of actual entries of the form: (word_id, len_entry, (shard_id, first_offset, last_offset), (shard_id, first_offset, last_offset) ...). Values in an entry might be compressed using varint, gamma, delta, or rice codes. It is designed to be able to fit entirely in memory on some machine.
  • Index shards. These consist of two kinds of entries: (1) a set of posting lists, (2) a set of doc entries.
  • The posting lists in a shard are concatenated one after another. Within a posting list a posting entry might look like (doc_offset, word_cnt) or (doc_offset, word_pos).
  • The first offset and last_offset for a shard in the global dictionary correspond to the start and end in the set of posting lists for a particular posting list.
  • doc_offset are offsets into the set of doc entries. A doc entry might look like (doc_id, doc_rank, doc_length, code of file storing doc, offset into file storing doc.
  • Index Shards are sized, so that they too can fit in memory on some machine (not necessarily the same as the one with the Global dictionary).
  • The queue server/indexer after indexing the amount of documents it can fix in memory, writes the shard to disk, and merges dictionary data to the global dictionary using a log-merge.

Serving Results

• When a query comes into one of the search engine server it looks up in its index the words you have typed.
• It adds the scores for each word for a particular document. If a word doesn't occur in a given document it is often dropped from the results.
• Related documents are then grouped, for example by url, and a total score for the group is generated.
• The results for all groups found are sorted and the results are formatted (keywords in summaries highlighted) and sent back to the user.

Conclusion

• So that's how you can go out and find platypus recipes from search engines.