Introduction

You want to make it easy for your customers to find and do business with you online.
Since a main avenue to your web-site is a search engine, it is important to make the relevant parts of your web-site rank highly (first page or two) with respect to the most popular search engines.
Today, we are going to look at some techniques for doing this.

Search Engine Algorithms

You can find which are the most popular search engines by looking at searchenginewatch.com or Neilson's.
In order, these are Google (1), Baidu (Chinese) (4), Bing (15), Yandex (Russian) (21), Naver (Korean) (62), Sogou (Chinese) (108), Gigablast (25,359). (I am only listing ones that mainly use their own crawler and indexer, not resellers. The number in parentheses is the global rank among all web sites as given by Alexa, May 11, 2016).
Google's famous PageRank (Brin Page 1998) algorithm assigns a rank, `r(P)`, to a page `P` roughly as the sum `sum_ i \frac{r(P_i)}{|P_i|}` over each page `P_i` which links to it. Here `|P_i|` is the number of links going out of page `P_i`.
Pages are returned from a search based on whether they have the keywords and how relevant these words are (using BM25) and based on the page rank.
Ask.com (Teoma) used to use another kind of algorithm called HITs (Kleinberg 1998). This works by iteratively computing two scores for a page: (a) an authority score measuring the sum of hubs linking to it and (b) a hub score measuring how good the authorities linked from this site are. It is query dependent in its calculation of (a) and (b).

More on Page Rank

Notice to calculate `r(P)` from the last slide we need to know `r(P_i)`.
To solve this, let `\vec{r}` be the vector of ranks of all web pages. i.e., the `j`th component of `\vec r` is `r(P_j)`. We assume the rank represents the probability a random person on the web happens to be on a particular page, so for now the ranks are between 0 and 1. Let `A_{ij}` be `1/|P_i|` if there is an edge from `i` to `j`; `0`, otherwise. Then roughly we want a `\vec{r}` such that `\vec{r} = \vec{A}\vec{r}`.
The idea is to start with a guess for `\vec{r}`, say `\vec{r'}`, and compute powers `\vec{A}^{(n)}\vec{r'}` until
`||\vec{A}^{(n+1)}\vec{r'} - \vec{A}^{(n)}\vec{r'}|| < \epsilon`
for whatever choice of `\epsilon` is you think is sufficiently small. Then, if we set `\vec{r} = \vec{A}^{(n)}\vec{r'}`, we will have
`\vec{A}\vec{r} = \vec{A}^{(n+1)}\vec{r'} \approx \vec{A}^{(n)}\vec{r'} = \vec{r}`
as desired.
This is called the power method for computing eigenvalues.
The Frobenius Perron Theorem shows you actually get convergence as `n` gets large provided `\vec{A}` meets certain conditions (which the adjacency matrix probably does not).

Tweaks on the basic matrix

The original Page Brin paper adds tweaks to the basic adjacency matrix known as the random surfer model to guarantee convergence.
For instance, whenever one has a node without outlinks, one pretends the node is connected to every other node on the web. This modification ensures the matrix `vec A` is stochastic.
It leaves the possibility though that the web is split into strongly connected components, which could mean that powering `\vec A` would cycle through a set of vectors but not converge. To fix this, one takes a linear combination `\alpha \vec A + (1-\alpha)\vec H` where `\vec H` is the matrix where every entry is `\frac(1)(\mbox{number of pages on the web})`. This modification can be viewed as saying that on any page on the web the random surfer might decide randomly to switch to any other page on the web.
Google supposedly uses `50` iterations to approximate `\vec{r}`.
The `r(P_i)` you get from the above calculations are numbers between `0` and `1`. Taking negative logs and scaling gives the more familiar values between `0` and `10`.

Factors which contribute to search engine ranking

From the above we discussion we see that we would like lots of links into our site.
Hopefully, these would be from sites that don't have too many outgoing links.
However, using a link farm or even being associated with the IP of such a site can get one banned from search results.
We also want to be using the correct keywords to get our potential customers to our product.
In some rankings it can help if we link to good related sites.
To get our site noticed at all as last resort you can submit your site to Google, Yahoo, DMOZ, etc using their sitemap APIs. That way, the crawlers will be aware of your site.
DMOZ (Open Directory Project) used to be especially important because Google for a while was using what is called Topic Sensitive Page Rank (TSPR).
In TSPR, you compute several modified page ranks (one for each topic you want to consider) and at query time compute a weighted sum of these ranks based on the query to give a final rank. The modified page ranks are computed by running the page rank algorithm where the values for the ranks of some pages set to a fixed high value. The pages to choose as being high were often coming from who was listed in DMOZ for that topic.

Link Optimizations

As we've seen above, links are very important to the ranking of web pages.
In order for a link to be used in a ranking algorithm it needs to be understood by the crawler or spider sent out by a search engine company to cache pages.
Links like: product.php?blah=1, product.php?blah=2, etc. will often either not be followed or may be followed once under product.php or may be followed less well.
Links that need posted data to work won't be cached at all.
Things like links in Javascript, Flash, etc tend to be less well indexed.
Sometimes people look at the User-Agent header to see if it is a spider and based on this serve a different page which is more friendly to the spider.
This is called cloaking and is risky as it can get one banned from search results.

URL Rewriting

To handle the problem of links like: product.php?blah=1, product.php?blah=2 you can use URL rewriting.
To do this you need to make sure the mod_rewrite module is loaded and added in Apache.
Then you need to have a line like: RewriteEngine on
For example, RewriteRule ^/product/(.*)/$ /product.php?blah=$1
We can of course use more general reg_exps than above

Site Map

A site map is one page on your website which contains links to all the most common other pages on your website.
It can help ensure that a spider will hit all the pages that you want on your site.
The major search engines have also agreed on an XML language for publishing sitemaps, the details of which can be found at http://www.sitemaps.org/.
Files written in this language can be linked to from your robots.txt using a line like
```
Sitemap: url
```

Let's Experiment

Download the robots.txt file for flikr.com. What are the different user-agents that are specified in it? What are the different directives you see? For each find out what it does. Download one of the sitemap xml files and see what tags it has try to find out what each does. Write up this info and post it to the May 11 Discussion Thread.

To find out about humans on flikr, download the humans.txt file.

RSS Feeds

Having a RSS feed is a way of publishing what's new on your website.
People, aggregator,etc will periodically download the feed file. For instance, every hour, so it is a very useful tool.

An example file looks like:

<?xml version="1.0"?> 
<rss version="2.0" xmlns:xlink="http://www.w3.org/1999/xlink"> 
<channel> 
<title>Computer Science Department, San Jose State University -
Old News Items </title> 
<link>http://www.cs.sjsu.edu/</link> 
<description>Archived of selected news items from the 
Department of Computer Science at SJSU.</description> 
<language>en-us</language> 
<lastBuildDate>Thursday, 16 February 2006</lastBuildDate>
<item> 
<title>Job opportunity for CS students or graduates.</title> 
<pubDate>Monday, 8 May 2006</pubDate> 
<description>	JOB OPPORTUNITY!
PageBites, a Palo Alto startup, is hiring. If you are interested </description> </item>
<!-- could add more items -->
</channel> 
</rss>

On-page Optimizations

Besides trying to improve the links on your site and links to your site, you also want to make sure that keywords associated with your product can be found when the spider hits the page.
One way to find which keywords related to your product are frequently searched on is to look at a site like: https://adwords.google.com/KeywordPlanner or http://www.wordtracker.com/
Some engines give more weight to keywords contained in certain places.
Some good places to put important keywords are the title and meta tags, in the first h1 and h2 tags of your document and in links.
If possible you also want links which point to your site to contain the keyword.

Search Engine Optimization

Outline