Introduction

On Wednesday, we were talking about data and process integration.
Recall data integration is the general set of techniques that allow us to create a unified view of data from different sources.
We looked at design patterns for Data Integration such as: Data consolidation, Data federation, Data propagation, Changed Data Capture, Data Virtualization, Data service composition via Service Oriented Architectures.
We then started talking about process integration - the integration and harmonization of business processes in an organization.
We said a business process is a set of activities with a certain order that must be executed to reach a certain organizational goal.
We said there are a variety of flow chart like tools to model such processes: business process model and notation (BPMN), Yet Another Workflow Language (YAWL), Unified Modeling Language (UML) Activity Diagrams, Event-driven Process Chain (EPC) diagrams...
Software to automate the act of following the steps of such business process charts is called a process engine.
We begin today by looking at some common approaches/patterns for process engines.

Approaches to Process Management

Process execution languages such as WS-BPEL often distinguish between two kinds of dependencies that relate one activity in a process has on another:
- A sequence dependency - execution of Service A must complete before execution of Service B. For example, a loan proposal can only be made after a positive credit score has been calculated.
- A data dependency - says that Service B requires data produced by Service A. For example, an interest rate of a loan proposal that depends on a credit score calculated during the credit check.
Two common patterns for managing sequence and data dependencies of businesses processes are:
- The orchestration pattern - (left above, used by WS-BPEL) Assume a single centralized executable business process called the orcestrator that coordinates the interaction among different services and subprocesses. The orchestrator is repsonsible for invoking, combining, and receiving updates of process and services.
- The choreography pattern - the participants in the process itself coordinate their collaboration.

Combining Data and Process Integration

There are two common ways that a data dependency between service A and service B can be resolved:
- Provide a data flow from A to B to make sure the necessary data are passed between the services at the level of the business process..
- Have Service A persist data into a data store which is also accessible through one of our data integration techniques by Service B.
To analyze data flow at the process and data integration level in a unified way, we consider three types of services:
- Workflow Services (might correspond to a process engine) coordinate the control flow and data flow of a business process by triggering its respective tasks in line with the sequence constraints in the process model, and according to an orchestration or choreography pattern. Steps may be executed by a machine or human and results returned to feed into rest of the process.
- Activity Services (might correspond to a web service) perform one task in a business process and return a result which can be used by the workflow process. For example, credit card validation
- Data Services provide unified access (say via federation, consolidation, virtualization, etc) to the business data in underlying data stores at some kind of level of quality of service as defined by response time, completeness, consistency, etc.

Data-related Infrastructure Services

Most service-oriented architecture data integration suites (used to build process and data integration architecture like we've been talking) provide different data-related infrastructure services that support the exploitation and management of data services.
These are often related to data quality and data governance.
Some example services include:
- Data profiling services - providing automated support for assessing and understanding content, quality, and structure of enterprise data.
- Data cleansing services - ensuring the validity and consistency of data For example, name and address cleansing
- Data enrichment services - enhance the data by exploiting external data sources.
- Data transformation services - tool for transforming data to match the target application's requirements
- Data event services - tools for monitoring data for state changes and rules, raising events that can be acted upon by other services.
- Metadata services - support the storage, integration and exploitation of diverse types of metadata

Quiz

Which of the following is true?

An Outrigger table is only used during the ETL process of Data Warehousing.
Data Lake and Data Mart are synonyms for Data Warehouse.
Data propagation is the data integration design pattern whereby updates are either synchronously or asynchronously propagated from a source to target system.

Searching Unstructured Data and Enterprise Search

The data integration patterns we have been considering are in principle equally applicable to structured, semi-structured, and unstructured data.
However, even if we have a unified place to access unstructured data such as multimedia data, full text documents, etc. It doesn't necessarily give us tools to search these collections efficiently.

If we think about the built-in search capabilities of SQL for example, they are quite limited:

SELECT * FROM documents
WHERE text LIKE '%SQL%' 
  OR text LIKE '%STRUCTURED QUERY LANGUAGE%'

If we have lots of documents we might want instead to sort them by some notion of relevancy, etc, which would be hard to achieve in SQL.
So we now look at some aspects of searching unstructured data.

Principles of Full Text Search

In class and on the homeworks, we have already described how to index full-text document into an inverted index.
In returning results to a query one typically:
- computes all the documents that have at least one term from the query (disjunctive query)
- computes all the documents that have all terms from the query (conjunctive query)
- computes all the documents that have an exact phrase match
- compute all the documents that have some combination of these (boolean query)
Results of these queries can be directly from posting lists of terms before looking up documents.
We also might filter this list of documents based on meta-information of the document: its size, when it was created, etc.
We then sort these results before returning them according to different criteria:
- Overall quality of the document (for example, page rank).
- Proximity of the terms within the document.
- Relevance of the terms to the document (might be measured for example by frequency of term in the document).
An enterprise search service might also provide additional features such as: thesaurus search (search triggers searches on related terms), fuzzy search (search that handles misspellings), text mining (compute most most relevant keywords for the document, classify documents into categories, etc).
To give a feel for these things, let's briefly look at how web search engines work... Next day, we will look at enterprise search.

What search engines do not do...

Search engines do not make use of leprechauns or any other magic.
To download a single page from the web in your browser can involve a number of steps:
- Find a numerical ip address of a machine associated with a particular site. For example,
```
http://www.cs.sjsu.edu/
```
  corresponds to the IP address
```
130.65.86.56
```
- Make a request for that page.
- Make a request for Javascripts, Flash objects, stylesheets, etc. on that page.
If after you typed your query into the search engine, the search engine had to go out to the web to try to find pages related to what you typed, it would be way too slow to get results.
Instead, the search engine already needs to know about everything that exists on the web first and then use that to try to give you search results.

Steps a search engine typically does

Downloads as much of the web as it can before ever serving the results of any search query.
Extracts words from the text on the pages it has downloaded.
Creates a big index associating each word found with a list of documents containing that word.

At this point, the search engine is ready to handle queries. To handle a query it might:

Look up each word in the query in the word-document index
Intersect the list of documents found for each word to produce a list of documents each of which has all of the words.
Group related documents.
Try to order the documents by how relevant they seem to be to the query.

A Diagram of Search Engine Parts

The fetcher, indexer, and web components of a search engine

You might have several queue servers each responsible for a range of URL hash values. Similarly, one would typically have several indexers.

Downloading the Web

The process of trying to download as many web pages as you can from the web is called crawling.
The GoogleBot is Google's crawler, BingBot are used by Bing, YioopBot is used by Yioop, etc.
The process of crawling is typically split among two kinds of modules: fetchers and queue servers. The results of their work (downloaded web pages) are often fed to a set of indexers, so they can be used by a query engine.
The queue server keeps a list of all the urls it knows about but has not downloaded.
This list begins with a set of seed sites.
It periodically generates schedules.
A fetcher who is looking for work, contacts the queue server to get a schedule and then start downloading all the pages listed in the schedule.
After downloading a page, the fetcher extracts words and links from it. It then might permanently store the page on disk and send a summary of the page contents back to the queue server.
The queue server looks at the links in the summary and tries to see if the links have already been crawled.
If not it adds the links to the queue as something to crawl in the future.
This process continues for as long as the person managing these machines has patience for.
Internet Archive did a crawl of the web in 2008. It took about 6 months to download 2 billion pages.
One of the fastest university crawlers, IRLbot, downloaded over a 1 gigabit/s connection 6 billion pages in 43 days, not counting downtime for crashes and maintenance.
The ClueWeb 2012 data set (1 billion pages) was crawled by five Dell PowerEdge R410 each with 64GB in 80 days (For ClueWeb 2009, a crawl of a billion pages was obtained in 2 months using 20 machines). Typically, these crawls are done on 1Gbps connections
What can you do on a home connection? My own set up at home (Yioop) was six 2011 Mac Mini's each with 8 GB Ram and a 4TB disk. My internet has a 20Mbps down connection. This crawled and indexed (rather than just crawled) 1 billion pages in 10 months in 2015.
My home set up is also used for serving search results which may seem a little slow. I am using disk and have close to 9billion records. For comparison, I was at a talk (circa 2013) where eBay was describing its indexes infrastructure. Typically, they partition their indexes into 20 million records/machine. Each machine has 64GB Ram and keeps the index in memory at all times. This means they can both do things way faster and handle more volume -- you get what you pay for.

Maintaining what to crawl next.

The queue server maintains a priority queue of urls it is aware of, but for which it hasn't downloaded the page yet.
For the Yioop! search engine this priority queue is implemented as a heap with an auxiliary hash table.
To add a url, one computes a 8 byte hash of the url and stores the url into a file of urls, obtaining a single integer offset into this file for the url.
One then stores the hash of the url into the priority queue with a weight used to determine when it will be crawled.
Storing this hash in the queue returns an offset into the priority queue, the hash of the url, the offset into the file of urls, and the offset into the priority queue are then stored in a hash table.
This set-up means we can adjust the weights of urls if needed without needing to keep the full url in memory (which might be long).

How to decide what to crawl next.

One way to weight things is to weigh a url as more than the weight of the page it was found on (assuming our priority queue has the min-weight object at the top).
This would have the effect of crawling the web in breadth-first order: You first crawl the seed sites, then you crawl the sites which are one link away from seed sites, then you crawl sites which are two links away from seed sites and so on.
Another technique (which assumes our priority queue has the max-weight object at the top)is to use a priority queue and something known as the online page importance algorithm (OPIC).
Here we imagine that each seed site is initially given some amount of money, say 10 dollars. When a fetcher downloads the pages and extracts the links it distributes the money equally amongst all of the links for that page. So if there were 10 links on the page the fetcher would give them each a dollar and then send the summary, now including dollar figures, to the queue server.
It is completely possible for more than one page to point to a given other page. For instance, many pages have links to yahoo.com.
If the queue server gets from a fetcher a link that is already in its queue, it adds the money that the fetcher gave to the url to the value the queue server has for that url.
When the queue server wants to decide what to crawl next it picks the page that has the most money.
Having a lot of money is a measure of how important the page was, so this crawls roughly according to page importance.

How to keep track of what we've already seen.

Some web-crawling strategies like Opic actually recrawl already seen pages with a frequency depending on how important the page was.
This gives a policy for incrementally recrawling the web, but it means you need some facility for aggregating crawls of a given page.
Another simpler strategy is to never crawl the same page twice, but we need to keep track of all urls we've seen
At the very least, we don't want to keep requesting the same robot file over and over for every page on a site.
Still other sites might be poorly constructed and have so called recursive urls
We want to keep a complete list of robots paths for all sites we've crawled and maybe a list of all sites we've already crawled in memory (RAM) so that its fast. Even keeping the 8 byte id's in a big hash table means we are taking a lot of space.
One strategy for doing this (used by Yioop! and Heritrix) is to use a Bloom filter. We allocate a fixed amount of memory (M bits) to hold all the urls and paths (roughly 3 bits per item, we need to store).
To add an item to the filter, we have a set of distinct hash function say 1,..,k, which return values between 0 and M-1. We compute the hash of the url_id with each hash function in turn and set the corresponding bit in the filter to one.
To see an item is already in the filter, we compute its hash with respect to each hash function and verify whether the corresponding bits are all one.

Preprocessing and Indexing

Before indexing begins you usually need to do some pre-processing.
For example, it can be the case that multiple urls points to essentially identical pages.
For this reason, a step called deduplication is done to eliminate duplicate pages. One way to do this is to check if two documents viewed as strings of characters have the same hash value according to some hash function.
To build an index, we imagine each document has an id (usually a hash of its url). Similarly, each word has an id (a hash of characters). This allows us to use the same number of bytes (say 8 bytes) as an identifier for each document and word.
We then cycle over each document we downloaded
We extract the words from the documents.
For each word found, we compute a score for how important that word seems to be for that document compared to how common a word it is in general (BM25).
We add the document id, the documents money rank, and the word score to the list associated with the word id in the index. This list is called a posting list.
Once we have done this, we sort each word list from highest to smallest as far as some function of money rank and word score.
We then compress the index so that it can be mostly stored in ram on a machine with a sufficient large memory and export it to our search engine servers.

Search Engine Algorithms

I won't go into too much detail on tecniques for ranking documents, but I want to describe one famous kind of document quality measure...
Google's famous PageRank (Brin Page 1998) algorithm assigns a rank, `r(P)`, to a page `P` roughly as the sum `sum_ i \frac{r(P_i)}{|P_i|}` over each page `P_i` which links to it. Here `|P_i|` is the number of links going out of page `P_i`.
Pages are returned from a search based on whether they have the keywords and how relevant these words are (using BM25) and based on the page rank.
Ask.com (Teoma) used to use another kind of algorithm called HITs (Kleinberg 1998). This works by iteratively computing two scores for a page: (a) an authority score measuring the sum of hubs linking to it and (b) a hub score measuring how good the authorities linked from this site are. It is query dependent in its calculation of (a) and (b).

More on Page Rank

Notice to calculate `r(P)` from the last slide we need to know `r(P_i)`.
To solve this, let `\vec{r}` be the vector of ranks of all web pages. i.e., the `j`th component of `\vec r` is `r(P_j)`. We assume the rank represents the probability a random person on the web happens to be on a particular page, so for now the ranks are between 0 and 1. Let `A_{ij}` be `1/|P_i|` if there is an edge from `i` to `j`; `0`, otherwise. Then roughly we want a `\vec{r}` such that `\vec{r} = \vec{A}\vec{r}`.
The idea is to start with a guess for `\vec{r}`, say `\vec{r'}`, and compute powers `\vec{A}^{(n)}\vec{r'}` until
`||\vec{A}^{(n+1)}\vec{r'} - \vec{A}^{(n)}\vec{r'}|| < \epsilon`
for whatever choice of `\epsilon` is you think is sufficiently small. Then, if we set `\vec{r} = \vec{A}^{(n)}\vec{r'}`, we will have
`\vec{A}\vec{r} = \vec{A}^{(n+1)}\vec{r'} \approx \vec{A}^{(n)}\vec{r'} = \vec{r}`
as desired.
This is called the power method for computing eigenvalues.
The Frobenius Perron Theorem shows you actually get convergence as `n` gets large provided `\vec{A}` meets certain conditions (which the adjacency matrix probably does not).

Tweaks on the basic matrix

The original Page Brin paper adds tweaks to the basic adjacency matrix known as the random surfer model to guarantee convergence.
For instance, whenever one has a node without outlinks, one pretends the node is connected to every other node on the web. This modification ensures the matrix `vec A` is stochastic.
It leaves the possibility though that the web is split into strongly connected components, which could mean that powering `\vec A` would cycle through a set of vectors but not converge. To fix this, one takes a linear combination `\alpha \vec A + (1-\alpha)\vec H` where `\vec H` is the matrix where every entry is `\frac(1)(\mbox{number of pages on the web})`. This modification can be viewed as saying that on any page on the web the random surfer might decide randomly to switch to any other page on the web.
Google supposedly uses `50` iterations to approximate `\vec{r}`.
The `r(P_i)` you get from the above calculations are numbers between `0` and `1`. Taking negative logs and scaling gives the more familiar values between `0` and `10`.

Data and Process Integration, Search

Outline