Introduction

On Monday, we began talking about data mining - the process of examining data and finding simple rules or models that summarize the data.
We developed a model framework called the market basket where we have a set of items people might purchase together in a basket at a store.
We were interested in finding association rules in this model, statements of the form `{i_1,i_2,...,i_n} =>j` where the `i_k`'s and `j` are items and the intended meaning of a rule is: if the `i_k`'s are in the basket, then so is `j`.
We said a useful association rule should satisfy: high support, the items `{i_1, i_2, ..., i_n, j}` occur in a lot of baskets; high confidence, the probability of finding item `j` in a basket that has all of `\{i_1, i_2, ..., i_n\}` is above some threshold; and interest, the probability of finding item `j` in a basket that has all of `\{i_1, i_2, ..., i_n\}` is significantly higher or lower than the probability of `j` in a random basket.
The most crucial of these we said is high support, so we began looking at algorithms to find frequent item sets. That is, item sets with frequency greater than some support threshold, `s>0`.
For computation efficiency, items sets are typically stored linearly in files or sequences of files. For example, {BI,BS,BU}{HP1,HP2,HP3}{BI,HP1}{BI,BS,HP1,HP2}{... rather than stored using relational DBMSs.
We measure algorithmic complexity in terms of numbers of passes over this data.
We gave two naive algorithms for computing frequent item sets: using an upper triangular matrix and using a hash table.
We then presented the A Priori algorithm.
We start today by showing this algorithm again and then presenting a common approach to optimizing it called the PCY algorithm.

A Priori Algorithm

The A Priori Algorithm (Rakesh Agrawal and Ramakrishnan Srikant 1994) is a famous algorithm for computing frequent item sets.

It operates as follows:

INPUT: A file D consisting of baskets of items, a support threshold s, 
    and a size limit q for the size of frequent item sets.
OUTPUT: The sets of itemsets F[1],F[2],..., F[q], where F[i] is the 
    set of all itemsets of size i that appear in at least s baskets of D.
METHOD:
(1) LET C[1] = all items that appear in file D;
(2) For n = 1 to q:
(3)    F[n] = those sets in C[n] that occur at least s times in D
(4)    IF n==q BREAK;
(5)    LET C[n+1] = all itemsets of S of size n+1 such that every
           subset S of size n is in F[n].

A Priori Memory Use

The heart of the A Priori algorithm is line (3) which is implemented as a single pass through the input data.
A Priori can be used only if there is enough main memory to satisfy the requirements to count all the candidate sets on each pass.
If there isn't enough memory, either a more space-efficient algorithm must be used, or several passes must be used for a single value of `n`.
We can use either an upper triangular matrix or hash table to organize the main-memory counts during a pass.
The hard value of `n` is usually 2, as there are usually many 1 item sets of high support and so there are quadratic in this many 2-item candidates in these 1-item sets to consider. If we can handle the 2-item set case, usually, the 3 and higher itemset cases involve small enough candidates sets that we'll have enough memory.
One approach to save memory is to perform pass 1 (compute counts of all 1 item sets), then scan through this list of (item_id, count_item) pairs, and output just items_ids (but not counts) above the threshold. Typically, we'd also sort these item_ids.
The figure above shows what main memory would like at this stage.
The PCY Algorithm which we now describe uses this frequent 1-itemset list then to determine the frequent 2-item sets.

PCY Algorithm

The PCY algorithm was proposed by J. S. Park, M.-S. Chen, and P. S. Yu 1997 to efficiently perform steps (3) and (5) of the A Priori algorithm.
The image above shows the memory we now have left to look for candidate pairs after the operations performed on the previous slide.
So the memory used to store the frequent items is less than was needed to store the item counts, so we can presumably consider more candidate pairs than if we hadn't done this.
The PCY algorithm tries to exploit this extra space to make a bitmap that can be used to further reduce the number of candidate pairs to consider.
To do this, during the first pass when we a considering each basket in turn and adding 1 to the counts of its items, we also hash each pair of items in its basket to a bucket in a hash table and add one to the count in that bucket.
We hope that some buckets will end up with a count less than `s`, the support threshold. So any pair `{i,j}` that hashes to that bucket would not be a frequent item set.
Between the first and second passes, we replace the buckets by a bitmap with one bit per bucket. Here the bit is 1 if the corresponding bucket is a frequent bucket. If we store buckets with 32 bit int's, then the bitmap will be 1/32 the size of these buckets.

Using the Bitmap

The above figure now shows the memory usage with bitmaps.
On the second pass through the dataset, we examine each basket in turn. For that basket, we consider the items {i, j} that it contains, we view {i,j} as a candidate and add 1 to its count only if both the following conditions are satisfied:
1. Both `i` and `j` are frequent items.
2. `{i, j}` hashes to a bucket that the bitmap tells us is a frequent bucket.

In-Class Exercise

Suppose we are computing the A Priori algorithm with and without using PCY on the above dataset with `s=3`.
How much memory would be needed for pass 1 without PCY? (Assume 4 byte int's.)
How much memory would be needed for pass 2 without PCY? (lets say we use one int with xFFFFFFFF in it to separate F[1] from 2 itemset data)
Use h(i,j) = (i+j)%3 as your hash function (let BI = 0, ...,HP3=5). What would the PCY bitmap look like when computing 2-candidates?
How much memory would be needed for pass 2 with PCY assuming we had 3 buckets.
Please post your solutions to the May 10 In-Class Exercise Thread.

When to Use the PCY Algorithm

If the set of candidate pairs is sufficiently irregular that we cannot use the triangular-matrix method for organizing counts; we must use a table of counts.
It does not make sense to use PCY unless the number of candidate pairs is reduced to at most 1/3 of all possible pairs.
For PCY to be an improvement over A Priori a good fraction of the buckets on the first pass must not be frequent; otherwise, the PCY bitmap checks do not eliminate many pairs.
Let `P` be the total number of occurrences of pairs of items among all the baskets in the dataset.
Since most of the main memory `M` can be devoted to buckets, the number of buckets will be approximately `M/4`.
The average count of a bucket will then be `(4P)/M`.
In order that there be many buckets that are not frequent, we need `(4P)/M < s`, or `M > (4P)/s`.

Multistage PCY

Instead of counting pairs on the second pass, as we do in A-Priori or PCY, we could use the same bucketing technique (with a different hash function) on the second pass.
To make the average counts even smaller on the second pass, we do not even have to consider a pair on the second pass unless it would be counted on the second pass of PCY; that is, the pair consists of two frequent items and also hashed to a frequent bucket on the first pass.
This idea leads to three-pass version of the multi-stage algorithm for finding frequent pairs.
It can, of course, be generalized to even more iterations.
Now when looking at candidate pairs {i,j} in a baskets, we only add to its count if:
1. Both `i` and `j` are frequent items.
2. `{i, j}` hashed to a frequent bucket on the first pass (determined using first bitmap).
3. `{i, j}` hashed to a frequent bucket on the second pass (determined using second bitmap).

More Data Mining

Outline

Introduction

A Priori Algorithm

A Priori Memory Use

PCY Algorithm

Using the Bitmap

In-Class Exercise

When to Use the PCY Algorithm

Multistage PCY