Outline

Grammar-Based Compression Algorithms

• Besides their use in defining parsable programming languages, CFGs, can also be used in text compression algorithms.
• The Liv Zempel family of compression algorithms (Liv Zempel 1977 and 1978, Welch 1984) can be viewed as an example of this approach.
• The idea is given a string we want to compress, we try to come up with a small CFG that generates only that string. Then we encode the string by its grammar written in some suitable encoding for grammars.
• There are brute force algorithms (i.e., slow) algorithms to find the smallest CFG for a given string.
• LZ 78 is fast and known to be able to approximate the smallest grammar to within a factor `Omega(n^(2/3)/log n)` as lower bound and to within a factor an `O((n/log n)^(2/3))` factor as an upper bound.
• It is known to approximate the smallest grammar to within a `8569/8568` factor is NP-hard (Lehman and Shelat 2002).
• Charikar, et al. ( IEEE Transaction of Information Theory. 2005. pp. 2554--2576) give a compression algorithm that approximates the smallest grammar to within a factor of `O(log(n/g^star))` where `g^star` is the size of the smallest grammar for that string. (Rytter 2002 has a similar result by compressing to CNF grammars written to be AVL trees).
• We will present a simpler to describe grammar-based compression algorithm known as SEQUITUR (Nevill-Manning, Witten, Maulsby, 1997) which runs in linear time and space and also approximates the smallest grammar to within a `Omega(n^(1/3))` factor as a lower bound and `O((n/log n)^(3/4))` as an upper bound.

SEQUITUR

• For this algorithm we will make use of an auxiliary string `mbox(Aux)` that contains the results of the compression process so far.
• We will use a hash table called `mbox(Hash)` that supports deletion which will store key values (a pair of symbols that appear in `mbox(Aux)`, position of pair in `mbox(Aux)`). This will allow fast look up of these pairs.
• We will also have a table of productions we have generated so far. The right hand side of these productions will always have length at most `2` before the final step of this algorithm is applied.
• At the start of the algorithm `mbox(Aux)` is empty, `mbox(Hash)` is empty, and our production table is empty. Let `w= w_1, ldots, w_n` be the string we'd like to compress. The algorithm proceeds on this input as follows:
  1. For `i = 1` to `n`: Add `w_i` to the end of `mbox(Aux)` and look at the last two symbols of `mbox(Aux)`. These might be a mix terminals and variables. If:
     1. The last two symbols can be found on the right-hand side of some production in our table, replace in `mbox(Aux)` these two symbols by the left hand side variable of the production in question. Mark the production as having been used more than once.
     2. The last two symbols occur somewhere else earlier in `mbox(Aux)` (we can check by looking up these two symbols in `mbox(Hash)`). Replace the two symbols with a new variable and a new production to our table consisting of this new variable going to these two symbols. Delete these two symbols from `mbox(Hash)`. The Hash structure should tell us which pair of symbols had the next immediately higher Aux position in the hash table. We replace the left character in the the two character pair with the new variable. We then place the last two symbols of `mbox(Aux)` in `mbox(Hash)`. We look at the pair of symbols in `mbox(Hash)` with the highest Aux position. We replace the rightmost character in this with the new variable.
     We perform the above two checks in sequence and might do both and repeat if we can still apply either.
  2. Remove rules that are only used once in the grammar by compressing them. So if `A->BC`, `B->de`, and `C->fg` and the latter two rules are only used once, then we delete these rules and add the rule `A->defg`.
  3. Finally, we output the grammar consisting of the rules in our production table together with a new start variable S that is used in the rule `S -> mbox(final value of Aux)`.

Example of Compressing with SEQUITUR

• Suppose the input string was `\a\b\b\a\b\a\b\b\a\b\b\a\b\b`. The following table shows the state of `mbox(Aux)` and the productions table after each letter is read:

  Letter	`mbox(Aux)`	Production Table	`mbox(Hash)` (Writing only keys, not position values, or pointer to next higher position value)
  a	a	`{}`	`{}`
  b	ab	`{}`	`{ab}`
  b	abb	`{}`	{ab, bb}
  a	abba	`{}`	`{ab, \b\b, ba}`
  b	abbab becomes AbA	`{A->ab}`	`{Ab, bA}`
  a	AbAa	`{A->ab}`	`{Ab, bA, Aa}`
  b	AbAab becomes AbAA	`{A->ab` marked}	`{Ab, bA, \A\A}`
  b	AbAAb becomes BAB	`{A->ab` marked, `B->Ab}`	`{\B\A, \A\B}`
  a	BABa	`{A->ab` marked,`B->Ab}`	`{\B\A, \A\B, \B\a}`
  b	BABab becomes BABA becomes CC	`{A->ab` marked, `B->Ab`, `C->BA}`	`{\C\C}`
  b	CCb	`{A->ab` marked, `B->Ab`, `C->BA}`	`{\C\C, \C\b}`
  a	CCba	`{A->ab` marked, `B->Ab`, `C->BA}`	`{\C\C, \C\b, ba}`
  b	CCbab becomes CCbA	`{A->ab` marked, `B->Ab`, `C->BA}`	`{\C\C, \C\b, \b\A}`
  b	CCbAb becomes CCbB	`{A->ab` marked, `B->Ab` marked, `C->BA}`	`{\C\C, \C\B}`

• Initially, we get the grammar `{A->ab`, `B->Ab`, `C->BA`, `S->\C\Cb\B}`
• After removing rules that were used only once we have `{A->ab`, `B->Ab`, `S->\B\A\B\Ab\B}`.
• As a crude, not too efficient example of how we could encode this... If we used `R` to denote the start of a rule, # followed by a sequence of digits for `n` to denote the `n` variable, and let `t` denote the terminal `t` unless `t` is `R`, '#', '\', or a digit in which case we use `\\t` is used to denote `t`, then the string `\a\b\b\a\b\a\b\b\a\b\b\a\b\b` might be compressed as RabR#0bR#1#0#1#0b#1.
• We don't have to list the left hand side variable for a rule, because each variable appears at most once on the left hand side and we can count `R`'s to figure out which rule we are.
• It works better on longer strings. For example, a string of 64 `a`'s might be encoded as RaaR#0#0R#1#1R#2#2R#3#3R#4#4, a string of length 28.

In-Class Exercise

• Build a table like the previous slide to show how the Sequitur algorithm would encode a string of string of 16 `a`'s.
• Post your solution to the Apr 8 In-Class Exercise Thread.

Closure Properties of CFL

CFL are closed under union

Proof idea: Let `G` and `H` with start symbols `S` and `T` respectively, be two CFGs for the CFL's we want to take the union of. Make a new grammar with the same alphabet, with the union of the two grammars productions (after renaming) together with the new rules `S'->S|T` where `S'` is the new start variable.

CFLs are closed under intersection by regular languages

Proof idea: use the Cartesian product construction on a PDA for the CFL together with a DFA for the regular language.

CFLs are not closed under intersection or complementation

Proof: The languages `{a^nb^nc^m | n, m ge 0}` and `{a^mb^nc^n | n, m ge0}` are both context-free. Their intersection is `{a^nb^nc^n | n ge 0}` which is not CFL by the pumping lemma. CFLs are not closed under complementation as using deMorgan rules, intersection can be defined from union and complementation.

Algorithms for CFLs

• We have already given an algorithm (CYK) for checking if a string is in a CFL.
• We have also given algorithms for checking for useless variables in grammars, as well as, for eliminating `epsilon`-rules from grammars.
• Here are a couple of more things we have CFG algorithms for:

There is an algorithm, which given a grammar `G` written down formally as a 4-tuple, can decide whether or not `L(G)` is empty.

To do this, the algorithm check if the start variable is useless. If it is we know the language is empty; otherwise it is not.

There is an algorithm to check given G whether or not L(G) is infinite.

To do this, we first eliminate `epsilon`-rules, unit-productions, and useless symbols from `G`. We then construct a graph where `(A,B)` is an edge for two variables `A`, `B` in the graph iff `A->xBy` for some production in `G`. If there is a cycle in this graph then `C=>^star uCv` for some variable `C` in the original grammar. As there are no useless symbols in this grammar, we must have `C=>^star z`, for some string `z` of terminals. Hence, also, `S=>^star sCt =>^star szt, S=>^star sCt =>^star suCvt =>^star suzvt`, etc. Thus, one can argue there is a cycle in the graph iff `G'`s grammar is infinite.

General Models of Computation

• So far we have looked at machines that either have bounded memory or access to memory limited to stack operations.
• We would like to consider models of computation which correspond to general purpose computers.
• In 1936 ("On Computable Numbers", London Math Society), Alan Turing presented such a general model of a computer, the Turing Machine.
• In this model, the machine has a finite control and an arbitrarily long tape of data consisting of squares able to hold one symbol. The machine also has a read head which can read one square at a time. Initially, this tape is blank except for the n first squares under and to the right of the tape head which have the input. The machine in one step is allowed to read what's under its tape head, write a new symbol, move left or right one square and change its state.
• The machine has special states which cause it to halt. The contents of the tape when these states are entered is the output of the machine.
• The first actual computer developed for code-breaking during World War II was partially based on his model.
• It turns out this model is actually equivalent to what can be done on modern computers.

A Real-Life Turing Machine

[YouTube video of above machine in action]

Example

• Let `B` be the language `{w\#w | w in {0,1}^star }`. This language is not context-free.
• A Turing Machine `M` that could accept this language might operate as follows:
  

   On input `w`:
  (1) Zig-zag across the tape to the corresponding positions on either 
  side of the # symbol to check whether these positions contain the 
  same symbol. If they do not, or if no # is found go into the 
  reject state. If they have the same symbol change the symbol 
  to a new symbol X (on both sides).
  (2) When all the symbols on the left side of the # have been 
  X'd out, check if there are any more symbols to the right of the #. 
  If yes reject; if not accept.
  

• By accept or reject, I mean we view the halting states as further partitioned into those which are accepting and those which are rejecting and we enter the appropriate kind of halting state.