Introduction

On Monday, we said that circuits lower bounds were one approach to the `P !=NP` problem and that they were also interesting for their connection to derandomization and cryptography.
The connection to cryptography is because to do derandomization our canonical approach is to make really strong pseudo-random number generators.
We gave lower bounds for general AND, OR, NOT circuits within `Sigma_3`-`TIME(2^{O(n)})`, `Sigma_2^p \cap Pi_2^p`, and `NP`.
We said that the best known lower bound on circuit size for a language in `NP`, was `5n-o(n)` using the method of gate elimination.
Today, we look at the problem of coming up with lower bounds for restricted circuit models.

Circuit Lower Bounds - Restricted Classes

We are only going to look at one restricted circuit lower bound result in detail.
Here are some known lower bounds for restricted circuit classes.
Recall `AC^0` meant constant depth, unbounded fan-in, AND, OR, NOT circuits. `AC^0[q]` also allows unbounded fan-in `MOD_q` gates for some prime `q`.
A monotone circuit is one without NOT gates.

Hard Function	Class	Proof Technique
Parity	`AC^0`	(FSS 83, Hastad85) Hastad's Switching Lemma: Given a k-DNF circuit, if one randomly assign some variable, the odds it can't be written as a CNF are `< 1`. Use this to switch a depth `d` circuit down to a depth 2 one, and directly a depth 2 circuit argue can't compute parity.
`MOD_p`	`AC^0[q]`	(Razborov Smolensky 87) Method of approximation - will give in detail in a minute
inner product	depth-2 TC^0	(Hajnal, et al 87) Convert such circuits into a randomized communication protocol, show a lower bound on this protocol for inner product.
`CLIQUE_{n,k}`	monotone P/poly	(Razborov 85) Define a notion of a crude circuit, a particular kind of DNF which tests if a family of subsets of a graph forms a clique. Shows any crude circuit for `CLIQUE_{n,k}` requires large size. Then shows any monotone circuit can be approximated by a crude circuit of roughly the same size. The proof is inductive and makes use of the Sunflower lemma from combinatorics.
`s-t`-connectivity	monotone `o(log^2)`-depth circuits	(KW 88) Converts circuit to a communication protocol in which only a certain number of bits are exchanged in a given round, then show the number of rounds is large for a problem FORK, and in turn lower bounds `s-t`-connectivity by FORK. Communication complexity has also been used to connect minimum number of gate wires with area and time of VLSI circuits.

`MOD_p !in AC^0[q]` for `p,q` prime

We will only show `MOD_2 !in AC^0[3]`, but the general result is similar.
We use the term `MOD_2` gate.
The proof involves two steps:
1. Show by induction on `h` that for any depth `h`, `MOD_3` circuit on `n` inputs and size `S`, there is a polynomial of degree `(2 L)^h`, which agrees with the circuit on `1 - (S/{2^L})` fraction of the inputs. Given a depth `d` circuit `C` we then set `2L = n^{1/(2d)}` to obtain a `sqrt(n)` degree polynomial that agrees with `C` on `1 - (S/{2^{1/2n^{1/(2d)}}})` fraction of the inputs.
2. Show that no `sqrt(n)`-degree polynomial agrees with `MOD_2` on more than `49/50` fraction of the inputs
The two steps imply any depth `d`, `AC[3]`-circuit for `MOD_2` must have size at least `S > 2^{1/2n^{1/(2d)}}/50`.

Proof of Step 1

Let `v` be a depth `h` gate in our circuit. Let `v` compute the function of the inputs, `g(x_1, ..., x_n)`.
We want a mod 3 polynomial `\tilde{g}(x_1, ..., x_n)` of degree `(2L)^h`, such that `g(x_1, ..., x_n) = \tilde{g}(x_1, ..., x_n)` for most inputs in `x_1, ..., x_n in {0,1}`.
The numbers mod 3 can be view as `{-1, 0, 1}`.
We will ensure for inputs from `{0,1}^n`, that `\tilde{g}(x_1, ..., x_n) in {0,1}`. This can ensure by squaring any polynomial that otherwise works as `-1^2=1`.
The approximating polynomial is constructed by induction on `h`.
When `h=0`, the gates is an input wire `x_i` and so the approximating polynomial is just `x_i`.
...

Proof of Step 1 - continued

Suppose we have constructed approximators for all gates up to `h-1`. Let `g` be a gate of height `h`.
1. If `g` is a NOT gate, then `g = neg f_1` for some gate `f_1` of height `h-1`. Let `\tilde{g} = 1 -tilde{f_1}`. Then whenever `f_1 = tilde{f_1}`, we'll have `g = \tilde{g}`, so this case does not introduce error.
2. If `g` computes `MOD_3(f_1, ..., f_k)` where the `f_i`'s are of height `h-1`, we set `\tilde{g} = (sum_{i=0}^k \tilde{f}_i)^2`. The degree of `\tilde{g}` is `2 times (2L)^{h-1} < (2L)^{h}`. Further, since `(-1)^2 = 1` and `0^2 = 0`, this case does not introduce error.
3. If `g` compute `vv_{i=0}^k(f_1, ..., f_k)`, the naive approach would be to set `tilde{g} = 1 - prod_{i=1}^k(1 - \tilde{f}_i)`. This would multiply the degree by `k`, in general, `O(n^c)` for `c \geq 1`, which would greatly exceed `(2L)`, which is `O(n^{1/(2d)})`. We know `g` is `1` iff at least one of its input `f_i`'s is 1. The random subsum principle says if there is some `i` such that `f_i=1`, then the sum of a random subset of `{f_i(x)}` will be 1 (mod 3) with probability `1/2`. The principle is true because a such a subset has a probability 1/2 of containing `f_i`.
  
  To compute `tilde{g}`, randomly pick `L` subsets `T_1, ... T_L` of `{1, ..., k}`. Compute the `L` polynomials `(sum_{j in T_1} \tilde{f}_i)^2, ... ,(sum_{j in T_L} \tilde{f}_i)^2`. The degrees of these are at most twice those of the `\tilde{f}_i`'s. Compute the OR polynomial of these using the naive approach to get a polynomial of degree at most `(2L) times (2L)^{h-1}`. For any input `x`, by using what we said above about the random subset principle, that the polynomial given by a random choice of subsets `T_i` differs from the OR of the `\tilde{f}_i`'s is at most `1/2^L`. If something happens with nonzero probability, there exists a situation in which it occurs, so there exists a choice of subsets `T_i` such that the probability so that the polynomial differs from the OR of `\tilde{f}_i`'s is at most `1/2^L`. We choose such a subset to define `\tilde{g}`.
4. If `g` compute `^^_{i=0}^k(f_1, ..., f_k)`, use De Morgan's law to handle using `neg` and `vv` approximations.
The complete process above give us an approximator for the output gate of degree `(2L)^d`.
Each operation introduces an error on at most `1/2^L` fraction of the inputs.
So the total error in approximating a circuit of size `S` is at most `S/2^L`.

Proof of Step 2

Claim. Let `f` be a `sqrt(n)` degree polynomial that agrees with the `MOD_2` function for all inputs in `G' subseteq {0,1}^n`. Then `|G'| < (49/50)2^n`.

Proof.

Perform the substitution `y_i = 1 + x_i (mod 3)` on `f`. This changes its domain from `{0,1}^n` to `{-1, 1}^n`. The resulting polynomial still has degree `sqrt(n)`. The subset `G'` is transformed to a subset `G` of `{-1, 1}^n` such that `|G'| =|G|`.
Under this substitution `MOD_2` is transformed to`prod_{i=1}^n y_i`. Thus, a degree `sqrt(n)` polynomial agrees this product of `n` things on inputs from `G`.
Let `F_G` be the set of all functions `S: G -> {-1, 0 ,1}`. So `|F_G| = 3^{|G|}`. We next show `|F_G| \leq 3^{(49/50)2^n}`, as the size of `|G'| = |G|` is at most `log_3 |F_G| = (49/50)2^n` this proves step 2.

Proof of Step 2 - continued

Subclaim. For every `S in F_G`, there is a polynomial `g_S`, which is the sum of monomials `a_{I}\prod_{i in I}y_i` where `|I| \leq n/2 + sqrt(n)` such that `g_S(x) = S(x)` for all `x in G`.

Proof. Let `\hat{S}: {-1,0, 1}^n -> {-1,0, 1}` be any function which agrees with `S` on `G`. Then `hat{S}` can be written a polynomial in the variables `y_i`. We are interested in the case where `(y_1, y_2, ..., y_n) in {-1, 1}^n`. So `(y_i)^2 =1` and every monomial `prod_{i in I} (y_i)^{r_i}` has `r_i leq 1`. Thus, `hat{S}` has degree at most `n`. Suppose `prod_{i in I} y_i` is monomial term in `\hat{S}` of degree `|I| > n/2`. We can rewrite this as:
`prod_{i in I} y_i = prod_{i=1}^n y_i \cdot prod_{i in \bar{I}} y_i.`
Since we are looking at values in `G`, a set on which `MOD_2 = prod_{i=1}^n y_i` can be approximated by a degree `sqrt(n)` polynomial, the right hand size has degree at most `n/2 + sqrt(n)`. So all monomials in `hat{S}` have at most this degree Q.E.D. Subclaim.

Given this, let's calculate `|F_G|`. The number of possible polynomials `g_S` is at most 3 to the power of the number of possible monomials, where the 3 comes from our choices for the coefficient in front a monomial.
The number of possible monomial is
`|{I subset [n] : |I| \leq n/2 + sqrt(n)}| = sum_{i=0}^{n/2 +sqrt(n)}((n),(i))`.
Using bounds on the tail of the binomial distribution it can be shown that the sum is less than `(49/50)2^n`. Q.E.D. Claim.

In-Class Exercise

Give an upper bound on the circuit size needed to compute parity using unbounded fan-in AND, OR, MOD3, gate and NOT gates.
Does the circuit family you constructed used `MOD_3` gates?
Post your solutions to the May 10 In-Class Exercise thread.

Some Concluding Remarks

Notice in the process of carrying out this proof, we approximated AND, OR, NOT, MOD_3 circuits with polynomials.
Given a sequence of input output pairs for a function we know to be computed by an `AC^0[q]` , we can try to compute the simplest polynomial of the kind we have been considering that approximates it.
From this we might then reconstruct an `AC^0[q]` circuit.
It turns out there has been recent work that shows that you can actually do this. I.e., the lower bound gives a learning algorithm.
In fact, it turns out that most of the lowers bounds that have been proven so far involve some kind of approximation-like property (natural property) that can be used to get a learning algorithm.
On the other hand, Razborov and Rudich have shown, that in general, assuming the existence of strong pseudo-random number generators that this is no such natural property that can be used to prove lower bounds for all of `P`/`poly`.

`AC^0[p]` Lower Bounds

Outline