Complexity

Growth Rates

Assume f, g: NAT -> NAT are increasing functions (n <= m => f(n) <= f(m)).

We want to formalize the statement:

growth rate of f <= growth rate of g

Definition: g majorizes f, notation: f = O(g), if for some c & k: f(n) <= c * g(n) for k < n

Example 1:

100n² + 1000 = O(n²) because 100n² + 1000 <= 101n² for n > sqrt(1000)

Also:

n² = O(100n² + 1000)

Example 2:

n^k = O(n^k+1)

Example 3:

n^k = O(2ⁿ)

Example 4:

log₂(n) = O(n)

Example 5:

n != O(1)

Theorem Assume f & g are increasing, then f = O(g) if and only lim_n->infinity(n)/g(n) < infinity

Example:

n³ != O(n²) because lim(n³/n²) = lim(3n²/2n) = lim(6n/2) = infinity

The hyper-exponential hierarchy

hyper₀(n) = (n == 0)? 1: (n == 1)? 2: n + 2

hyper_k+1(n) = hyper_k⁽ⁿ⁾(1) = hyper_k(hyper_k(hyper_k(...hyper_k(1)..))) n-times

And so...

hyper₁(n) = hyper₀(hyper₀(hyper₀(...hyper₀(1)..))) (n-times) = 2 + 2 + 2 + ... + 2 (n times) = 2 * n

hyper₂(n) = hyper₁(hyper₁(hyper₁(...hyper₁(1)..))) = 2 * 2 * ... * 2 * 1 (n times) = 2ⁿ

hyper₃(n) = 2^(2^(2^...(2^1)...)) n-times

Obviously, hyper_k is a rapidly increasing function.

Theorem hyper_k = O(hyper_k+1) but hyper_{k + 1} != O(hyper_k)

Definition ack(n) = hyper_n(n)

Theorem For each k hyper_k = O(ack) but ack != O(hyper_k)

The classes O(hyper_k) form an expanding hierarchy among increasing functions in NAT->NAT:

Theorem

If p is any polynomial, then p = O(hyper₂), but hyper₂ != O(p)

Exercise: Implement hyper_k in PL – {loop}

Exercise: Implement ack in PL

Note: This is hard and may require adding stacks or arrays to the PL VM

Algorithm Complexity

Assume P is a PL program that implements a function f: NAT -> NAT.

T_P(n) = the number of instructions from P executed by the PL VM given input n

S_P(n) = the number of variables used by P given input n

Definition The depth of a PL program = 1 if it contains no loop instructions, otherwise it's 1 + maximum depth of all loop bodies it contains.

Definition LOOP(n) = all LOOP programs with depth <= n. In other words, loops can only be nested <= n times

Theorem hyper_n can be implemented by a LOOP(n) program, but not a LOOP(n – 1) program

Proof

We show hyper_n can be implemented by a LOOP(n) program using induction.

Clearly hyper₀(m) can be implemented in LOOP(0):

inc m
inc m

Let load a, hyper_n(m) be a macro that expands in to a LOOP(n) program by inductive assumption. Then hyper_n+1(m) is computed by

load r, 1
loop m
   load r, hyper_n(r)
end

To show that this can't be done in LOOP(n-1) follows from the next theorem.

QED.

Theorem If P is a LOOP(n) program, then T_P = O(hyper_n)

Proof (sketch)

What we will show by induction on n is that there is a k such that for T_P(V) <= hyper_n^(k)(V) where V = max(vars)

Assume n = 0. But then P has no loops but then P has no loops so T_P(V) = |P| = the number of instructions in P, which we take to be our k.

Assume the result is true for n, and P is a LOOP(n + 1) program of the form loop y P' end where P' is a LOOP(n) program.

So the first time P' is executed requires at most hyper_n^(k)(V) steps.

Fact 1: For an program P, max(vars) <= T_P(max(vars)) + max(vars) since a step can at most increment a variable by 1.

From this fact it follows that:

max(vars) <= hyper_n^(k)(V) + V.

Fact 2: hyper_n^(k)(V) + V <= hyper_n^{(k + 1)}(V) for 1 <= n

From this fact we see that the next execution of P' requires at most hyper_n^(k)( hyper_n^(k+1)(V)) steps and from fact 1 we have:

max(vars) <= hyper_n^(k)(hyper_n^(k+1)(V)) + hyper_n^(k+1)(V)

Fact 3: hyper_n^(k)(x) + x <= hyper_n^(k+1)(x) for 1 <= n

From this fact it follows that

max(vars) <= hyper_n^(k)(hyper_n^(k+1)(V)) + hyper_n^(k+1)(V) <= hyper_n^(k+1)(hyper_n^(k+1)(V)) = hyper_n^2(k+1)(V)

This is also a bound on the number of steps required for the first two iterations of P'.

The number of iterations of P' is <= V and so

T_P(V) <= hyper_n^V(k+1)(V) <= hyper_n^V(k+2)(1) <= hyper_n^W(1) where W = hyper_n+1^(k+1)(V)

But:

hyper_n^W(1) = hyper_n+1(hyper_n+1^(k+1)(V)) <= 2 * hyper_n+1^(k+2)(V) <= hyper_n+1^(k+3)(V)

There are a few other cases, like when P is a sequence of loop P' end commands.

QED

Theorem If P is a PL – {goto} program, and T_P = O(hyper_n) , then there is a LOOP(n) program Q such that P and Q implement the same function.

Theorem ack can't be implemented in PL – {goto}

Theorem f is primitive recursive if and only if f can be implemented in LOOP.

Proof: We haven't defined primitive recursive, so this can be taken as a definition.