Time Complexity

We begin today by formalizing the notion of running time. As every nontrivial computational task requires at least reading the input, we count the number of basic steps of a computation as a function of the input length.

Definition. Let `f:{0,1}^star -> {0,1}^star` and let `T: NN -> NN` be some functions, and let `M` be a TM. We say that `M` computes `f` if for every `x in {0, 1}^star`, whenever `M` is initialized to the start configuration on input `x`, then after computing it halts with `f(x)` written on its output tape. We say that `M` computes `f` in time `T(n)` if its computation on every input `x` require at most `T(|x|)` many steps.

Example. The TM we gave last day for palindrome recognition runs in time `3(n+2)`. To see this recall it takes `n` steps to copy the input tape to a second tape, another `n` steps to rewind the input tape, and up to `n` steps to check going forwards on the input tape and backwords off the second tape whether the string was a palindrome.

Linear Speedup

When we think of microprocessors in our every day lives, we expect that a say 64-bit processor can compute things faster that a 32-bit processor. The next result illustrates that something like this holds for Turing machines. Notice that the speedup though is at most a linear factor.

Theorem. Let `f` be a Boolean function computable in time `T(n)`. Then for any `1 > \epsilon > 0`, `f` is also computable in `T'(n)` where `T'(n) = \epsilon T(n) + n + |~n/(|~6/\epsilon~|)~|`.

Proof. Let `M=(\Gamma, Q, \delta)` be a `k` tape machine that computes `f` in time `T(n)`. Recall that in our definition of TM we assumed `\Gamma` contains `0, 1, square, \Delta`, but it also might contain additional other symbols. We will get the speedup by making a simulating `k+1` tape machine `M'=(\Gamma', Q', \delta')` that encodes several moves of `M` by one move of `M'`. To be specific let `\Gamma'=\Gamma\cup \Gamma^m` -- we'll fix `m` in a moment. To begin `M'` scans the input `x= x_1 ... x_n` writing for each `m` adjacent symbols a single symbol in `\Gamma'` on the second tape. At the end of this `n+2` step process the second tape has `|~n/m~|` squares with symbols. We spend another `|~n/m~|` steps rewinding this tape. We thereafter treat the second tape as the input tape, and use the remaining `k` tapes to simulate the read/write tapes of the original machine. The machine `M'` then simulates `m` steps of `M` using 6 or fewer steps. At the beginning of a simulating stage the state of `M'` consists of a `k+1` tuple (really k+2, but we are ignoring the original input tape now) `(q, j_1,..,j_k)` where `q` is `M'`s state at the start of the stage, and the `j_i`s represent where within the current `m`-block on each tape Ms head would be. For each tape `M'` then scan one square left, two right, one left. It remembers in its controls the symbols it saw to the left and right. Using this information `M'` can now completely predict where `M` would go in its next `m` moves. It then updates the at most two tape square it needs to update and starts simulating the next. If `M` halts with a `1` or `0` on its output, so does `M'`. Simulating `T(n)` steps takes time `|x| + |~|x|/m~| +6|~(T(|x|))/m ~|` steps. So for `m = |~ 6/\epsilon ~|` the theorem holds.

"Good" Time Complexities

What should be legal functions `T` to consider when measuring run time? Weirdly oscillating functions might be difficult to reason about. The next definition tries to capture the notion of reasonable run time bound...

Definition. A function `T:NN -> NN` is time constructible if `T(n) ge n` and there is a TM `M` that computes the function `x mapsto lfloor T(|x|) rfloor` in time `T(|x|)`.

If a function is time contructible, then we can imagine have a TM go from the input, write a time bound on an auxiliary tape, and then use that tape to do a countdown that would also us to time a computation. As we will see, this can be useful in coming up with simulating algorithms.
Functions like: `n log n`, `n^2`, and `2^n` are all time constructible.
In this definition, we are assuming that we can expand the alphabet of our TMs and use this to speed up computations by up to a linear factor.

Example

Claim. The function `31(n+2)` is time constructible.

Proof. We will use a three tape machine to carry out the proof. On input `x`, our machine on the input tape will read one square of the input and advance to the right. It will do this until it reaches the first blank square. Then it will rewind the tape and move until it sees the start symbol. Then it will reverse direction again. It will scan the input 31 times in this fashion then halt giving the run time bound. As the first tape head is doing this we will be computing stuff, using the other two tapes, with the output tape eventually writing down the number `31(|x| + 2)`. On the first, left-right pass of the input tape, the output tape gets `0` (zero in binary written on it). In this same pass we write a string of length |x| of 1's on the second tape. While the input tape head continues to scan back and forth, we do the following routine on the other two tapes. Move left on the second tape by one, add 31 in binary to the output tape. If we are on the start symbol of the second tape, then we stop doing this routine. We claim that this routine will take fewer than `30(n+2)` steps, and so it will complete before we are done scanning the input tape back and forth 31 times. Since we are adding 31 in binary (that is, `11111`) `|x|+2` times, the output tape will have `31(|x|+2)` on it when the computation halts. To see the routine takes at most `30(n+2)` steps we note when we add 31, that `(n+2)/2` times we have to carry at most once any beyond the 5 digits in 31, `(n+2)/4` times we have to carry at most twice, `(n+2)/8` we need to carry at most three times, etc. Carrying `i` times plus moving the 5 digit positions of 31 in binary takes at most `2(i+5)` steps. So to add `31`, `n+2` times takes at most `\sum_{i=1}^{log n+5}(n+2)\cdot(2(i+5))/2^i` steps. For `i \geq 8`, `2(i+5) < (3/2)^i`, so
`\sum_{i=1}^{log n+5}(n+2)\cdot(2(i+5))/2^i = (12/2 + 14/2^2 + \cdots + 24/2^7)(n+2) + \sum_{i=8}^{log n+5}(n+2)\cdot (2(i+5))/2^i <`
`14(n+2) + \sum_{i=8}^{log n+5}((n+2)\cdot (3/2)^i)/2^i`
As `(3/4)^8 < 1/8`, the above is in turn less than
`14(n+2) + 1/8\sum_{i=0}^{log n+5}((n+2)\cdot (3/4)^i) =14(n+2) + (n+2)/8 \sum_{i=0}^{log n+5}(3/4)^i =`
`14(n+2) + (n+2)/8 (1 - (3/4)^{log n+5})/(1-3/4) < 14(n+2) + (n+2)/8 \cdot 4 < 15(n+2)< 30(n+2)`.

Quiz

Which of the following statements is true?

`n=o(n)`.
The input tape of the Turing Machine model we considered was read-only.
There are Boolean functions computable by a Java program that cannot be computed by a Turing Machine.

Robustness of our definition

The specific details of our definition of TM are quite arbitrary: Most changes to our model do not yield a substantially different model as our model can simulate any of these new models.
In computational complexity, however, we need to verify not only that one model can simulate another, but also it can do it efficiently.
We next give a few results of this kind with the intent of being able to conclude that the exact model is unimportant if we are willing to ignore polynomial factors in the running time.
We will only give proof sketches of these results.

Alphabet doesn't Matter

Claim. For every `f:{0, 1}^star -> {0, 1}` and time constructible `T: NN -> NN`, if `f` is computable in time `T(n)` by a TM `M` using alphabet `Gamma`, then it is computable in time `4 (log |Gamma|)T(n)` by a machine `bar{M}` using alphabet `{0, 1, square, Delta }`.

Proof. Suppose `M` has `k`-tapes and uses states `Q`. We will define an `bar(M)` which also uses `k` tapes but a new set of states `Q'`. The idea is that we will encode the symbols of `Gamma` as strings over `{0, 1}` of length `log |Gamma|`. Each tape cell on any tape of the original machine will now be represented by `log |Gamma|` cells on `bar{M}`.

To simulate one step `M`, `bar{M}` will (1) use `log |Gamma|` steps to read from each tape the `log |Gamma|` bits encoding a symbol `Gamma`, (2) use its state register to store the symbols read, (3) use `M`'s transition function to compute the symbols `M` writes and `M`'s new state information, (4) store this information in its state register, (5) use `log |Gamma|` steps to write the encoding of these symbols on its tapes, and (6) use up to `log |Gamma|` steps to move appropriate each of the simulating tape heads.

`bar{M}` needs to have access to a register that can store `M`'s states, the `k` symbols from `Gamma`, and a counter from `1` to `log|Gamma|` for reading symbols. So to make `bar{M}` we will need `O(|Q||Gamma|^{k+1})` states. The runtime bound follows from the time to simulate one step: That is, read symbol on each tape, write symbol on each tape, rewind to start of symbol, move to next symbol on each tape, if necessary. Each of these take `log |Gamma|` steps.

Number of Tapes doesn't Matter

Claim. Define a single-tape machine to be a TM that has only one read-write tape, that is used as input, work, and output tape. For every `f:{0, 1}^star -> {0, 1}` and time constructible `T: NN -> NN`, if `f` is computable in time `T(n)` by a TM `M` using `k` tapes, then it is computable in time `5k(T(n))^2` by a single tape TM `bar(M)`.

Proof. Let `x` be the input. We encode the `k` tapes on a single tape by using locations `|x|+1`, `|x|+k+1`, `2k+1`, ... of the single tape to code the squares of the first tape, locations `2`, `k+2`, `2k+2`, ... to encode the second tape, etc. For every symbol `a` in `M`'s alphabet `bar(M)` will have both the symbol `a` and `bar(a)`.

`bar(M)` does not touch the `n` input tape squares, but instead takes `O(n^2)` steps to copy the input bit by bit into the rest of the tape, using the encoding scheme above. S

To simulate one step of `M`, `bar(M)` makes two sweeps of its work tape. First, it sweeps the tape in a left-to-write direction and records to its register the `k` symbols marked with a bar. Then it uses `M`'s transition function to determine the new state, symbols, and head movements. Finally, it sweeps the tape back in the right-to-left direction to update the encoding accordingly.

After the carrying out its simulation, `bar(M)` will produce as output the encoding in the above way of the output of `M`. Since `M` does not read more than `T(n)` squares, `bar(M)` will never need to read more than `2n + kT(n) le (k+2)cdot T(n) le 2k cdot T(n)` locations, meaning that for each of at most `T(n)` steps, `bar(M)` performs at most `5 cdot k cdot T(n)` work. So the run-time bound follows.

Obliviousness doesn't Matter.

Remark. We can modify the proof of the claim of the last slide, so that `bar(M)`'s heads' movements do not depend on the input, but only on the input length. That is, for every input `x in {0,1}^star` and `i in NN`, the location of each of `M`'s heads at the `i`th step of execution on input `x` is only a function of `|x|` and `i`. A machine with this property is called oblivious.

Doubly-Infinite Tape Doesn't Matter.

Claim. Define a bi-directional TM to be a TM whose tapes are infinite in both directions. For every `f:{0, 1}^star -> {0, 1}` and time constructible `T: NN -> NN`, if `f` is computable in time `T(n)` by a bi-directional TM `M`, then it is computable in time `4T(n)` by a standard (unidirectional) TM `bar(M)`.

Proof. The idea is we will use the `i` tape square of a one way tape to represent both the squares `-i` and `i` of tape of a two-way tape.
Image of tape folding scheme
Our new machine's alphabet is `Gamma cup (Gamma times Gamma)` where `Gamma` was the original machines alphabet. Suppose `M'` is a `k'`-tape machine showing `T`'s time constructibility. Our simulating machine will use `k+1+k'` tapes where `M` had `k` tapes. `bar(M)` runs `M'` on its last `k'`-tapes for `T(|x|)` steps to compute `T(|x|)`. While it is doing this it converts its input into the second coordinates of the squares of the second work tape followed by `T(|x|) -|x|` blank ordered pairs, `(\square,square)`, and on the next `k-1` tapes it just writes `T(|x|)` many pairs `(\square,square)`. We rewind all of the tapes, having now taken `2T(|x|)` steps. We now treat the tape 1 through `k+1` tapes of `bar(M)` as simulating the tapes of `M`. To simulate a step `bar(M)` performs the corresponding operation that `M` did for that tape based on the coordinate that it is reading from. If `M` moves left off the `0th` position it switches to using the left coordinating for that tape and left moves of `M` will map to right moves of `bar(M)` and vice-versa. Similarly, if `bar(M)` more right off of the `0` position of a tape if switches to using the right coordinate for that tape. After at most `T(|x|)` steps carrying out this simulation, we need to spend an additional `T(|x|)` steps converting the output back from ordered pairs from `(Gamma times Gamma)` to symbols from `Gamma`.

TM Variants and Time Complexity

Outline