Lectures in Computational Complexity

Jin-Yi Cai
Department of Computer Sciences
University of Wisconsin
Madison, WI 53706
Email: [email protected]

January 28, 2003

2
Contents

1 Genesis 5
1.1 Turing Machines and Undecidability . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Time, Space and Non-determinism . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 Hierarchy Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 Complexity Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.5 Reductions and NP-Completeness . . . . . . . . . . . . . . . . . . . . . . . . 15
1.6 Polynomial Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.7 Counting Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.8 Complexity Classes - Review . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2 Space Bounded Computation 17

2.1 Configuration Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Savitch’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Immerman-Szelepcsenyi Theorem . . . . . . . . . . . . . . . . . . . . . . . . 19

3 Randomization 21
3.1 Randomized Algorithms – Examples . . . . . . . . . . . . . . . . . . . . . . 21
3.2 Probabilistic inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3 Randomized Complexity Classes . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.4 Sipser-Lautemann theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.5 Universal Hashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.6 BPP ⊆ Σp2 - Revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4 Sparse Sets 25
4.1 Sparse set, Polynomial circuits, P/poly . . . . . . . . . . . . . . . . . . . . . 25

3
4 CONTENTS

4.2 Polynomial Circuits for BPP . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.3 Karp–Lipton Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.4 Mahaney’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

5 Hartmanis Conjectures 29

6 Interactive Proof Systems 31

6.1 Interactive Proofs – An Example . . . . . . . . . . . . . . . . . . . . . . . . 31
6.2 Arthur-Merlin Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.3 Merlin–Arthur Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.4 AM With Multiple Rounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.5 AM and Other Complexity Classes . . . . . . . . . . . . . . . . . . . . . . . 38
6.6 LFKN Protocol for Permanent . . . . . . . . . . . . . . . . . . . . . . . . . . 40

7 IP=PSAPCE 45

8 Derandomization 47
8.1 Pseudorandom Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
8.2 One-way Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
8.3 Goldreich-Levin Hardcore Bit . . . . . . . . . . . . . . . . . . . . . . . . . . 53
8.4 Construction of Pseudorandom Generators . . . . . . . . . . . . . . . . . . . 55

9 Circuit Lower Bounds 57

9.1 Approximating Constant Depth Circuits by Low Degree Polynomials . . . . 57
9.2 Low Degree Polynomials Cannot Approximate Parity . . . . . . . . . . . . . 60

10 Switching Lemmas 61
Chapter 1

Genesis

Chapter Outline: Brief overview of complexity theory. Hilbert’s Tenth Problem. Turing
Machines. Undecidability. Cantor’s method of diagonalization. Undecidability of the halting
problem. Time and space bounded Turing machines. Hierarchy Theorems. Complexity
Classes (P, NP, PSPACE). NP-Completeness. Polynomial Hierarchy. Counting class #P.
Easy containment among “classical” complexity classes.

1.1 Turing Machines and Undecidability

There have always been two major strands of mathematical thought since the time of antiq-
uity: Structural Theory and Computational Methods. For example, Euclid’s Elements is a
synthesis of much that was known in geometry up till that time, it is also largely structural
in that its emphasis (and establishes) theorems and deductive proofs, however the writing
of Diophantus was primarily algorithmic. Of course these strands of mathematical thought
are not in opposition to, but rather, complement each other. Even in Euclid’s Elements, one
finds algorithmic gems such as The Euclidean Algorithm which finds the greatest common
divisor of two positive integers. It is a shining example of an early triumph in algorithm
design, whose correctness and efficiency demands proof in a purely structural sense. Outside
of the Greek tradition, other ancient civilizations also had various emphasis on either the
Structural Theory which prizes the framing and proof of general theorems by deductive rea-
soning, or the Computational Methods which seek efficient computational method to solve
problems.
To a great majority of the classical masters throughout history, starting with Archimedes,
Newton, Leibniz, Euler, Lagrange, Gauss, . . ., computation is an inseparable aspect of math-
ematics as a whole. It is a rather recent phenomenon that the computation is somehow del-
egated as secondary to the Big Math edifice, perhaps helped along by the influential schools
such as Bourbaki. But in reality, much of the motivation for the big structural discoveries
were computational originally. For example, Calculus was invented so as to facilitate com-
putation of orbits of heavenly bodies as well as measuring surface areas and volumes; Galois

5
6 CHAPTER 1. GENESIS

Theory and finite group theory was a discovery to investigate the solvability of equations by
radicals; the Prime Number Theorem was first conjectured by Gauss after much computa-
tional experiments. It is also true that much of the advances made in structural mathematics
had also greatly influenced the advances in computational mathematics.
While it can be said that the subject of algorithms is as old as mathematics itself, the
study of algorithms as a subject, rather than the use of them, is a relatively new development.
Perhaps one could trace this beginning to set theory, that most structural of all subjects.
In his study of Fourier series (surely one of most computational subject in origin), Cantor gave
birth to a set of ideas that we now call (naive) set theory. Cantor’s ideas are revolutinary in
many aspects. In its basic framework it is highly non-constructive. For example, Cantor gave
a conceptually crisp and simple proof of the existence of transcendental numbers, whereby
inventing his famous diagonalization method. This proof is remarkable in many ways:
Firstly, it is much simpler than the monumental achievement of Hermite and Lindemann
on the transcendence of e and π respectively. Perhaps one can still make the case that
the “real” transcendental number theory is more along the lines of Hermite, Lindemann
and Liouville, and not the mere existence proof by the magic of diagonalization. But even
the most dedicated practitioners of “hard analysis” today will not dismiss the elegance and
efficiency of Cantor’s method. On the other hand, today many interesting computational
problems, such as basis reductions for lattices, simultaneous Diophantine approximations,
and volume estimations of convex bodies, form very active research areas which can be traced
directly to the work such as Dirichlet, Liouville, Hermite and Minkowski.
Secondly, as Kronecker was quick to point out, Cantor’s method is inherently non-
constructive, and in his view, borders on the “philosophical”. In particular it did not conform
to the strictly finitistic and constructive approach that Kronecker had been advocating. To
the end of his day, Kronecker never accepted Cantor’s idea. The finitists distrust it on philo-
sophical ground, which is ironic because the finitists are particularly concerned with the
soundness of mathematical foundation, which is to be demonstrated in coming years to be
closely related to computational undecidability, in which Cantor’s diagonalization method is
a forerunner.
Thirdly, the diagonalization method was to find its great application in Turing’s undecid-
ability proof of the Halting Problem. It subsequently became one of the basic mathematical
tools in recurcsion theory, and in the founding of complexity theory with the proof of the
time and space hierarchy theorems.
Because of its fundamental importance we will give the diagonalization proof by Cantor.
An algebraic number is a root of a polynomial with integral coefficients. A non-algebraic
number is called a transcendental number. A set is countable if it can be put into one-to-
one correspondence with the integers. It is clear that the set of all algebraic numbers is
countable, since we can count all integral polynomials, and each polynomial of degree n has
at most n roots.

Theorem 1.1 The set of real numbers is uncountable; in particular, there are non-algebraic
real numbers.
1.1. TURING MACHINES AND UNDECIDABILITY 7

A curious historical note: In order not to offend Kronecker, who was powerful and some-
what petty at the same time and might block the publication of this work, Cantor had to
phrase his main result strictly on the existence of non-algebraic numbers.
Proof Consider all binary infinite sequences B = {β}, where

β = b 1 b2 . . . bn . . . ,

and bi ∈ {0, 1}.

Claim: B is uncountable.
Suppose otherwise. Let β1 , β2 , . . . , βn , . . . , be an enumeration of all B, where each βi =
bi,1 bi,2 . . . bi,n . . .. Now write these sequences by the rows, and we obtain an infinite table as
follows. table here!!!

Now we define a particular β = b1 b2 . . . bn . . . ∈ B, which by its definition can not be on

this list: We go down the diagonal of this infinite table; for the n th entry, if it is 0 we let
our new bn = 1, and if it is 1 we let bn = 0. Formally, ∀n ≥ 1,
(
0 if bn,n = 1
bn =
1 otherwise.

Thus we “constructed” our β ∈ B to be “disagreeable”: It disagrees with the n th item on

the list in the n th place. Hence this β cannot be among those listed. ♠
It takes a little further fiddling to show that this B can be “essentially” identified as the
set of all real numbers between 0 and 1, via the binary expansion. 1
It is self evident why Cantor’s method is called the diagonalization method.
Kronecker’s objections not withstanding, Cantor’s set theory opened up a mathematical
“paradise”, from which, Hilbert was said to have remarked that mankind will never be driven
out again. Nevertheless, it did bring troubles with it. In particular, at the turn of the last
century, a number of set theoretic paradoxes were found that pertain to the foundation of
mathematics.
1
There is some overlap, on those binary expansions that end with all 0’s or all 1’s. However that overlap is
inconsequential, since it is countable. But we will not dwell on such inconsequential matters, as our purpose
here is to introduce the diagonalization method.
8 CHAPTER 1. GENESIS

Here is the famous Russell’s paradox.

“Surely” pure logic dictates that a set A either satisfies A ∈ A or A 6∈ A. Let’s form
A = {A | A 6∈ A}. In Cantor’s naive set theory this seems perfectly legitimate. If, Russell
says, we can form this set A, then is A ∈ A? If A ∈ A, then by its definition, A 6∈ A.
However, if A 6∈ A, then again by definition, A ∈ A.
At the turn of the last century, paradoxes such as this one stirred up a lot of uneasi-
ness with the foundation of mathematics, which was to be formulated increasingly based on
Cantor’s naive set theory. This created the strong desire to re-examine the foundation of
mathematics, and try to formalize all mathematics on an axiomatic basis without contradic-
tion. (In the modern axiomatic set theory, such paradoxes are avoided by being more careful
as to what is admissible as a set construction.)
Hilbert was a leading advocate of the formalist school. In his famous address to the
International Mathematicians Congress in 1900, he listed 23 problems which he thought to
be most likely to excite the imagination of mathematicians worldwide in the new century.
The 10th problem on the list asks the following: Find a systematic procedure to determine
if a polynomial with integral coefficients in several variables has an integral solution. More
broadly then, Hilbert initiated the study of Decision Problems where the aim is to find
an algorithm to decide each instance of a problem. Several other Hilbert problems also
had a profound impact on the future development of the foundation of mathematics and
computability theory, such as the Continuum Hypothesis, but it was the Decision Problem,
a.k.a. Entscheidungsproblem, that was the primary impetus to Turing’s seminal work which
established the foundation of computability theory.
It turns out that the notion of a systematic procedure to compute is intimately related to
the notion of a formal proof in axiomatic system. The work of Turing, Gödel, Church, Post,
and others, established that the original program envisioned by Hilbert cannot be complete.
While Hilbert only asked for an algorithm for the 10th problem, the possibility of the non-
existence of such an algorithm probably never crossed his mind, this turned out to the the
case in the end. But, it was Hilbert who raised the question, and focus the attention on the
very notion of what constitutes an algorithm, what is computation. In answering Hilbert,
computability theory was born.
A Turing Machine has a finite state control and an infinite input/output tape with a
reading/writing head. A deterministic Turing Machine moves in each step in a unique way
determined by its current state, and the symbol it is currently scanning. A move consists
of a possible change of state and scanned symbol, and moving the head left or right. A
non-deterministic Turing Machine may have several legal moves, which are still determined
by its current state and the scanned symbol. An input x is accepted by a deterministic
Turing Machine if the computation starting with initial configuration ends in an accepting
state. For a non-deterministic Turing Machine acceptance is defined by the existence of some
sequence of legal moves that ends in an accepting state. For space bounded Turing Machines
with sublinear space bound we allow a separate read-only input tape, and a work tape only
on which the space bound is counted. We assume the readers are already familiar with these
notions; for detailed definitions, please see [?, ?, ?]. The Church-Turing thesis states that
1.2. TIME, SPACE AND NON-DETERMINISM 9

whatever can be computed can be computed by a Turing Machine.

Cantor’s diagonalization method was technically the opening salvo by Alan Turing.
Adapting Cantor’s method, Turing showed that there are problems which can not com-
puted by any Turing Machine, and thus, by the Church-Turing thesis, uncomputable by
any algorithm whatsoever. Such problems are called undecidable problems. In particular,
the so-called Halting Problem for Turing Machines is one such problem. Furthermore, a
reduciton theory is developed whereby a host of problems can be shown undecidable.
The proof of the undecidability of the Halting Problem goes as follows.
List all the Turing Machines M1 , M2 , . . . row by row, and mark the outcome of Mi (j),
the outcome of machine Mi on input j, on the j th entry of that row for Mi . Thus we obtain
an infinite table, much like that in Cantor’s proof.
Now suppose there is a decision procedure in the form of a Turing Machine M that,
for any hi, ji, can decide whether Mi (j) halts or not in a finite number of steps. Let’s say
M (hi, ji) = 1 if Mi (j) halts and = 0 otherwise. Then one can design another Turing Machine
M 0 as follows: On any i, M 0 (i) simulates M (hi, ii). If M (hi, ii) = 1, then M 0 (i) enters a loop
and never halts. If M (hi, ii) = 0, then M 0 (i) halts.
Now since all Turing Machines have been enumerated, there exists a k, such that our M 0
is Mk . But what happens to Mk (k)?
If Mk (k) eventually halts, then by assumption on M , M (hk, ki) = 1, and thus M 0 (k),
which is Mk (k) never halts. If Mk (k) does not halt, then again by assumption, M (hk, ki) = 0,
and M 0 (k) halts. Either way we have a contradiction.
Next we will define time and space bounded computation. This is the domain of compu-
tational complexity theory. We will see that diagonalization method reappears, to establish
that more time or more space provably can compute more.

1.2 Time, Space and Non-determinism

The following notions are basic, and can be found in more details in any reference books
[DK].
We use multitape TMs. The time complexity of machine M is

timeM (n) = max{# of steps in M (x), for |x| = n}.

For any function f (n), DTIME[f ] = {L | for some M , L(M ) = L, andtimeM (n) ≤ f (n), for all large n.}.
For technical reasons we will only consider “nice” functions f (n), called fully time-constructible
functions. This means that there is some TM M , which for every n, and any input x of size
n, M (x) runs in exactly f (n) steps and then halts. Almost any reasonable functions ≥ n,
k k n
such as n, nk , ni (log n)j , 2(log) , 2n , and 22...2 . 2 In other words, DTIME[f ] consists of
2
It is a fact in complexity theory that there exist functions which are not fully time-constructible, but we
will not be concerned with that.
10 CHAPTER 1. GENESIS

problems that are computable by some TM with running time at most f (n) asymptotically.
By some silly tricks such as enlarging the alphabet set, and the set of finite states, one can
show that any constant factor in f (·) does not matter.
The model of TM is chosen because it is relatively robust. One can show, for in-
stance, that any k-tape TM running in time f (n) can be simulated by a 2-tape TM in
time O(f (n) log f (n)). For our purposes we will only need the more trivial simulation in
time O(f (n)2 ), even by 1-tape TM. This simulation can be seen easily as follows: Devide
the single tape into 2k tracks, and use a large alphabet set with more than 22k symbols, say.
Keep on the single tape the configurations of all k tapes, together with a mark for each head
position. Then one step of the computation of the the k-tape TM is simulated by the 1-tape
TM with 2 sweeps. Note that each sweep of the tape area used takes at most f (n) steps.
Let poly denote the class of polynomials, or simply ni + i, i = 1, 2, . . .. Then the union
[
P = DT IM E[f ]
f ∈poly

is the class of deterministic polynomial time. Clearly this definition is invariant when re-
stricted to 1-tape TMs. One can simularly define exponential time classes
[
E= DT IM E[2kn ]
k>0
[ k
EXP = DT IM E[2n ]
k>0

One can define space complexity similarly. In one aspect it is even simpler, since we can
use k tracks to mimic k tapes and there is no additional space overhead in the simulation.
So we will have one work tape. However in another respect, there is a slight complicaiton.
This happens when we wish to study sublinear space complexity, which consists of important
problems. In order to account for sublinear space, one uses a separate read-only input tape,
in addition to a read-write work tape. On the read-only input tape, the input of length n is
written, but these n cells do not count toward space complexity. We only count the number
of tape cells used on the read-write work tape. Thus for space complexity, the standard
model is what is known as an off-line TM, which has one read-only input tape, and one
read-write work tape. Then one can define, in an obvious way,
spaceM (n) = max{# of cells on work tape used in M (x), for |x| = n}.
Again we restrict to “nice” functions, called fully space constructible functions. (What’s a
suitable deinition?) For any such function f (n), define DSPACE[f ] = {L | for some M , L(M ) =
L, andspaceM (n) ≤ f (n), for all large n.}.
We define [
P SP ACE = DSP ACE[f ],
f ∈poly

the class of polynomial space. (There is a reason why we omit the word “deterministic”,
as we shall explain later.) We also have deterministic logarithmic space L = DSPACE[log].
S
Note again that constant factors do not matter, thus DSPACE[log] = c>0 DSPACE[c log].
1.2. TIME, SPACE AND NON-DETERMINISM 11

The notion called non-determinism is of central importance in complexity theory. This

notion is related to the process of guess and verification. Consider the following two ex-
amples. Suppose we are given a Boolean formula ϕ in propositional logic, i.e., it is a
well-formed-formula made up from logical AND (∧), OR (∨) and NOT (¬), and Boolean
variables x1 , x2 , . . . , xn . The problem is whether there is a truth assignment for the vari-
ables x1 , x2 , . . . , xn such that ϕ evaluates to True. This problem is called the Satisfiability
Problem. The set of all satisfiable formulae is denoted as SAT.
No polynomial time algorithm is know for SAT. But whenever a formula ϕ(x1 , x2 , . . . , xn )
is satisfiable, i.e., ϕ ∈ SAT , there is an assignment, if somehow one is given that assignment,
one can easily verify that ϕ is satisfiable, in particular it evaluates to True under that
assignment. We consider this process as consisting of two stages, first guess a witness,
in this case a satisfying assignment, second verification, to check indeed it is a satisfying
assignment. Note that ϕ is satisfiable if and only if such a guess exists which will pass the
verification stage. Of course, formally one can consider this guess and verify stages go hand
in hand. In this Satisfiability Problem for example, we can imagine one guesses one bit at a
time for each variable, and evaluate the formula as each variable is assigned. We will formally
define non-deterministic TM (NTM) as TM’s where for each state and tape symbol being
read (q, A), the transition function δ provides a finite set of allowable next-step transitions
consisting of a state q 0 , a tape symbol A0 and a left or a right movement, ∆ = L or R,
of the tape head. A NTM is defined by this set of quintuples of the form (q, A, q 0 , A0 , ∆).
For DTM, every (q, A) has at most one quintuple of the form (q, A, q 0 , A0 , ∆) signifying at
most one legal move; with NTM, each (q, A) can have a finite set of quintuples the first
two entries being q and A (this finite set if defined as part of the definition of δ). These
two views of non-deterministic computations are clearly equivalent for SAT. For problems
such as SAT, or in general, for non-deterministic time classes (which is at least linear cn),
the guess and verify is more intuitive. However, for formal treatment of non-deterministic
computations, it is easier with the second approach. This is especially so when one is dealing
with possibly non-terminating computations. 3 Another place the second approach is more
suitable is when we consider space bounded computation with sub-linear space bound as
in the following example (thus there is not enough space to guess and write down all the
non-deterministic moves at the beginning.)
Consider the following Graph Accessibility Problem: Given a directed graph G, and two
vertices s and t, we ask whether there is a directed path from s to t. We denote by GAP
the set of all instances (G, s, t) where a directed path exists in G from s to t. (This problem,
when restricted to undirected graphs, is also very important. By replacing each undirected
edge by a pair of directed edges in the opposite direction, it is clear that the directed GAP
is a more general problem than the undirected version.)
For GAP, there is a simple polynomial time algorithm based on Depth-First-Search.
However that also uses at least linear space. By contrast, if the space bound is limited, we
have the following non-deterministic algorithm: Start from v0 = s, successively guess the
3
We do not concern ourselves much in these lectures. We will not be concerned with non-terminating
computations. For example, one can design a subroutine which acts as a time clock that runs exactly n i + i
steps and shuts itself off, or a space bound which marks f (n) (e.g. f (n) = log n or ni + i) many tape squares.
12 CHAPTER 1. GENESIS

next vertex vi and verify that (vi−1 , vi ) is an edge. We can keep a counter to count up to n,
and accept iff within n steps, some vk = t. Note that we only need to keep at all times the
pair of current vertices on the work tape; when we guess for vi+1 we need only to remember
vi (in order to verify that (vi , vi+1 ) is an edge), but we no longer need to keep vi−1 and
all previously guessed vertices. In addition we only need to keep a counter to count up to
n. For a graph with n vertices, to name a vertex takes only log n space. Thus GAP is in
non-deterministic logspace.
We will only be concerned with time/space bounded NTM, and thus we can assume each
computational path terminates within the specified time/space bound. Then for a NTM N
and a fully time constructible f we define

timeN (x) = max{n + 1, min # of steps in N (x), and N (x) accepts along p}},
p

timeN (n) = max{timeN (x), |x| = n},

N T IM E[f ] = {L | for some N , L(N ) = L, andtimeN (n) ≤ f (n), for all large n.}.
Of course L(N ) = L means that x ∈ L iff there is some computation path p, along which N
accepts x. Similarly we can define non-deterministic space classes.

[
NP = N T IM E[f ],
f ∈poly

[
N P SP ACE = N SP ACE[f ],
f ∈poly

and also [
NL = N SP ACE[c log].
c>0

To recap, formally a non-deterministic TM (NTM) is simply a TM which has possibly

more than one legal moves at any configuration. These next-moves are specified by the
current state and the tape symbols it is currently reading. A NTM accepts an input x iff
there exists a legal computational path that ends in acceptance.
It is known that L ⊆ N L ⊆ P ⊆ N P ⊆ P SP ACE ⊆ EXP .
We will show later that N P SP ACE[f ] ⊆ DP SP ACE[f 2 ], thus N P SP ACE = P SP ACE.
For the time complexity version, the relationship of P and N P is the most outstanding
open problem in Computer Science.

1.3 Hierarchy Theorems

In Theorem ?? we showed the existence of undecidable languages. Next we extend the proof
to show that for any time bound T (n), there is a decidable language that cannot be decided
in time O(T (n)).