Design and Analysis of Algorithms Chapter 9

Lower Bounds

Chapter 11 Lower bound:

bound: an estimate on a minimum amount of work
needed to solve a given problem

Limitations of Examples:
Algorithm Power number of comparisons needed to find the largest element
in a set of n numbers
number of comparisons needed to sort an array of size n
number of comparisons necessary for searching in a sorted
array
number of multiplications needed to multiply two n-by-
by-n
matrices

Copyright © 2007 Pearson Addison-Wesley. All rights reserved.

Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-2

Lower Bounds (cont.) Methods for Establishing Lower Bounds

Lower bound can be
trivial lower bounds
• an exact count
• an efficiency class ( )
information--theoretic arguments (decision trees)
information
Tight lower bound: there exists an algorithm with the same
efficiency as the lower bound
adversary arguments
Problem Lower bound Tightness
sorting (nlog n) yes
problem reduction
searching in a sorted array (log n) yes
element uniqueness (nlog n) yes
n-digit integer multiplication (n) unknown
multiplication of n-by-
by-n matrices (n2) unknown

Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-3 Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-4

Trivial Lower Bounds Decision Trees

Trivial lower bounds:
bounds: based on counting the number of items Decision tree — a convenient model of algorithms involving
that must be processed in input and generated as output comparisons in which:
internal nodes represent comparisons
Examples
finding max element leaves represent outcomes
polynomial evaluation Decision tree for 3-
3-element insertion sort
abc
sorting yes no
a<b

element uniqueness
abc bac
yes no yes no
Hamiltonian circuit existence b< c a<c

a<b<c acb bca

b<a<c
Conclusions yes
a<c
no yes
b<c
no

may and may not be useful

be careful in deciding how many elements must be processed a<c<b c<a<b b<c<a c<b<a

Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-5 Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-6

1
Design and Analysis of Algorithms Chapter 9

Decision Trees and Sorting Algorithms Adversary Arguments

Any comparison-
comparison-based sorting algorithm can be represented Adversary argument:
argument: a method of proving a lower bound by
by a decision tree playing role of adversary that makes algorithm work the hardest
by adjusting input
Number of leaves (outcomes) n!
Example 1: “Guessing” a number between 1 and n with yes/no
Height of binary tree with n! leaves log2n! questions
Adversary: Puts the number in a larger of the two subsets
Minimum number of comparisons in the worst case log2n! generated by last question
for any comparison-
comparison-based sorting algorithm
Example 2: Merging two sorted lists of size n
log2n! n log2n a1 < a2 < … < an and b1 < b2 < … < bn
Adversary: ai < bj iff i < j
This lower bound is tight (mergesort) Output b1 < a1 < b2 < a2 < … < bn < an requires 2n
2n-1 comparisons
of adjacent elements

Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-7 Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-8

Lower Bounds by Problem Reduction Classifying Problem Complexity

Idea: If problem P is at least as hard as problem Q, then a lower Is the problem tractable
tractable,, i.e., is there a polynomial-
polynomial-time (O(
(O(pp(n))
bound for Q is also a lower bound for P. algorithm that solves it?
Hence, find problem Q with a known lower bound that can
be reduced to problem P in question. Possible answers:
yes (give examples)
Example: P is finding MST for n points in Cartesian plane
Q is element uniqueness problem (known to be in (nlog logn
n)) no
• because it’s been proved that no algorithm exists at all
(e.g., Turing’s halting problem)
problem)
• because it’s been be proved that any algorithm takes
exponential time

unknown
Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-9 Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-10

Problem Types: Optimization and Decision Class P

problem: find a solution that maximizes or
Optimization problem: P: the class of decision problems that are solvable in O(p
O(p(n)) time,
minimizes some objective function where p(n) is a polynomial of problem’s input size n

Decision problem:
problem: answer yes/no to a question Examples:
searching
Many problems have decision and optimization versions.
element uniqueness
E.g.: traveling salesman problem
optimization:: find Hamiltonian cycle of minimum length
optimization graph connectivity
decision:: find Hamiltonian cycle of length m
decision
graph acyclicity
Decision problems are more convenient for formal investigation of
their complexity. primality testing (finally proved in 2002)
Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-11 Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-12

2
Design and Analysis of Algorithms Chapter 9

Class NP Example: CNF satisfiability

NP (nondeterministic polynomial):
polynomial): class of decision problems whose Problem: Is a boolean expression in its conjunctive normal
proposed solutions can be verified in polynomial time = solvable form (CNF) satisfiable, i.e., are there values of its
by a nondeterministic polynomial algorithm variables that makes it true?

This problem is in NP
NP.. Nondeterministic algorithm:
A nondeterministic polynomial algorithm is an abstract two-
two-stage
procedure that: Guess truth assignment
generates a random string purported to solve the problem Substitute the values into the CNF formula to see if it
evaluates to true
checks whether this solution is correct in polynomial time
By definition, it solves the problem if it’s capable of generating and Example: (A | ¬B | ¬C
¬C)) & (A | B ) & (¬ B | ¬D | E ) & (¬D | ¬E)
¬E )
verifying a solution on one of its tries Truth assignments:
ABCDE
0 0 0 0 0
Why this definition? . . .
led to development of the rich theory called “computational 1 1 1 1 1
complexity” Checking phase: O(n)
Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-13 Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-14

What problems are in NP

NP?
? NP--Complete Problems
NP
Hamiltonian circuit existence A decision problem D is NP
NP--complete if it’s as hard as any
Partition problem: Is it possible to partition a set of n problem in NP
NP,, i.e.,
integers into two disjoint subsets with the same sum? D is in NP
Decision versions of TSP, knapsack problem, graph every problem in NP is polynomial-
polynomial-time reducible to D
coloring, and many other combinatorial optimization
problems. (Few exceptions include: MST, shortest paths) NP problems

All the problems in P can also be solved in this manner (but

no guessing is necessary), so we have: NP -complete
problem

P NP

Big question: P = NP ?
Cook’s theorem (1971): CNF
CNF--sat is NP
NP--complete
Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-15 Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-16

NP--Complete Problems (cont.)

NP P = NP ? Dilemma Revisited
P = NP would imply that every problem in NP, including all
Other NP
NP--complete problems obtained through polynomial
polynomial-- NP--complete problems, could be solved in polynomial time
NP
time reductions from a known NP
NP--complete problem If a polynomial-
polynomial-time algorithm for just one NP
NP--complete
NP problems
problem is discovered, then every problem in NP can be solved
in polynomial time, i.e., P = NP
NP problems
known
NP-complete
problem
candidate
for NP -
completeness NP -complete
problem

Examples: TSP, knapsack, partition, graph-

graph-coloring and
Most but not all researchers believe that P NP , i.e. P is a
hundreds of other problems of combinatorial nature proper subset of NP

Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-17 Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 11 11-18