Data Structures and Algorithms

Lecture 2: Analysis of Algorithms,

Asymptotic notation

Lilia Georgieva

© 2004 Goodrich, Tamassia

 Outline
 Pseudocode
 Theoretical Analysis of Running time
 Primitive Operations
 Counting primitive operations
 Asymptotic analysis of running time

Analysis of Algorithms 2
 Pseudocode
 In this course, we will
mostly use Example: find max
pseudocode to element of an array
describe an algorithm Algorithm arrayMax(A, n)
 Pseudocode is a high- Input: array A of n integers
level description of an Output: maximum element of A
algorithm
 More structured than currentMax ← A[0]
English prose for i ← 1 to n − 1 do
 Less detailed than a if A[i] > currentMax then
program currentMax ← A[i]
 Preferred notation for return currentMax
describing algorithms
 Hides program design
issues Analysis of Algorithms 3
 Pseudocode Details
Algorithm arrayMax(A, n)
 Control flow Input: array A of n integers
 if … then … [else …] Output: maximum element of A
 while … do …
 repeat … until … currentMax ← A[0]
for i ← 1 to n − 1 do
 for … do …
if A[i] > currentMax then
 Indentation replaces currentMax ← A[i]
braces
return currentMax

 Method declaration
Algorithm method (arg, arg…)
Input …
Output …

Analysis of Algorithms 4
 Pseudocode Details
Algorithm arrayMax(A, n)
 Method call
Input: array A of n integers
var.method (arg [, arg…]) Output: maximum element of A
 Return value currentMax ← A[0]
return expression for i ← 1 to n − 1 do
 Expressions if A[i] > currentMax then
currentMax ← A[i]
← Assignment
return currentMax
(like = in Java)
= Equality testing

(like = = in Java)
n2 superscripts and
other mathematical
formatting allowed

Analysis of Algorithms 5
 Comparing Algorithms
 Given 2 or more algorithms to solve the
same problem, how do we select the best
one?
 Some criteria for selecting an algorithm
1) Is it easy to implement, understand, modify?
2) How long does it take to run it to completion?
3) How much of computer memory does it use?
 Software engineering is primarily
concerned with the first criteria
 In this course we are interested in the
second and third criteria

Analysis of Algorithms 6
 Comparing Algorithms
 Time complexity
 The amount of time that an algorithm needs to
run to completion
 Space complexity
 The amount of memory an algorithm needs to
run
 We will occasionally look at space
complexity, but we are mostly interested
in time complexity in this course
 Thus in this course the better algorithm is
the one which runs faster (has smaller
time complexity)
Analysis of Algorithms 7
 How to Calculate Running time
 Most algorithms transform input objects into
output objects
sorting
5 3 1 2 1 2 3 5
algorithm
input object output object

 The running time of an algorithm typically

grows with the input size
 idea: analyze running time as a function of input
size

Analysis of Algorithms 8
 How to Calculate Running Time
 Even on inputs of the same size, running time
can be very different
 Example: algorithm that finds the first prime
number in an array by scanning it left to right
 Idea: analyze running time in the
 best case
 worst case
 average case

Analysis of Algorithms 9
 How to Calculate Running Time
 Best case running best case
time is usually average case
worst case
useless 120

 Average case time is 100

very useful but often

Running Time
80

difficult to determine 60

 We focus on the 40

worst case running 20

time 0
1000 2000 3000 4000
 Easier to analyze Input Size
 Crucial to applications
such as games,
finance and robotics
Analysis of Algorithms 10
 Experimental Evaluation of Running Time

9000
 Write a program 8000

implementing the 7000

algorithm 6000

Time (ms)
 Run the program with 5000

inputs of varying size and 4000

composition 3000

 Use a method like 2000

System.currentTimeMillis( 1000

) to get an accurate 0
0 50 100
measure of the actual I nput Size
running time
 Plot the results
Analysis of Algorithms 11
 Limitations of Experiments
 Experimental evaluation of running
time is very useful but
 It is necessary to implement the
algorithm, which may be difficult
 Results may not be indicative of the
running time on other inputs not included
in the experiment
 In order to compare two algorithms, the
same hardware and software
environments must be used
Analysis of Algorithms 12
Theoretical Analysis of Running Time

 Uses a pseudo-code description of

the algorithm instead of an
implementation
 Characterizes running time as a
function of the input size, n
 Takes into account all possible
inputs
 Allows us to evaluate the speed of
an algorithm independent of the
hardware/software environment
Analysis of Algorithms 13
 RAM: The Random Access Machine
 For theoretical analysis, we assume RAM
model for our “theoretical” computer
 Our RAM model consists of:
 a CPU
 a potentially unbounded bank of
memory cells, each of which can hold an
arbitrary number or character
 memory cells are numbered and
accessing any cell in memory takes unit
time.

1 2 3 ……………………………………
Analysis of Algorithms 14
 Primitive Operations
 For theoretical analysis, we will count
primitive or basic operations, which are
simple computations performed by an
algorithm
 Basic operations are:
 Identifiable in pseudocode
 Largely independent from the programming
language
 Exact definition not important (we will see
why later)
 Assumed to take a constant amount of time
in the RAM model
Analysis of Algorithms 15
 Primitive Operations
 Examples of primitive operations:
 Evaluating an expression x2+ey
 Assigning a value to a variable cnt ← cnt+1
 Indexing into an array A[5]
 Calling a method mySort(A,n)
 Returning from a method return(cnt)

Analysis of Algorithms 16
Counting Primitive Operations
 By inspecting the pseudocode, we can determine
the maximum number of primitive operations
executed by an algorithm, as a function of the
input size
Algorithm arrayMax(A, n)
currentMax ← A[0] 2
for i ← 1 to n − 1 do 2+n
if A[i] > currentMax then 2(n − 1)
currentMax ← A[i] 2(n − 1)
{ increment counter i } 2(n − 1)
return currentMax 1
Total 7n − 1

Analysis of Algorithms 17
 Estimating Running Time

 Algorithm arrayMax executes 7n −1 primitive

operations in the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive
operation
 Let T(n) be worst-case time of arrayMax.
Then
a (7n −1) ≤ T(n) ≤ b(7n −1)
 Hence, the running time T(n) is bounded by
two linear functions
Analysis of Algorithms 18
 Growth Rate of Running Time
 Changing the hardware/ software
environment
 Affects T(n) by a constant factor, but
 Does not alter the growth rate of T(n)
 Thus we focus on the big-picture which is
the growth rate of an algorithm
 The linear growth rate of the running time
T(n) is an intrinsic property of algorithm
arrayMax
 algorithm arrayMax grows proportionally with n,
with its true running time being n times a
constant factor that depends
Analysis on the19
of Algorithms specific
 Constant Factors

 The growth rate is not affected by

 constant factors or
 lower-order terms
 Examples
 102n + 105 is a linear function
 105n2 + 108n is a quadratic function
 How do we get rid of the constant factors to
focus on the essential part of the running
time?

Analysis of Algorithms 20
 Big-Oh Notation Motivation

 The big-Oh notation is used widely to

characterize running times and space
bounds
 The big-Oh notation allows us to ignore
constant factors and lower order terms
and focus on the main components of a
function which affect its growth

Analysis of Algorithms 21
 Big-Oh Notation Definition
 Given functions f(n) 80
and g(n), we say that 70 3n
f(n) is O(g(n)) if there 60 2n+
are positive 50
10
constants 40
n

c and n0 such that 30

20
f(n) ≤ cg(n) for n ≥ n0
10
 Example: 2n + 10 is 0
0 5 10 15 20 25 30
O(n) n
 2n + 10 ≤ cn
 (c − 2) n ≥ 10
 n ≥ 10/(c − 2)
 Pick c = 3 and n0 = 10
Analysis of Algorithms 22
 Big-Oh Example
 Example: the 100,000
90,000 n^2
function n2 is not
80,000 100n
O(n) 70,000 10n
 n2 ≤ cn 60,000 n
50,000
 n≤ c 40,000
 The above 30,000
inequality cannot be 20,000
satisfied since c 10,000
must be a constant 0
0 100 200 300 400 500
n

Analysis of Algorithms 23
 More Big-Oh Examples
 7n-2

7n-2 is O(n)
need c > 0 and n0 ≥ 1 such that 7n-2 ≤ c•n for n ≥ n0
this is true for c = 7 and n0 = 1

3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0 ≥ 1 s.t. 3n3 + 20n2 + 5 ≤ c•n3 for n ≥ n0
this is true for c = 4 and n0 = 21

3 log n + 5
3 log n + 5 is O(log n)
need c > 0 and n0 ≥ 1 s.t. 3 log n + 5 ≤ c•log n for n ≥ n0
this is true for c = 8 and n0 = 2

Analysis of Algorithms 24
 Big-Oh and Growth Rate
 The big-Oh notation gives an upper bound on
the growth rate of a function
 The statement “f(n) is O(g(n))” means that the
growth rate of f(n) is no more than the growth
rate of g(n)
 We can use the big-Oh notation to rank
functions according to their growth rate
f(n) is O(g(n)) g(n) is O(f(n))
g(n) grows Yes No
more
f(n) grows more No Yes
Same growth Yes Yes
Analysis of Algorithms 25
 Big-Oh Rules

 If is f(n) a polynomial of degree d, then

f(n) is O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
 Use the smallest possible class of
functions
 Say “2n is O(n)” instead of “2n is O(n2)”
 Use the simplest expression of the class
 Say “3n + 5 is O(n)” instead of “3n + 5 is
O(3n)”
Analysis of Algorithms 26
 Big-Oh Rules

 If is f(n) a polynomial of degree d, then

f(n) is O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
 Use the smallest possible class of
functions
 Say “2n is O(n)” instead of “2n is O(n2)”
 Use the simplest expression of the class
 Say “3n + 5 is O(n)” instead of “3n + 5 is
O(3n)”
Analysis of Algorithms 27
 Asymptotic Algorithm Analysis
 The asymptotic analysis of an algorithm
determines the running time in big-Oh notation
 To perform the asymptotic analysis
 We find the worst-case number of primitive
operations executed as a function of the input size
 We express this function with big-Oh notation
 Example:
 We determine that algorithm arrayMax executes at
most 7n − 1 primitive operations
 We say that algorithm arrayMax “runs in O(n) time”
 Since constant factors and lower-order terms
are eventually dropped anyhow, we can
disregard them when counting primitive
operations
Analysis of Algorithms 28
 Seven Important Functions
 Seven functions 1E+30
that often appear in 1E+27 Cubic
algorithm analysis: 1E+24 Qua-
 Constant ≈ 1 1E+21
dratic
 Logarithmic ≈ log n 1E+18
Linear
Linear ≈ n

T(n)

1E+15
 N-Log-N ≈ n log n 1E+12
 Quadratic ≈ n2 1E+9
 Cubic ≈ n3 1E+6
 Exponential ≈ 2n 1E+3
1E+0
 In a log-log chart, 1E+0 1E+2 1E+4 n 1E+6 1E+8 1E+10
the slope of the line
corresponds to the
growth rate of the
function
Analysis of Algorithms 29
 Computing Prefix Averages
 We further illustrate
asymptotic analysis with
35
two algorithms for prefix
averages X
30 A
 The i-th prefix average
of an array X is average
25
of the first (i + 1)
elements of X:
20
A[i] = ( X[0] + X[1] + … + X[i])/
(i+1)
15
 Computing the array A
of prefix averages of 10
another array X has
applications to financial
5
analysis Analysis of Algorithms 30
Prefix Averages (Quadratic)
 The following algorithm computes prefix averages in quadratic time by
applying the definition

Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X
#operations
A ← new array of n integers n
for i ← 0 to n − 1 do n
s ← X[0] n
for j ← 1 to i do 1 + 2 + …+ (n − 1)
s ← s + X[j] 1 + 2 + …+ (n − 1)
A[i] ← s / (i + 1) n
return A 1
Analysis of Algorithms 31
 Arithmetic Progression

 The running time of 7

prefixAverages1 is 6
O(1 + 2 + …+ n)
5
 The sum of the first n
integers is n(n + 1) / 2 4
 There is a simple 3
visual proof of this fact
2
 Thus, algorithm
prefixAverages1 runs in 1
O(n2) time 0
1 2 3 4 5 6

Analysis of Algorithms 32
 Prefix Averages (Linear)
 The following algorithm computes prefix averages in linear time by
keeping a running sum

Algorithm prefixAverages2(X, n)
Input array X of n integers
Output array A of prefix averages of X
#operations
A ← new array of n integers n
s ←0 1
for i ← 0 to n − 1 do n
s ← s + X[i] n
A[i] ← s / (i + 1) n
return A 1

 Algorithm prefixAverages2 runs in O(n) time

Analysis of Algorithms 33
 More Examples

Algorithm SumTripleArray(X, n)
Input triple array X[ ][ ][ ] of n by n by n integers
Output sum of elements of X #operations
s ←0 1
for i ← 0 to n − 1 do n
for j ← 0 to n − 1 do n+n+…+n=n2
for k ← 0 to n − 1 do n2+n2+…+n2 = n3
s ← s + X[i][j][k] n2+n2+…+n2 = n3
return s 1

 Algorithm SumTripleArray runs in O(n3) time

Analysis of Algorithms 34
 Useful Big-Oh Rules
 If is f(n) a polynomial of degree d, then
f(n) is O(nd)
2 d
f ( n )=a 0 +a 1 n+a2 n +. ..+a d n
 If d(n) is O(f(n)) and e(n) is O(g(n)) then
 d(n)+e(n) is O(f(n)+g(n))
 d(n)e(n) is O(f(n) g(n))
 If d(n) is O(f(n)) and f(n) is O(g(n)) then d(n)
is O(g(n))
 If p(n) is a polynomial in n then log p(n) is
O(log(n))

Analysis of Algorithms 35
 Relatives of Big-Oh
 big-Omega
 f(n) is Ω(g(n)) if there is a constant c > 0

and an integer constant n0 ≥ 1 such that

f(n) ≥ c•g(n) for n ≥ n0

 big-Theta
 f(n) is Θ(g(n)) if there are constants c’ > 0

and c’’ > 0 and an integer constant n0 ≥ 1

such that c’•g(n) ≤ f(n) ≤ c’’•g(n) for n ≥ n0

Analysis of Algorithms 36
Intuition for Asymptotic Notation

Big-Oh
 f(n) is O(g(n)) if f(n) is asymptotically less than or

equal to g(n)
big-Omega
 f(n) is Ω(g(n)) if f(n) is asymptotically greater than

or equal to g(n)
 Note that f(n) is Ω(g(n)) if and only if g(n) is O(f(n))

big-Theta
 f(n) is Θ(g(n)) if f(n) is asymptotically equal to g(n)

 Note that f(n) is Θ(g(n)) if and only if if g(n) is O(f(n))

and if f(n) is O(g(n))

Analysis of Algorithms 37
Example Uses of the Relatives of Big-Oh
 5n2 is Ω(n2)
f(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1
such that f(n) ≥ c•g(n) for n ≥ n0
let c = 5 and n0 = 1
 5n2 is Ω(n)
f(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1
such that f(n) ≥ c•g(n) for n ≥ n0
let c = 1 and n0 = 1
 5n2 is Θ(n2)
f(n) is Θ(g(n)) if it is Ω(n2) and O(n2). We have already seen the former,
for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and
an integer constant n0 ≥ 1 such that f(n) < c•g(n) for n ≥ n0
Let c = 5 and n0 = 1

Analysis of Algorithms 38
 Math you need to Review
 Summations
 Logarithms and Exponents
 properties of
logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa = alogbx
logba = logxa/logxb
 properties of
exponentials:
a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
b = a logab
bc = a c*logab Analysis of Algorithms 39
 Final Notes
Running time
 Even though in this course we focus
on the asymptotic growth using big-Oh
notation, practitioners do care about A
constant factors occasionally
 Suppose we have 2 algorithms
 Algorithm A has running time 30000n
 Algorithm B has running time 3n2 B
 Asymptotically, algorithm A is better
than algorithm B
 However, if the problem size you deal
with is always less than 10000, then problem size
the quadratic one is faster

Analysis of Algorithms 40