Publicaciones Electrónicas

Sociedad Matemática Mexicana

Theory of Algorithms
-Course Handouts-

Guillermo Morales-Luna

www.sociedadmatematicamexicana.org.mx

Serie: Textos. Vol. 11 (2009)

Guillermo Morales-Luna

Theory of Algorithms
– Course Handouts –

November 30, 2009

Cinvestav-IPN
Foreword

This course follows general lines in Algorithm Design as suggested by the Asso-
ciation of Computing Machinery (ACM). It is addressed to both the professional
engineer or technician, working on software implementation of general computing
procedures, and to students in Computer Engineering or Computer Science, who are
prospective instructors or developers.
We follow closely classical texts in the area. Indeed, textbooks that are also con-
tinuous reference books. In chronological order, a short list contains
[Aho et al(1974)Aho, Hopcroft, and Ullman], [Brassard and Bratley(1988)] (with a
version in Spanish, [Brassard and Bratley(1990)]), [Manber(1989)],
[Sedgewick and Flajolet(1996)], [Papadimitriou and Steiglitz(1998)],
[Hopcroft et al(2001)Hopcroft, Motwani, and Ullman],
[Cormen et al(2003)Cormen, Leiserson, Rivest, and Stein] and [Levitin(2007)].
Textbooks with emphasis in implementational matters are
[Gonnet and Baeza-Yates(1991)], [Wood(1993)] and [Cattaneo and Italiano(1999)].
In the Algorithms Design a good background in Mathematics is always necessary.
Here the suggested lectures are [Graham et al(1994)Graham, Knuth, and Patashnik]
and [Knuth(1999)]. For the interested reader in the Computing limitations, the clas-
sical reference is [Garey and Johnson(1979)].
The current exposition has been taught in a series of courses lectured at the In-
formation Technology Laboratory, Cinvestav-IPN, in Tamaulipas.

Mexico, July 2009 Guillermo Morales-Luna

v
Contents

1 Basic Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Best, average and worst case behaviors . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 General definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.2 A particular example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.3 Estimating computational costs . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Asymptotic analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.1 Growth orders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.2 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.3 Exponentials and logarithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.4 Factorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 Sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3.1 Telescopic sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3.2 Sums of integer powers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3.3 Power sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 Empirical measurements of performance . . . . . . . . . . . . . . . . . . . . . . . 13
1.5 Time and space tradeoff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.6 Recurrence relations in recursive algorithms analysis . . . . . . . . . . . . . 15
1.6.1 A divide and conquer algorithm . . . . . . . . . . . . . . . . . . . . . . . . 15
1.6.2 Substitution method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.6.3 Iteration method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.6.4 Master method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2 Algorithmic Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1 Brute-force algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 Greedy algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.1 Activities-selection problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.2 Knapsack problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3 Divide-and-conquer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.1 Raising to an integer power . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.3.2 Integer multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

vii
viii Contents

2.3.3 Closest pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.3.4 Convex hull . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.3.5 Strassen method for matrix multiplication . . . . . . . . . . . . . . . . 31
2.4 Backtracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.1 A general approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.2 Ordering of combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.4.3 Traveling salesman problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.4.4 Queens problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.4.5 Graham’s scan for convex hull . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.5 Heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.5.1 Simulated annealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.5.2 Tabu search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.5.3 Genetic algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.5.4 Lagrangian relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.6 Pattern matching and syntactical algorithms . . . . . . . . . . . . . . . . . . . . 42
2.6.1 General notions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.6.2 Research with an automaton . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.6.3 Knuth-Morris-Pratt algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.6.4 Suffix trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.7 Numerical approximation algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.7.1 Roots of functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.7.2 Iterative methods to solve linear systems of equations . . . . . . 54
2.7.3 Discretization of differential equations . . . . . . . . . . . . . . . . . . 55
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3 Fundamental Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.1 Simple numerical algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.2 Sequential and binary search algorithms . . . . . . . . . . . . . . . . . . . . . . . . 61
3.3 Quadratic sorting algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.4 Quicksort type algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.5 Hash tables, including collision-avoidance strategies . . . . . . . . . . . . . 63
3.6 Binary search trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.7 Representations of graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.7.1 Adjacency-lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.7.2 Adjacency-matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.8 Depth- and breadth-first traversals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.9 Topological sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.10 Shortest-path algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.11 Transitive closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.12 Minimum spanning tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4 Distributed Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.1 Distributed systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.1.1 Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.1.2 Transition systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Contents ix

4.1.3 Distributed systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.2 Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2.1 Exclusive and concurrent access . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2.2 Dining philosophers problem . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.3 Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.3.1 Byzantine Generals problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.3.2 Routing algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Chapter 1
Basic Analysis

Abstract We introduce here the most basic concepts in measuring algorithms com-
plexities. Initially we distinguish behaviors according to the computational costs due
to their inputs. Then we introduce some mathematical functions of extended use in
measuring algorithmic performances, and the growth orders of maps. We study the
sums, with either finite or countable number of summands, and the convergence
properties. We outline some policies in order to get good algorithm implementa-
tions. Then we discuss the tradeoff among time and space in computing algorithms.
Finally we sketch the most fundamental criteria of growth estimation of functions
determined by recurrence relation. The current chapter is a introduction to the math-
ematics of the analysis of algorithms.

1.1 Best, average and worst case behaviors

1.1.1 General definitions

Let Σ be a finite alphabet. Σ ∗ denotes the dictionary over Σ , or the collection of

words of finite length with symbols in that alphabet. For any word σ ∈ Σ ∗ , the
number of symbols conforming the word is called the length of σ and it is written
as either |σ | or len (σ ). The empty word λ ∈ Σ ∗ is the unique word with zero length,
len (λ ) = 0.
In the functional connotation, an algorithm may be seen as a map A : Σ ∗ → Σ ∗ ,
transforming a word σ ∈ Σ ∗ either into a word τ ∈ Σ ∗ or producing no result. In
the first case we write τ = A(σ ), and it is said that τ is the output corresponding
to the input σ . In the second case we write A(σ ) ↑ and we say that the algorithm A
diverges for the input σ . The domain of the algorithm A, dom (A), is the collection
of those words producing a definite output under A.
In the procedural connotation, an algorithm A determines a sequence of primitive
actions for each input σ ∈ Σ ∗ . If A(σ ) ↑ it may happen that either the corresponding

1
2 1 Basic Analysis

sequence of primitive actions is infinite or it is undefined (at some step, it is not

defined the next step). If σ ∈ dom (A), let tA (σ ) be the number of primitive actions
performed by A from its starting at σ to its reaching of an ending state, producing
thus the output τ = A(σ ). The map tA : Σ ∗ → N, σ 7→ tA (σ ), is called the running
time function, and has the same domain as the algorithm A, dom (tA ) = dom (A).
On the other side, any device, or engine, running the algorithm A, initially shall
occupy some memory locations in order to store the input σ ∈ Σ ∗ and the current
initial state, and thereafter, at each step it must occupy memory locations in order
to store the current configuration, e.g. partial results, current instruction, pointers to
necessary data in the computation, and extra control data. If σ ∈ dom (A), let sA (σ )
be the maximum number of memory locations occupied by A at any computing step
going from the starting at σ to an ending state producing the output τ = A(σ ). The
map sA : Σ ∗ → N, σ 7→ sA (σ ), is called the running space function, and has the same
domain as the algorithm A, dom (sA ) = dom (A).
For any natural number let Σ n be the collection of words over Σ of length n,
Σ = {σ ∈ Σ ∗ | len (σ ) = n}.
n

For any algorithm A and for each non-negative integer n ∈ N, let dom n (A) =
Σ n ∩ dom (A) be the collection of input words of length n producing an output under
A. Let us define the following maps:
Time. TA : N → N, n 7→ max{tA (σ )| σ ∈ dom n (A)}.
Space. SA : N → N, n 7→ max{sA (σ )| σ ∈ dom n (A)}.
Thus, TA (n) is the greatest number of steps that can be achieved by an input of
length n within algorithm A. SA (n) is analogous with respect to space. Both maps
are dependent on the algorithm A and on lengths of words, not just on words.
The maps TA and SA determine the complexity measures of algorithm A, the first
gives the time complexity and the other the space complexity.

1.1.2 A particular example

Any non-negative integer number x ∈ N can be written in binary using `(x) =

dlog2 (x + 1)e bits. `(x) is the length, or the size, of integer x. Any integer may
be represented by a string of length `(x) + 1, where the extra bit can be considered
as a sign: 0 for non-negative integers and 1 for negative integers. Explicitly this
representation is  
1 − sgn (x)
ρ0 : x 7→ ρ0 (x) = (|x|)2 ? .
2
For instance, if x = 2009, its base-2 representation is (x)2 = 11111011001, thus
ρ0 (2009) = 111110110010 while ρ0 (−2009) = 111110110011. Hence the size of
an integer is proportional to the logarithm of its absolute value in base 2.
Now, let ρ1 : (0 + 1)∗ → (0 + 1)∗ be the map that in each word changes each 1
by 10 and leaves each 0 as 0 itself. For instance,
1.1 Best, average and worst case behaviors 3

ρ1 (ρ0 (2009)) = 10101010100101000100

ρ1 (ρ0 (−2009)) = 101010101001010001010

Clearly, for each integer x ∈ Z, len (ρ1 (ρ0 (x))) = (`(|x|) + 1) + w(ρ1 (ρ0 (x))) where
w(ρ1 (ρ0 (x))) is the number of 1’s in ρ1 (ρ0 (x)). Consequently, len (ρ1 (ρ0 (x))) ≤
2(`(|x|) + 1).
Let ρ2 : Z∗ → (0 + 1)∗ be the map that to each finite sequence of integers x =
(x0 , . . . , xm−1 ) it associates the bit-string

ρ2 (x) = ρ1 (ρ0 (x0 )) ? 11 ? ρ1 (ρ0 (x1 )) ? 11 ? · · · ? 11ρ1 (ρ0 (xm−1 )).

ρ2 is an injective map. Clearly, ∀x = (x0 , . . . , xm−1 ) ∈: Z∗ :

m−1
len (ρ2 (x)) = 2(m − 1) + ∑ len (ρ1 (ρ0 (x j )))
j=0
≤ 2(m − 1) + m max len (ρ1 (ρ0 (x j )))
j∈[[0,m−1]]
 
≤ 2(m − 1) + m 2 1 + max `(|x j |)
j∈[[0,m−1]]
 
= 2m 2 + max `(|x j |) − 2. (1.1)
j∈[[0,m−1]]

Thus the length len (ρ2 (x)) can be upperly bounded by a polynomial expression
depending on x, or rather on its dimension m and its entries x j .
Alternatively we may fix a given size to represent positive integers and use a
convention for negative integers (as two’s complement). This has been, as is well
known, the standard approach in industrial implementations, and can be checked in
detail in several texts [Fernandez et al(2003)Fernandez, Garcia, and Garzon].

1.1.3 Estimating computational costs

Let us consider the sorting algorithm depicted at table 1.1.

If the input array A has dimension n, then the number of operations can be esti-
mated as follows:
4 1 Basic Analysis

InsertionSort
Input. An integer array A ∈ Z∗
Output. The same array ordered non-decreasingly
1. For each j ∈ [[2, len (A)]] Do
a. CurVal := A[ j] ;
b. i := j − 1 ;
c. While (i > 0) ∧ (A[i] > CurVal) Do
i. A[i + 1] := A[i] ;
ii. i − −
d. A(i + 1) := CurVal

Table 1.1 Algorithm InsertionSort.

Sta Cost Meaning NR

1. c1 cost of control operations in this greater loop n
1.a c2 assignment cost: done once at each greater itera- n−1
tion
1.b c3 assignment cost: done once at each greater itera- n−1
tion
n
1.c c4 cost of control operations in this lesser loop ∑j
j=2
n
1.c.i c5 assignment cost: done once at each lesser iteration ∑ ( j − 1)
j=2
n
1.c.ii c6 assignment cost: done once at each lesser iteration ∑ ( j − 1)
j=2
1.d c7 assignment cost: done once at each greater itera- n−1
tion
(Sta: Statement NR: Number of repetitions)
We may assume c2 = c3 = c5 = c6 = c7 , since those values are just assignment
costs to variables. Thus the full time cost is
!
n n
T (n) = c1 n + c2 3(n − 1) + 2 ∑ ( j − 1) + c4 ∑ j.
j=2 j=2

However, the exact cost depends on the input array.

Best case The minimum cost appears when the input array is already non-decreas-
ingly ordered, because the lesser loop is skipped. In this case, Tb (n) = c1 n +
3c2 (n − 1).
Worst case The maximum cost appears when the input array is given decreas-
ingly ordered, because the lesser loop is always performed. In this case
1.1 Best, average and worst case behaviors 5
!
n n
Tw (n) = c1 n + c2 3(n − 1) + 2 ∑ ( j − 1) + c4 ∑ j
j=2 j=2
 c4  2  c4 
= c2 + n + c1 + 2c2 + n − (3c2 + c4 )
2 2
Average or typical cases It is expected that the lesser loop is realized half the
number of possibilities at each greater iteration. Hence,
!
n n
j−1 j
Ta (n) = c1 n + c2 3(n − 1) + 2 ∑ + c4 ∑
j=2 2 j=2 2
 
1  c4 2
 5c2 c4
= c2 + n + c1 + + n − (3c2 + c4 )
2 2 2 4

We see that the worst case and the averages cases produce a quadratic cost function.
The ratio of them is such that limn→+∞ TTwa (n) 1
(n) = 2 , thus in the limit the worst case is
twice as expensive as the average case. However limn→+∞ TTba (n)(n) = 0 thus in the limit
the cost of the best case is negligible with respect to the average case.
The best cases of an algorithm correspond to lower computational costs but also
to particular assumptions on the inputs that may be hard to be achieved. The worst
cases correspond to higher computational costs and may also appear in extremely
uncommon, bad luck indeed, cases. The average cases may computational costs
lying between the worst and the best cases. For some algorithms these may be closer
to the worst cases, as for the shown InsertionSort, but for other algorithms
they can be closer to the best cases.
For practical purposes the estimation of time-complexities is useful in order to
know the size of instances suitable to be treated in reasonable times. Let us consider
the nanosecond (ns) as an unit of time (typically the time required to execute a
primitive instruction in a physical conventional computer). Another time unit u is a
multiple of ns, let len (u) denote the length of u, as a multiple of ns, expressed in
base 2. The following table summarizes the values of len (u):

second minute hour day week year decade century

u sec min hr day wk yr dec ct
len (u) 30 36 42 47 50 55 59 62
Let us consider, for instance, the following increasing functions:

log(n) f4 (n) = n · log(n) f7 (n) = n10

f1 (n) = √
f2 (n) = n f5 (n) = n2 f8 (n) = 2n
f3 (n) = n f6 (n) = n3 f9 (n) = n!

f3 is the identity map and it is, obviously, a linear map. The maps f1 and f2 are
sublinear and the maps fk , k ∈ [[4, 9]], are superlinear. For each of these superlinear
increasing maps, fk , let Dk be the interval consisting of those values n ∈ N such that
fk (n), measured in ns, lies between one second and one century. Then:
6 1 Basic Analysis

fk Dk
f4 (n) = n · log(n) 4 × 10 ≤ n ≤ 6 × 1016
7

f5 (n) = n2 3 × 104 ≤ n ≤ 2 × 109

f6 (n) = n3 1 × 103 ≤ n ≤ 2 × 106
f7 (n) = n10 8 ≤ n ≤ 72
f8 (n) = 2n 30 ≤ n ≤ 63
f9 (n) = n! 12 ≤ n ≤ 20

Observe that the polynomial map f7 initially restricts n to be very small compared
with the initial value of the exponential map f8 . However at the end of the shown
time scale, f7 allows a greater value for n than f8 . This is just an illustration that
although initially a polynomial map may seem to have a greater growth than an
exponential map, eventually the exponential growth will win the race!

1.2 Asymptotic analysis

1.2.1 Growth orders

Let f : N → N be a map on the non-negative integers. Let us define the following

function classes:

O( f ) = {g : N → N|∃c > 0, n0 ∈ N : (n ≥ n0 ⇒ g(n) ≤ c · f (n))},

Θ ( f ) = {g : N → N|[g ∈ O( f )] ∧ [ f ∈ O(g)]},
Ω ( f ) = {g : N → N|∃c > 0, n0 ∈ N : (n ≥ n0 ⇒ f (n) ≤ c · g(n))}

If O is any of the symbols O,Θ , Ω we will write g(n) = O( f (n)) to mean g ∈ O( f ).

Remark 1.1. For any two maps f , g : N → N the following conditions are direct:
1. g(n) = O( f (n)) ⇔ f (n) = Ω (g(n)),
2. g(n) = Θ ( f (n)) ⇔ [g(n) = O( f (n))] ∧ [g(n) = Ω ( f (n))].
(here, the symbol “∧” is read and).

Let us also define:

o( f ) = {g : N → N|∀c > 0∃n0 ∈ N : (n ≥ n0 ⇒ g(n) ≤ c · f (n))},

ω( f ) = {g : N → N| f ∈ o(g)},

in other words,
g(n)
g(n) = o( f (n)) ⇔ lim = 0,
n→+∞ f (n)
1.2 Asymptotic analysis 7

in this case it is said that the growth order of g is strictly lower than that of f , and

g(n)
g(n) = ω( f (n)) ⇔ lim = +∞,
n→+∞ f (n)

and in this case it is said that the growth order of g is strictly greater than that of f .

Remark 1.2. The following conditions are immediate:

Transitivity: For any symbol O ∈ {O,Θ , Ω , o, ω} the following implication holds:

g(n) = O( f (n)) ∧ f (n) = O(h(n)) ⇒ g(n) = O(h(n)).

Reflexivity: For any symbol O ∈ {O,Θ , Ω } there holds the relation:

f (n) = O( f (n)).

Simmetry: f (n) = Θ (g(n)) ⇔ g(n) = Θ ( f (n)).

Consequently, the relation g(n) = Θ ( f (n)) defines an equivalence relation in the
space of functions. Each equivalence class is a growth order.

A map f is monotone function if it is either an increasing function or a decreasing

function. It is increasing if ∀n0 , n1 : (n1 < n2 ⇒ f (n1 ) < f (n2 )).
It is decreasing if ∀n0 , n1 : (n1 < n2 ⇒ f (n1 ) > f (n2 )).
It is a non-decreasing function if ∀n0 , n1 : (n1 ≤ n2 ⇒ f (n1 ) ≤ f (n2 )).
It is a non-increasing function if ∀n0 , n1 : (n1 ≤ n2 ⇒ f (n1 ) ≥ f (n2 )).
Thus a constant function is non-decreasing and non-increasing.

1.2.2 Polynomials
m
A one-variable polynomial has the form p(X) = ∑ ai X i . The integer m is called the
i=0
degree of the polynomial, ∂ p, and the real numbers a0 , . . . , am are the coefficients
of the polynomial. The coefficient am corresponding to the power of the degree is
called the leading coefficient.
m
A several-variables polynomial has the form p(X) = ∑ ai Xi , where A ⊂ Nk is a
i∈A
k
i
finite set and if X = (X1 , . . . , Xk ) and i = (i1 , . . . , ik ) then Xi = ∏ X j j .
j=1
Here we will consider just one-variable polynomials. Those of degree 1 are called
linear polynomials, those of degree 2 quadratic polynomials and those of degree 3
cubic polynomials. For any two polynomials p1 (X), p2 (X):
8 1 Basic Analysis

p1 (n) = O(p2 (n)) ⇔ ∂ p1 ≤ ∂ p2 ,

p1 (n) = Θ (p2 (n)) ⇔ ∂ p1 = ∂ p2 ,
p1 (n) = o(p2 (n)) ⇔ ∂ p1 < ∂ p2 .
[
Let us define the class of functions polynomially bounded as nO(1) = O(nm ).
m≥0

1.2.3 Exponentials and logarithms

1.2.3.1 Exponentials

Let us recall that for any non-zero real number a 6= 0,

a0 = 1 & ∀n ∈ N : an+1 = an · a.
1
For negative exponents, say n1 = −n with n ∈ N, it is defined an1 = an . If the base
p
p
number a is positive, for any rational number with q > 0, it is defined a q = x
q,
where x is the real positive number x such that xq = a p . By continuity, az is well
defined for any z ∈ R.
The following power rules hold:

an am = an+m & (an )m = anm . (1.2)

Besides, for any non-negative base number a,


 0 if a < 1
an −→ 1 if a = 1 (1.3)
n→+∞ 
∞ if a > 1

Conventional base numbers are, for instance, a = 10 or a = 2, but the most im-
portant base number in mathematics is e, the base of natural logarithms, which is
defined, among a great variety of different ways, as

1 n
 
e = lim 1 + (1.4)
n→+∞ n

and an approximation to its value is e ≈ 2.7182818284590452354 . . . A well known

series expansion of the exponential function is

xm x2 x3 x4
∀x ∈ R : ex = ∑ = 1+x+ + + +··· (1.5)
m≥0 m! 2 6 24
1.2 Asymptotic analysis 9

1.2.3.2 Logarithms

For any positive base number a > 0, the logarithm in base a of x > 0 is the real
number loga x = y such that ay = x. In symbols:

loga x = y ⇔ ay = x (1.6)

Then, from the power rules (1.7),

loga (xy) = loga x + loga y

loga (xy ) = y · loga x (1.7)
log x
logb x = loga b
a

The first equation in (1.7) asserts that, for each base a > 0, the logarithm function
is a homomorphism among the multiplicative structure of the real positive numbers
and the additive structure of the whole real line.
On the other hand, let us consider the “reciprocal map”, t 7→ 1/t, defined over
the positive real numbers, and its integral,
Z x
dt
F : x 7→ .
1 t
Clearly, ∀x, y ∈ R+ :
Z xy
dt
F(xy) =
1 t
Z x Z xy
dt dt
= +
1 t x t
Z x Z y
dt d(xt1 )
= +
1t 1 (xt1 )
Z x Z y
dt dt
= +
1 t 1 t
= F(x) + F(y)

hence, F is also a homomorphism among the multiplicative structure of the real

positive numbers and the additive structure of the real line. Necessarily, there might
be a base e ∈ R such that F(x) = loge x, and this is indeed the number defined
at (1.4). The natural logarithm is the map x 7→ ln x = loge x.
For any positive base a > 0:
x
ax = eln a = ex ln a (1.8)

and this, together with eq. (1.5) gives a procedure to compute ax .

As another procedure for calculation, we recall that for any real number with
absolute value lower than 1, x ∈ I =] − 1, 1[:
10 1 Basic Analysis

xm x2 x3 x4 x5 x6
ln(1 + x) = ∑ (−1)1+m m = x−
2
+ − + − ··· .
3 4 5 6
(1.9)
m≥1

Right member in last equation is known as the Mercator series.

For each positive real number y ∈ R+ let ny = max{m ∈ Z|em ≤ y} be the char-
acteristic of the natural logarithm of y. We may write y = eny (1 + (y e−ny − 1)), thus

xym
ln y = nx + ln xy = nx + ∑ (−1)1+m m (1.10)
m≥1

where xy = y e−ny − 1, and xy ∈ I. The value ln xy is the mantissa of the natural

logarithm of y.
The Mercator series has a convergence rate extremely low, thus in practice it is
necessary to apply some transforms on it in order to speed-up convergence.

1.2.3.3 Comparison with polynomial maps

For any positive base number a > 0, and any positive integer m, from relations (1.8)
and (1.5) we have
nm
lim n = 0 (1.11)
n→+∞ a

hence nm = o(an ). Thus, whenever a > 0 and f (n) = nO(1) necessarily f (n) =
o(an ). Consequently, any exponential function grows faster than any polynomially
bounded map. [
Let (loga n)O(1) = O((loga n)m ) be the class of functions bounded polylog-
m≥0
arithmically. In order to compare this class with the polynomials, let us write
n1 = loga n. Then, from relation (1.11),

(loga n)m (loga n)m nm

1
= = −→ 0.
n aloga n an1 n→+∞
Hence (loga n)m = o(n).
Consequently, any polylogarithmic map grows slower than the identity map,
hence slower than any other polynomial function.

1.2.4 Factorial

The map n 7→ n! (factorial-n) is defined recursively:

0! = 1 & ∀n ∈ N : (n + 1)! = n! · (n + 1).

1.3 Sums 11
√
n
As an approximation we have n! = 2πn ne 1 +Θ 1n and in a more precise

way,
√  n n √  n n+ 1 · 1
12 n
2πn ≤ n! ≤ 2πn .
e e
Hence n! = o(nn ), n! = ω(2n ), and log(n!) = Θ (n log n).

1.3 Sums

Finite sums If A = (a1 , . . . , an ) is a finite array of numbers, its sum is

n
∑ ai = a1 + · · · + an .
i=1

Series If A = (an )n≥0 is a sequence of numbers, its series is

+∞ n
∑ an = ∑ an = lim
n→+∞
∑ ai .
n=1 n≥0 i=1

Let us review some estimation criteria.

1.3.1 Telescopic sums

n
If A = (a0 , a1 , . . . , an ) is a finite array of numbers, then ∑ (ai − ai−1 ) = an − a0 (the
i=1
middle terms cancel each other). This is called a telescopic sum.
For instance:
n n  
1 1 1 1
∑ i(i + 1) = ∑ i − i + 1 = 1 − n .
i=1 i=1

1.3.2 Sums of integer powers

n
Let smn = ∑ im be the addition of the m-powers of the first (n + 1) non-negative
i=1
integers. From Newton’s Binomial Formula, for each integer i,
m  
m m k
(i + 1) = ∑ i
k=0 k

thus
12 1 Basic Analysis

m−1  
m m m k
(i + 1) − i = ∑ i (1.12)
k=0 k
Summing-up these values gives a telescopic sum, hence
m−1  
m
(n + 1)m − 1m = ∑ skn (1.13)
k=0 k

The following equation system results,

(n + 1) − 1 = s0n
(n + 1)2 − 1 = s0n + 2s1n
(n + 1)3 − 1 = s0n + 3s1n + 3s2n
(n + 1)4 − 1 = s0n + 4s1n + 6s2n + 4s3n
(n + 1)5 − 1 = s0n + 5s1n + 10s2n + 10s3n + 5s4n
(n + 1)6 − 1 = s0n + 6s1n + 15s2n + 20s3n + 15s4n + 6s5n
.. ..
. .

and by solving for the first values of smn we get

s0n = n
1 1
s1n = n(1 + n) = s0n (1 + n)
2 2
1 1
s2n = n(1 + n)(1 + 2n) = s1n (1 + 2n)
6 3
1
s3n = n2 (1 + n)2 = s21n
4
1 1
s4n = n(1 + n)(1 + 2n)(−1 + 3n + 3n2 ) = s2n (−1 + 3n + 3n2 )
30 5
1 2 1
s5n = n (1 + n)2 (−1 + 2n + 2n2 ) = s3n (−1 + 2n + 2n2 )
12 3
.. ..
. .
n
∑ im = O nm+1 .


Clearly,
i=1

1.3.3 Power sums

For each non-zero real number x 6= 0 and any integer n ≥ 0,

n n
xn+1 − 1 = ∑ (xi+1 − xi ) = ∑ xi (x − 1),
i=0 i=0
1.4 Empirical measurements of performance 13
n
1 − xn+1
thus ∑ xi = 1−x
.
i=0
1
Consequently, whenever |x| < 1 we should have ∑ xn = 1 − x .
n≥0
x
By deriving each term and multiplying by x we obtain ∑ nxn = (1 − x)2 , clearly
n≥0
just for |x| < 1.

1.4 Empirical measurements of performance

Empirical analysis is exposed in greater detail at conventional books on Algorithms,

e.g. [Wood(1993)], [Sedgewick and Flajolet(1996)] or [Cattaneo and Italiano(1999)].
In the formal analysis of algorithms several features are considered: correct-
ness, time-efficiency, space-efficiency, clarity and optimality among others. How-
ever, on implementing them, several other features influence the performance of an
algorithm, e.g. the platform and operating system, the compiler, the programming
language and access time to storage devices, among many other “real world com-
puting” features. An implementation of an algorithm may show discrepancies with
respect to the expected performance due to a careless direct transcription of the
algorithm without considering the implementation issues.
Although mostly the empirical analysis is put aside in courses of algorithms it
is important to realize this analysis. It is required to have a clear understanding of
the theoretical basis of the algorithm, to distinguish the performance parameters
to be measured, to choose an appropriate hardware, to implement time measuring
processes, to select non-biased input data in order to perform representative results,
to interpret the results and to correlate them with the theoretical analysis.
For instance, consider LinearSearch: Given a numeric array A of length n
and a query number x it produces as output the least index j such that A[ j] = x, if
x does indeed appear in A, or the value 0 otherwise. LinearSearch tests left-to-
right whether each entry in A coincides with x, and it stops at the first successful test
giving it as output, or outputs the value zero if all the attempts have failed.
If the array A have values uniformly distributed and x indeed appears at A then
we may expect that in the average with (n + 1)/2 tests LinearSearch will finish
correctly. If p denotes the probability that x appears at A, then the expected number
of tests is the weighted average:
n+1  p p
T (n) = p + (1 − p)n = 1 − n+ . (1.14)
2 2 2
An interesting exercise consists in implementing LinearSearch and to check
whether the number of tests fits to the expected value given by (1.14).
Among the main tasks in implementation there are the following:
• Data structures and basic algorithms engineering
14 1 Basic Analysis

• Program design and program coding

• Algorithm verification
• Debugging
• Algorithm testing
• Performance analysis
• Performance tuning
In fact, empirical analysis is an initial form of computer system performance evalu-
ation.

1.5 Time and space tradeoff

In general, running-time and storage-space are inversely proportional in algorithms:

storage can be reduced at the cost of longer times, or equivalently, at the cost of pro-
ducing slower programs. Conversely, quicker programs would cause greater stor-
ages. This phenomenon is known as time-space tradeoff.
For instance, frequently used computable values may be precalculated, then
stored in a given space, called a lookup table, in order to be recalled each time
they are needed. Since calculation times are avoided, programs may run faster but,
obviously, storage is increased. Conversely, instead of storing values, they can be
calculated each time they are required. Thus, storage will be reduced, but comput-
ing times will increase. The storage cachés are used typically as lookup tables and
they are built in faster memory devices. Also, hash tables associate short identifi-
cations to information pieces to order to save time in accessing data. A hash table
should be easily computable and it shall distribute uniformly the keys of information
pieces.
It can be shown that sorting procedures with smallest storages, say proportional
to the number n of elements to be sorted, may run in time proportional to n log n,
and although some procedures may run in linear time with n, they would require
greater storages.
In Complexity Theory, time-space tradeoff is an area of intense research. Namely,
the research in time-space tradeoff lower bounds seeks optimal programs imple-
menting a given procedure.
Many algorithms require a precomputing step and a further computation step (for
instance in accessing data into a database a precomputing step may consist in sorting
the database, and the computing step in accessing to data through a logarithmic
search). Such an algorithm split can be regarded as a space-for-time tradeoff.
1.6 Recurrence relations in recursive algorithms analysis 15

1.6 Recurrence relations in recursive algorithms analysis

1.6.1 A divide and conquer algorithm

Divide and conquer (latin, Divide et impera) is a typical recursive methodology to

deal with computational problems. An instance is divided into simpler instances,
each of them is solved, and then all the solutions of the simpler instances are col-
lected or merged in order to recover a solution of the original global problem.
For instance, Merge-Sort proceeds as follows:
Divide an array of dimension n into two n/2-dimensional subarrays.
Recursively sort each array by this procedure.
Conquer by merging the sorted subarrays.
The following pseudocode specifies in a more specific way the algorithm.
MergeSort
Input. An n-dimensional array A.
p, r extreme indexes of the A’s segment to be sorted.
Output. The sorted segment A[p · · · r].
1. If p < r Then
***Divide Step***
a. q := p+q
 
2 ;
b. MergeSort(A, p, q) ;
c. MergeSort(A, q + 1, r) ;
***Merge Step***
d. Ptr1 := p ; Ptr2 := q + 1 ; Ptr3 := 1 ;
e. While Ptr1 ≤ q and Ptr2 ≤ r Do
i. If A[Ptr1 ] ≤ A[Ptr2 ] Then { B[Ptr3 ] := A[Ptr1 ] ; Ptr1 ++ }
Else { B[Ptr3 ] := A[Ptr2 ] ; Ptr2 ++ }
ii. Ptr3 ++
f. If Ptr1 > q Then B := B ? A[Ptr2 · · · r]
Else B := B ? A[Ptr1 · · · q] ;
g. Let A[p · · · r] := B
Let us denote by T (n) the number of real number comparisons realized by MergeSort
within an input of n values. Clearly,

0 si n = 1,
T (n) = (1.15)
2T n2 + n si n > 1.

For any n enough big, the following equations hold:

16 1 Basic Analysis
n
T (n) = 2T +n
 2 n  n 
= 2 2T 2 + +n
 n 2 2
= 22 T 2 + 2n
 2 n  n 
2
= 2 2T 3 + 2 + 2n
 n 2 2
3
= 2 T 3 + 3n
2
..
. n
= 2κ T κ + κn
2
Thus, if n = 2k is a power of two, k = log2 n, then for κ = k,

T (n) = 2k T (1) + kn = 2k · 0 + kn = n log2 n,

and the solution of (1.15) is T (n) = O(n log n).

We see thus that recurrence equations as (1.15) are useful in the analysis and
design of algorithms. In the current section we are going to sketch some solving
methods.

1.6.2 Substitution method

Given a recurrence relation, one may guess a bounding function and then to prove
the correctness of that bound.
For instance, let us consider:

n  
si n = 1,
T (n) =
2T n2 + n si n > 1.

According to the above example, we may guess that the solution has order
O(n log n). So let us prove that there is a constant c such that T (n) ≤ cn log n.
We may proceed by induction on n.
Let us assume as induction hypothesis: T 2n ≤ c 2n log 2n . Then,
 
   

jnk jnk
T (n) ≤ 2c log +n
2 2
n
≤ cn log + n
2
≤ cn log n − cn log 2 + n
≤ cn log n − cn + n
≤ cn log n
1.6 Recurrence relations in recursive algorithms analysis 17

whenever c > 1.
As a second example, let us consider
l n m j n k
T (n) = T +T +1
2 2
and let us conjecture that its solution is O(n). Let us prove that T (n) ≤ cn − b for
some constants c, b.
As induction hypothesis let us assume T (m) ≤ cn − b, for any m < n. Then
 lnm   jnk 
T (n) ≤ c −b + c −b +1
2 2
≤ cn − 2b + 1
≤ cn − b

whenever c, b > 1.
Sometimes it is convenient to do a change√of variables.
For instance, let us consider T (n) = 2T (d ne) + log2 n.
Let m = log
l 2 n.mThen n = 2m and the recurrence relation can be written as
m
T (2m ) = 2T 2 2 + m. Hence it has the form S(m) = 2S m2 + m, which
 

has solution S(m) = O(m log m). Thus the solution of the original recurrence is
T (n) = O(log n log log n).

1.6.3 Iteration method

Let us introduce this procedure following an example. Let

n
T (n) = 3T + n.
4
By substituting consecutive expressions of the recurrence we get:
18 1 Basic Analysis
n
T (n) = 3T +n
 4  
1 n n
= 3 3T · + +n
4 4 4
 
n 3
= 32 T 2 + +1 n
4 4
     
2 1 n n 3
= 3 3T · + 2 + +1 n
4 42 4 4
 2 !
n 3 3
= 33 T 3 + + +1 n
4 4 4
.. ..
. . !
 k−1
n 3 3
= 3k T k + +···+ +1 n
4 4 4
.. ..
. .
 k !
log4 n 3
≤3 Θ (1) + ∑ n
k≥0 4

 k
3 1
Since 3log4 n = nlog4 3 and ∑ = = 4, then
k≥0 4 1 − 34

T (n) = Θ (nlog4 3 ) + 4n = 4n + o(n) = O(n).

1.6.4 Master method

Theorem 1.1 (Master’s –). Let a, b > 0 and f : N → R be a map. The solution of
the recurrence n
T (n) = aT + f (n)
b
satisfies the following relations which are dependent of f :
1. If f (n) = O nlogb a−ε for some ε > 0, then T (n) = Θ nlogb a .

2. If f (n) = Θ nlogb a , then T (n) = Θ nlogb a log n .

3. If f (n) = Ω nlogb a+ε for some ε > 0 and

n
∃c ∈]0, 1[, n0 ∈ N : (n ≥ n0 ⇒ a f ≤ c f (n)),
b
then T (n) = Θ ( f (n)).
In other words, the growth of the solution is determined by the comparison among
f and nlogb a :
1.6 Recurrence relations in recursive algorithms analysis 19

• If f “is below, a little bit more” than nlogb a , then the solution grows as nlogb a .
• If f “grows” as nlogb a , then the solution grows as nlogb a log n.
• If f “is above, a little bit less” than nlogb a , then the solution grows as f .
The proof of the theorem is out of the scope of our presentation, it can be found
in full detail in [Cormen et al(2003)Cormen, Leiserson, Rivest, and Stein].

Example 1.1. Let us consider T (n) = 9T n3 + n. Since nlog3 9 = n2 , and obviously,

n = O
nlog3 9−ε whenever ε ≤ 1. Thus by case 1. in the Master’s Theorem T (n) =

Θ n2 .
log 3 1
2
= n0 = 1, and


Example 1.2. Let T (n) = T 3n + 1. Since n 2

log 3 1
f (n) = 1 = Θ (1) = Θ (n 2 ),

by case 2. in the Master’s Theorem T (n) = Θ (log n).

Example 1.3. Let T (n) = 3T n4 + n log n. Since nlog4 3 = O n0.793 , and f (n) =

Ω nlog4 3+ε , where ε is greater than 0.2, for c = 43 , we have 3 · n4 log n4 ≤ 34 ·

n log n. Thus by case 3. in the Master’s Theorem T (n) = Θ (n log n).

Example 1.4. Let T (n) = 2T n2 + n log n. On one hand nlog2 2 = n. But,

n log n
= log n = o(nε ),
nlog2 2
for any ε > 0. Thus there is no ε > 0 such that f (n) = Ω nlog2 2+ε . Consequently,

no consequence can be drawn from the case 3. in the Master’s Theorem.

There are other solving methods as the method of generator maps, and the method
of powers.

Problems

1.1. Design an algorithm of complexity O(n) to evaluate polynomials of degree n.

Hint. ∑ni=0 ai xi = x ∑ni=1 ai xi−1 + a0 .

1.2. Which is the greatest value of n > 1 such that any algorithm with running time
8n2 runs slower than an algorithm with time 2n in the same computer?

1.3. Which is the greatest value of n > 1 such that any algorithm with running time
2n runs faster than an algorithm with time 100n2 in the same computer?

In the following exercises, for any two maps g1 , g2 : N → N let us write g1  g2

(g2 eventually uppers g1 ) whenever g1 = O(g2 ).
20 1 Basic Analysis

1.4. Compare the following two maps:

(log2 n)! , (log2 n)2

1.5. Compare the following two maps:

√ log2 n n
2 , 22

1.6. Compare the following two maps:

n
22 , 2log2 n

1.7. Compare the following two maps:

√
2log2 n , 2 2 log2 n

1.8. Compare the following two maps:

n2 , n!

1.9. Compare the following two maps:

3 n


n! , 2

1.10. Compare the following two maps:

1
3 n


2 , n log2 n

1.11. Compare the following two maps:

1
n log2 n , n2n

1.12. Compare the following two maps:

n3 , (log2 n!)

1.13. Compare the following two maps:

(log2 n!) , ln ln n

1.14. Compare the following two maps:

ln ln n , ln n

1.15. Compare the following two maps:

ln n , n ln n
1.6 Recurrence relations in recursive algorithms analysis 21

1.16. Compare the following two maps:

nlog2 log2 n , 1

1.17. Compare the following two maps:

1 , 2log2 n

1.18. Compare the following two maps:

2log2 n
p
, log2 n

1.19. Compare the following two maps:

p
log2 n , 2n

1.20. Compare the following two maps:

n+1
22 , (n + 1)!

1.21. Compare the following two maps:

(n + 1)! , 4log2 n

1.22. Compare the following two maps:

4log2 n , en

1.23. Compare the following two maps:

en , (log2 n)log2 n

1.24. Show that for any two constants a, b > 0 we have (n + a)b = Θ (nb ).

1.25. Express the sum ∑ni=1 (2i − 1) as a “single” formula.

1.26. Show that the solution of T (n) = T n2 + 1 is O(log n).

 

1.27. Find tight bounds (upper and lower) to the recurrence T (n) = 3T (n/3 + 5) +
n/2.

1.28. Find tight bounds (upper and lower) to the recurrence T (n) = T (n − 1) + 1/n.
n  
n
1.29. Calculate the sum ∑ i .
i=1 i
 n
n
1.30. Calculate the sum ∑ .
i=1 i
22 1 Basic Analysis
n
1.31. Calculate the sum ∑ iai .
i=1

1.32. Solve the recurrence

T (0) = c
T (n) = aT (n − 1) + bn.

1.33. Solve the recurrence

T (0) = c
T (n) = aT (n − 1) + bnk .

1.34. Solve the recurrence

T (0) = c
n
T (n) = T + bn log n.
2
1.35. Write a program to approximate e according to relation (1.4).

1.36. Write a program to approximate ex according to relation (1.5).

1.37. Write a program to approximate ax according to relation (1.8).

1.38. Write a program to approximate ln(1 + x) according to relation (1.9).

1.39. Let the integers pn , qn be defined such that

n
pn 1
=∑ ,
qn k=1 k

without common divisors among them. Find recursive relations to calculate the se-
quences of numbers (pn )n≥1 and (qn )n≥1 .

1.40. Write a program to calculate the sequences (pn )n≥1 and (qn )n≥1 introduced
above. Decide whether the harmonic series ∑k≥1 1k is convergent.

1.41. Design an algorithm to check experimentally the equation

n
1
∑i = log(n) + O(1).
i=1

1.42. Design an algorithm to check experimentally the equation

n
1 1
∑ ik = k + (1 − k) nk−1 + O(1).
i=1
Chapter 2
Algorithmic Strategies

Abstract We present here general strategies for algorithms design with representa-
tive examples. The brute-force algorithms are variants of exhaustive search for ef-
fective solutions and they depend on the enumeration processes of the search space.
Greedy algorithms tend to build a solution based on immediate gains of some utility
function. Divide-and-conquer are typical for algorithm optimization, and they are
intended to reduce time complexities of algorithms. Backtracking is a search proce-
dure based on a stacking device and depth-first traversing of tree structured search
spaces. Heuristics is at present time an important area of development in Computer
Engineering. The algorithms on syntactical pattern matching are illustrative of some
procedures to reduce times on query processing through a preprocessing. Finally, in
numerical approximation although there are no exact solutions they provide effec-
tively and efficiently approximations to the solutions with different levels of close-
ness.

2.1 Brute-force algorithms

In section 1.1.1 we have seen several connotations of algorithm. Also, the notion of
problem has several connotations.
A decision problem can be formally stated as a set of words Σ ∗ and a partition
{Y, N,U} of Σ ∗ into three sets: the Yes-answered instances, the No-answered in-
stances, and the Undefined-answer instances. And for a given instance x ∈ Σ ∗ the
goal of the decision problem is to decide whether x ∈ Y or x ∈ N or x ∈ U.
A search problem instead can be seen as a set P ∈ Σ ∗ × Σ ∗ . If (x, y) ∈ P then
it is said that y is a solution for the instance x. For any instance x ∈ Σ ∗ let P(x) =
{y ∈ Σ ∗ | (x, y) ∈ P} be the set of solutions corresponding to x. The goal of a search
problem is that, given an instance x ∈ Σ ∗ it should find y ∈ Σ ∗ such that y ∈ P(x), if
such a y does exist, or recognize that P(x) is empty, otherwise.
Example 2.1 (Factorization). Given an integer number it is required to obtain the
integers dividing that number.

23
24 2 Algorithmic Strategies

As a decision problem, let Y = {(x, y) ∈ N2 | x 6= 0 & ∃z ∈ N : yz = x} be the set

of Yes-answers, let the undefined-answer set consists of pairs in which the dividend
is zero U = {(0, y)| y ∈ N} and let N = N2 − (Y ∪U) be the set of No-answers.
As a search problem, for any instance x ∈ N, x 6= 0, a solution is a divisor of x,
i.e. an integer y ∈ N such that ∃z ∈ N : yz = x.

Example 2.2 (Map evaluation). Given a map f : A → A, where A is a non-empty set,

and a point x ∈ A, the value y = f (x) shall be produced.
As a decision problem, the graph Y = {(x, y) ∈ A2 | f (x) ↓ & y = f (x)} is the
set of Yes-answered instances, the undefined instances are in the set U = {(x, y) ∈
A2 | f (x) ↑} and N = A2 − (Y ∪U) is the set of No-answers.
As a search problem, for any instance x ∈ A, a solution is y ∈ A such that y = f (x)
if f (x) ↓. Otherwise, no solution exists.

A brute-force algorithm tries to solve a search problem in a characteristic way:

Given an instance x ∈ Σ ∗ , following an effective enumeration of the (image) space
Σ ∗ it tests each element y ∈ Σ ∗ as a solution corresponding to x and it stops the first
time it gets a successful value.
Since the enumeration of the search space is relevant, these algorithms are also
called generate-and-test algorithms.
A brute-force algorithm may be easy to implement and it is effective because
through an exhaustive search, whenever there is a solution it will find it. The time
complexity of the algorithm is proportional to the size of the search space. Conse-
quently an approach to optimize the algorithm is to narrow or prune the search space
in order to avoid the unfeasible part of that space.
Brute-force algorithms are suitable for problems in which there is no criteria to
guide the search of solutions, or when there is no any solving heuristic. Indeed the
brute-force strategy maybe considered as the simplest meta-heuristic to solve the
problem. Brute-force algorithms are plausible only for problems with small search
spaces.

Example 2.3 (Factorization). Let us continue the example 2.1. Given a positive inte-
ger x ∈ N, which in base 2 is written with, say k bits, according to the Prime Number
Theorem the number of primes less than x is approximately

x 2k
∼ = O(2k ). (2.1)
ln x k
Thus a naive brute-force algorithm has exponential time complexity. √
However, if x is has a proper factor then it has a prime factor less than x which
can be written with 2k bits. The number of such prospective primes is approximately
k k+2
22 2 2
k
= . (2.2)
2
k
2.2 Greedy algorithms 25

Although it also is determining an exponential time complexity, the gained speed-

k−1
up is important because the ratio of relations (2.1) and (2.2) is 2− 2 and this value
tends to 0 as k grows.

Example 2.4 (Eight queens puzzle). Let us consider the following puzzle: Locate 8
queens in a chessboard in such a way that no queen is attacking any other queen.
Let us assume first that the queens are enumerated as q1 , . . . , q8 . Then the number
of ways to locate these queens in the 64-cells chessboard is
64!
c0 = 64 · 63 · · · · · 58 · 57 = = 178, 462, 987, 637, 760
56!
However, if we make abstraction of the queens numbering, then the number of ways
to select 8 cells in order to put in there the queens is
 
64 c0
c1 = = = 4, 426, 165, 368
8 8!

But, since the queens cannot attack each other, at each column j in the chessboard
there must be a queen in a row, say i j , and no two such row values can coincide.
Thus the map π : j 7→ i j is a permutation [[1, 8]] → [[1, 8]]. Namely, the search space
can be restricted to the space of permutations of 8 elements, and its cardinality is

c2 = 8! = 40, 320

The brute-force algorithm based on this last search space consists in an enumeration
of the permutations [[1, 8]] → [[1, 8]].

2.2 Greedy algorithms

These algorithms deal mostly with optimization problems: Given a real function de-
fined in a domain f : D → R it is required to calculate an extreme value x0 ∈ D such
that f (x0 ) ≤ f (x), ∀x ∈ D, if it is a minimization problem or f (x0 ) ≥ f (x), ∀x ∈ D, if
it is a maximization problem. A greedy algorithm proceeds by local approximations:
at any current point, it moves in the direction pointing to the greatest gain in opti-
mizing the objective function. The main disadvantage with this approach is that it
does not always succeed in its goal: the procedure may fall into a local extreme point
rather than in a global extreme point. However, for certain settings in the problem,
greedy algorithms may be effective, and most probably, also efficient.
26 2 Algorithmic Strategies

2.2.1 Activities-selection problem

Let us assume a set of activities A = {Ai }i∈I ,each with a duration interval [bi , ei [,
ci < ai , where bi is the beginning time and ei is its ending time, and let us assume
that there is just one server, or processor, able to realize the activities, one at any
time. A set of activities AJ = {Ai }i∈J⊂I is consistent if

∀ j1 , j2 ∈ J : ( j1 6= j2 ⇒ [b j1 , e j1 [∩[b j2 , e j2 [= 0).
/

A consistent set AJM is maximal if

∀B : [AJM ⊆ B ⊆ A & AJM 6= B =⇒ B is inconsistent.] .

Problem Activities selection.

Instance: A set A = {Ai = [bi , ei [|1 ≤ i ≤ n} of n activities.
Solution: A maximal consistent set AJM ⊆ AJM .
A “natural” greedy algorithm proceeds as follows:
Greedy activities selection
Input. A set A = {Ai = [bi , ei [|1 ≤ i ≤ n} of n activities.
Output. A maximal consistent set AJM ⊆ AJM .
1. Sort the activities in increasing order according to the ending
times (ei )1≤i≤n .
2. Let the first activity be the current activity.
3. Store the current activity in the output set.
4. While there is an activity whose beginning time is greater than
the ending time of the current activity do
a. Choose as next activity the first activity according to the test.
b. Update the current activity as the next activity.
c. Store the current activity in the output set.
5. Output the built collection.
Remark 2.1. The following assertions hold:
1. There is a maximal consistent set whose first activity is the first activity according
to ending times.
2. The problem Activities selection is indeed solved by the algorithm
Greedy activities selection.
3. The time complexity of Greedy activities selection is Θ (n) if the
input array is given already sorted, otherwise the time complexity is Θ (n log n).

Proof. If AJM is maximal and its first activity AM1 is not the first activity A1 of the
whole collection, then we may substitute AM1 by A1 within AJM , and this substitu-
tion produces a maximal consistent set as claimed. u t
2.3 Divide-and-conquer 27

2.2.2 Knapsack problems

In these problems there is a collection of items, each having a volume and producing
an utility, and there is also a knapsack with a fixed capacity. The goal of the problems
is to put into the knapsack a collection of items in such a way that the knapsack’s
capacity is not exceeded by the sum of the put item volumes and the total utility of
the put items is maximal. In (0-1)-knapsack, or integer knapsack, it is allowed just to
put a given item entirely, while in fractional knapsack it is allowed to put fractions
of items into the knapsack, with proportional volume and utility.
Problem (0-1)-Knapsack.
Instance: A collection C = {Ci = (ti , ui )|1 ≤ i ≤ n} of n items (as pairs volume-
utility) and a total capacity T .
Solution: A subset I ⊂ [[1, n]] such that ∑i∈I ti ≤ T and ∑i∈I ui is maximal.
Problem Fractional Knapsack.
Instance: A collection C = {Ci = (ti , ui )|1 ≤ i ≤ n} of n items (as pairs volume-
utility) and a total capacity T .
Solution: A subset I ⊂ [[1, n]] and a sequence of fractions Q = (qi )i∈I ⊂ Q+ such
that ∑i∈I qiti ≤ T and ∑i∈I qi ui is maximal.

This last problem is suitable to be treated by a greedy algorithm. Namely:

Greedy Fractional Knapsack
Input. A collection C = {Ci = (ti , ui )|1 ≤ i ≤ n} of n items (as pairs
volume-utility) and a total capacity T .
Output. A subset I ⊂ [[1, n]] and a sequence of fractions Q =
(qi )i∈I ⊂ Q+ such that ∑i∈I qiti ≤ T and ∑i∈I qi ui is maximal.
 
1. Let R := utii be the sequence of utility–volume ratios.
1≤i≤n
2. Sort C non-increasingly with respect to R.
3. I := 0/ ; Q := 0/ ;
4. T 0 := T ; U := 0 ; i := 1 ;
5. While T 0 > 0 and i ≤ n Do
n 0 o
a. qi = min Tti , 1 ;
b. I := I ∪ {i} ; Q := Q ∪ {qi } ;
c. T 0 := T 0 − qiti ; U := U + qi ui
6. Output (U, I, Q).

2.3 Divide-and-conquer

Given a problem, say P(x), with input size s(x) = n, it is split as a smaller problems,
P0 (x0 ), . . . , Pa−1 (xa−1 ), with instances of reduction ratio b, s(x j ) = nb , with a, b ∈
Z+ , producing corresponding solutions y0 , . . . , ya−1 . Then they are merged in order
28 2 Algorithmic Strategies

to obtain y = M(y0 , . . . , ya−1 ) as a solution of the original problem P(x). In fig. 2.1
we sketch this procedure with a = b = 2. In section 1.6.1 we have already given an

x y0
-0 -
P n
2

x - -y

- -
x1 y1
Dn Pn2 Mn

Pn

Fig. 2.1 Divide-and-conquer scheme: Dn is an instance divider or splitter, Pk is a solver for in-
stances of size k, and Mn is a merger of solutions of size n instances.

example of a divide-and-conquer algorithm.

If T (n) is a complexity measure of P for instances of size n, e.g. the number of
primitive function calculations, then, we get

aT nb + (d(n) + m(n)) if n > 0



∀n : T (n) = (2.3)
t0 if n = 0

where d(n), m(n) are the respective measures of the divider Dn and the merger Mn .
Either the substitution method or the Master’s Theorem 1.1 gives a solution of the
recurrence (2.3).
For instance, for a = b = 2, whenever d(n) + m(n) = Θ (n), the solution of the
recurrence (2.3) is T (n) = O(n log n). Hence the complexity of a divide-and-conquer
algorithm is, for a = b = 2, n log n, thus it is superlinear but subquadratic (in fact it
is closer to a linear behavior: n log n − n = o(n2 − n log n)).
We will illustrate this algorithmic paradigm through several typical examples.

2.3.1 Raising to an integer power

Given a positive base x > 0, it is required to compute the map Pow : N → R, n 7→ xn .

Clearly,
2.3 Divide-and-conquer 29

n = 2m =⇒ Pow(x, n) = Pow(x, m) · Pow(x, m)
∀n, m :
n = 2m + 1 =⇒ Pow(x, n) = Pow(x, m) · Pow(x, m) · x

and this gives an immediate divide-and-conquer algorithm to compute Pow.

2.3.2 Integer multiplication

Given two n-bit integers x = ∑n−1 j n−1 j

j=0 2 x j and y = ∑ j=0 2 y j it is required to compute
the product z = x · y.
Formally, we have !
2n−2
z= ∑ 2k ∑ xi y j , (2.4)
k=0 i+ j=k

and due to bit-carry the result may consist of 2n bits. The whole evaluation of ex-
pression (2.4) entails O(n2 ) bit-products.
Alternatively, let us write
n −1 n −1 n −1 n −1
2 2 2 2
j
xlow = ∑ 2 xj ; xhigh = ∑ 2 j x n2 + j ; ylow = ∑ 2 yj j
; yhigh = ∑ 2 j y 2n + j
j=0 j=0 j=0 j=0

n n
Then, x = 2 2 xhigh + xlow and y = 2 2 yhigh + ylow . Consequently
n
z = 2n xhigh · yhigh + 2 2 xlow · yhigh + xhigh · ylow + xlow · ylow .

On the other hand,

(xhigh + xlow ) · (yhigh + ylow ) = xhigh · yhigh + xlow · yhigh + xhigh · ylow + xlow · ylow .

Thus, n
z = 2n z2 + 2 2 (z3 − z1 − z2 ) + z1 (2.5)
where

z1 = xlow · ylow ; z2 = xhigh · yhigh ; z3 = (xhigh + xlow ) · (yhigh + ylow ).

The relation (2.5) is the basis of a divide-and-conquer procedure.

Let K(n) denote the number of bit-products required by this procedure. Then, the
recurrence (2.3) is reduced in this case to the following:

3K n2 if n > 0


∀n : K(n) = (2.6)
1 if n = 1

By the Master’s Theorem 1.1 we see that K(n) = O(nlog2 3 ) = O(n1.58496··· ).

The outlined procedure is called Karatsouba’s multiplication method.
30 2 Algorithmic Strategies

2.3.3 Closest pair

Given a finite set X = ((xi , yi ))ni=1 of points in the real plane, it is asked to find the
pair {(xi0 , yi0 ), (xi1 , yi1 )} of points in X realizing the minimum distance.
A brute-force approach would check all pairs, hence it has complexity O(n2 ).
A divide-and-conquer algorithm proceeds as follows:
1. Sort X according to the abscissae: i < j ⇒ xi ≤ x j .
2. Divide X into two sets Xle f t , Xright , of almost equal cardinality, such that the
abscissae of points in Xle f t are all not greater than the abscissa of any point in
Xright .
3. Find the pairs {(x j0 , y j0 ), (x j1 , y j1 )} and {(xk0 , yk0 ), (xk1 , yk1 )} realizing respec-
tively, the minimum distances dle f t and dright in Xle f t and Xright .
4. The solution {(xi0 , yi0 ), (xi1 , yi1 )} of the original problem P may have either its
both points at Xle f t , or its both points at Xright or have its points in different sets.
In the first two possibilities, the solution shall coincide with the solution of the
set giving the least distance. In order to deal with last situation:
a. Let d = min{dle f t , dright } and let ` : x = xλ be a boundary line between Xle f t
and Xright .
b. Let L = {(x, y) ∈ X| |x − xλ | ≤ d} be the collection of points whose distance
to ` is bounded by d.
c. Sort L according to the y-coordinate.
d. For each point in L check whether it has a neighbor closer than d.
5. Output the found closest pair.
In an implementation, the first sorting can be made as a preprocessing step, hence it
can be assumed that the whole list X is already sorted.

2.3.4 Convex hull

Given a finite set X = ((xi , yi ))ni=1 of points in the real plane, it is asked to find the
convex hull of X.
The convex hull of a set X is the smallest convex set containing X, it is indeed
the smallest convex polygon having as vertices some points of X and covering the
whole set X.
A divide-and-conquer algorithm proceeds as follows:
1. Sort X according to the abscissae: i < j ⇒ xi ≤ x j .
2. Divide X into two sets Xle f t , Xright , of almost equal cardinality, such that the
abscissae of points in Xle f t are all not greater than the abscissa of any point in
Xright .
3. Find the respective convex hulls Yle f t , Yright of Xle f t , Xright .
2.3 Divide-and-conquer 31

4. Let Tm be a common tangent line of Yle f t and Yright such that both of them lie
completely at the same side of Tm . Let (xlm , ylm ) and (xrm , yrm ) be the vertices at
Yle f t and Yright passing through Tm .
5. Let TM be a common tangent line of Yle f t and Yright such that both of them lie com-
pletely at the same side of TM , opposite side of Tm . Let (xlM , ylM ) and (xrM , yrM )
be the vertices at Yle f t and Yright passing through TM .
6. The points (xlm , ylm ) and (xlM , ylM ) split Yle f t into two segments, in a clockwise
traversing, [(xlm , ylm ), (xlM , ylM )] and [(xlM , ylM ), (xlm , ylm )], this last one is closer
to Yright .
7. Similarly, the points (xrm , yrm ) and (xrM , yrM ) split Yright into two segments, in a
clockwise traversing, [(xrm , yrm ), (xrM , yrM )] and [(xrM , yrM ), (xlm , ylm )], the first
one is closer to Yle f t .
8. Suppress [(xlM , ylM ), (xlm , ylm )] and add the edge [(xlM , ylM ), (xrM , yrM )]. Sup-
press [(xrm , yrm ), (xrM , yrM )] and add the edge [(xrm , yrm ), (xlm , ylm )].
9. Output the resulting polygon.
The search of the tangents Tm and TM can be done by clockwise traversing the ver-
texes of Yle f t and by counterclockwise traversing the vertexes of Yright .

2.3.5 Strassen method for matrix multiplication

Given two matrices A = [ai j ]1≤i, j≤n , B = [bi j ]1≤i, j≤n it is required to calculate the
matrix product C = A · B.
By definition we have
n
∀i, j ∈ [[1, n]] : ci j = ∑ aik bk j .
k=1

Let Td (n) be the number of products necessary to compute the whole matrix multi-
plication. Since for each entry there are involved n products, we have Td (n) = n3 .
The Strassen method reduces the number of multiplications. Let us partition the
factor matrices as
   
A11 A12 B11 B12
A= & B= .
A21 A22 B21 B22

Then the matrix product is  

C11 C12
C=
C21 C22
and it can be calculated according to the following scheme:
32 2 Algorithmic Strategies

P1 = (A11 + A22 )(B11 + B22 )

P2 = (A21 + A22 )B11
C11 = P1 + P4 − P5 + P7
P3 = A11 (B12 − B22 )
C12 = P3 + P5
P4 = A22 (B21 − B11 ) (2.7)
C21 = P2 + P4
P5 = (A11 + A12 )B22
C22 = P1 + P3 − P2 + P6
P6 = (A21 − A11 )(B11 + B12 )
P7 = (A12 − A22 )(B21 + B22 )

The relations (2.7) entail a divide-and-conquer algorithm: The current problem is

split into 7 subproblems of half size, thus for a = 7 and b = 2, the recurrence rela-
tion (2.3) becomes, for Ts (n) denoting the number of products necessary to compute
the whole matrix multiplication by relations (2.7):

7Ts 2n if n > 0


∀n : Ts (n) =
1 if n = 1

By the Master’s Theorem 1.1 we see that Ts (n) = O(nlog2 7 ) = O(n2.80735··· ).

Consequently, Ts (n) = o(Td (n)). Indeed, for n ∼ 36, the Strassen method will
realize half the number of products than the matrix multiplication by the sole defi-
nition.

2.4 Backtracking

2.4.1 A general approach

Let us recall that a tree is a connected graph T = (V, E), where V is the set of
vertexes and E is the set of edges, such that any two vertices are connected by just
one path and there is a distinguished vertex υ ∈ V called the root. For any vertex
v ∈ V , the parent of v is the first vertex, say u, visited by the unique path going from
v to the root υ, and it is written u = parent (v). In this case it is also said that v is
a child of u. The vertexes with no children are called leaves, and the vertexes which
are not leaves are called inner. An ordered tree is a tree such that for each inner tree
the collection of its children is totally ordered. Thus an ordered tree can be specified
as a list of registers in V ×V ∗ : if (v; [v0 , . . . , vk−1 ]) appears in that list it means that
v is a node with children v0 , . . . , vk−1 .
A depth-first traversing in an ordered tree is such that at any vertex, the traversing
firstly visits the children, in their own ordering, and thereafter it visits that vertex,
i.e. for any (v; v0 , . . . , vk−1 ) the visiting sequence is v0 → · · · → vk−1 → v.
2.4 Backtracking 33

Depth-first
Input. An ordered tree as a list (v, children(v))v∈V .
Output. A numbering v 7→ i ∈ N according to the depth-first travers-
ing
1. Let stack := [(υ, on)] ; counter := 0 ;
2. While stack 6= [] Do
a. (v, signal) := pop(stack) ;
b. If signal = on Then
i. If children(v) 6= [] Then
A. push((v, off), stack) ;
B. For each u ∈ children(v) Do push((u, on), stack)
Else
A. counter + + ; i[v] := counter ;
Else
i. counter + + ; i[v] := counter ;
3. Output [ i[v]| v ∈ V ]
A backtracking procedure is basically a brute-force procedure in which the
search space, or domain space, has a structure of an ordered tree and the ordering
of testing is a depth-first traversing of the domain space.

2.4.2 Ordering of combinations

Let us recall that, for r, n ∈ N, r ≤ n, an r-combination in n indexes is a one-to-one

n!
map [[1, r]] → [[1, n]]. There are (n−r)! such combinations. Let Dn be the union of all
r-combinations in n indexes, with r = 0, 1, . . . , n. Thus card (Dn ) = n! ∑nj=0 1j! .
The collection Dn has a natural ordering structure by the prefix relation:

∀c, d : [c ≤ pre d =⇒ ∃e : d = ce] .

The diagram of this ordering is an ordered tree, and its leaves correspond exactly to
the permutations [[1, n]] → [[1, n]], namely, there are n! leaves in the tree.
A depth-first traversing of this tree gives an enumeration of the whole set Dn .
Let us show now some examples of some particular backtracking procedures.

2.4.3 Traveling salesman problem

Suppose given n cities in a region and a distance di j associated to each pair {ci , c j }
of cities. The traveling salesman problem (TSP) consists in selecting the minimum-
distance traversing of all the cities without passing twice through any city.
34 2 Algorithmic Strategies

Let Cn be the set of cities. If the salesman visits r cities, since he is not allowed to
visit a city twice, then his traversing may be seen as an r-combination in the set Cn .
Then using an effective enumeration of the cities, the set Cn may be identified with
the set [[1, n]] and the collection of allowed traversings with the collection Dn of all
r-combinations in n indexes, r ≤ n. For any combination [c0 , . . . , cr−1 ], its weight is
r−1
w[c0 , . . . , cr−1 ] = ∑ d(c j−1 , c j ).
j=1

The backtracking algorithm to solve TSP consists in the calculation of

min{w(c)| c ∈ Dn }

following a depth-first traversing of the diagram of Dn .

2.4.4 Queens problem

This is a generalization of the problem at example 2.4. Suppose given an (n × n)-

chessboard. The goal is to place n queens in such a way that no queen is threatening
another queen.
A solution is indeed a permutation π : [[1, n]] → [[1, n]] with the connotation that
at each column i, π(i) gives the row at which the i-th queen should be placed in that
column. Since the queens may move along diagonals, any solution must also satisfy

∀i1 , i2 : [i1 6= i2 =⇒ (π(i1 ) − i1 6= π(i2 ) − i2 ) & (π(i1 ) + i1 6= π(i2 ) + i2 )] (2.8)

The collection Dn introduced in section 2.4.2 of combinations in n indexes can be

put in correspondence with the partial placements of queens. Thus, a backtrack-
ing algorithm to solve the Queens Problem consists in pruning all nodes in which
condition (2.8) has failed and interrupting the whole process the first time that con-
dition (2.8) succeeds at a leaf of the tree Dn .

2.4.5 Graham’s scan for convex hull

As an alternative for convex hull calculation, there is a kind of backtracking algo-

rithm called Graham’s scan. The algorithm proceeds as follows:
2.5 Heuristics 35

Graham’s scan
Input. A set X = ((xi , yi ))ni=1 of points in the real plane.
Output. The convex hull co(X).
1. Let x0 be the point in X with least x-coordinate.
2. Sort X − {x0 } according to the angle of the segment joining each
point with x0 and the parallel line to the “y”-axis passing through
x0 . Let us assume that after sorting, X − {x0 } = {x1 , . . . , xn−1 }.
3. Initially, let CH := [x1 , x0 ], and ToBeTested := [x2 , . . . , xn−1 ]
4. While ToBeTested 6= [] Do
a. Let xmiddle , x prev be the first two elements at CH ;
b. xnew := pop(ToBeTested) ;
c. While
the angle (xmiddle , x prev , xmiddle , xnew ) is a right turn
Do
i. drop(xmiddle ,CH) ;
ii. let xmiddle , x prev be the first two elements at CH ;
d. push(xnew ,CH) ;
5. Output CH.
The step 4.c is properly a backtracking procedure till a turn to the left is got in the
three current tested points. In order to decide whether going from a vector x0 =
(u0 , v0 ) to a vector x1 = (u1 , v1 ) makes a left turn or a right turn it must be checked
 
u v 
d(x0 , x1 ) = det  0 0  = u0 v1 − u1 v0 :
u1 v1

[d(x0 , x1 ) > 0 =⇒ left turn] & [d(x0 , x1 ) < 0 =⇒ right turn] .

2.5 Heuristics

Probably the best definition of heuristic algorithm is “common sense procedure”.

Such an algorithm may work fine for most, or just for some, cases, but there lacks a
formal proof of its effectiveness. Heuristics are used when no effective or tractable
solving strategy is known, and they consists of implementations of “rules of thumb”.
Namely, when it is unknown an efficient algorithm to solve a problem, and the ex-
pectation that a pathological instance would occur is rather small, the heuristic may
provide a good solving alternative. Although a heuristic may solve some instances,
or at least give very good solving approximations, there may be other instances in
which the same heuristic fails to get its goal. For many practical problems, a heuris-
tic algorithm may be the only way to get good solutions in a reasonable amount of
time. However, success guarantee does not exist within this algorithmic paradigm.
In ordinary life, heuristics are used in a natural way. For instance, a rule for
weather prediction may be: “Tomorrow’s weather will be as is today”. Clearly, in
36 2 Algorithmic Strategies

many cases it is quite effective and its computational cost is negligible! When pack-
ing items of odd-shape into a box, usually one tries to put the largest items first
and to use the available space to place smaller items. Similarly, TSP may be solved
in a greedy way: At each city, the traveller continues his route through the closest
non-yet-visited city.
General rules hinting particular ways to build heuristics for specific classes
of problems are called metaheuristics. The greedy algorithms, as reviewed in
section 2.2 provide a metaheuristic. Simulated annealing, tabu search, genetic
algorithms, and Lagrangian relaxation, among others, give proper metaheuris-
tics [Michalewicz and Fogel(2004)].

Example 2.5 (Assignment problem). Let us assume there are n tasks and n proces-
sors and a cost matrix C = (ci j )i, j∈[[0,n−1]] : ci j is the cost to assign the j-th task to the
i-th processor. It is sought the assignment α : [[0, n − 1]] → [[0, n − 1]] of least cost
cα = ∑n−1
i=0 ciα(i) .
The first approach is to proceed randomly: For each processor i select randomly
a task among the not-yet-assigned tasks and let α(i) be this chosen task.
This is a quite economical assignment: it runs in linear time. But since it does
not take care of the cost matrix, the probability to find a minimum-cost assignment
is rather small.
As an alternative, for each k ∈ [[0, n − 1]], let m be the least cost appearing at the
current cost matrix and let i, j be such that ci j = m. Let α(i) = j and suppress the
i-th row and the j-th column in the cost matrix.
Evidently in the more advanced stages of the algorithm, the “freedom of choice”
is narrower. Thus it may happen that the criteria to broke the ties, when choosing
the current minima, may affect the algorithm effectiveness.
Let us remark finally that in the shown heuristics no mathematical formulation
has been done. The goal may be stated as an integer programming problem which
has an effective solution (although computationally expensive, since Integer Pro-
gramming is NP-hard [Aho et al(1974)Aho, Hopcroft, and Ullman]).

2.5.1 Simulated annealing

This technique is used to locate extreme values of a given real function defined over
a certain domain. Let us assume that the extrema sought values are minima. The
most general discrete algorithms proceed through a local search: Beginning at an
initial point, at each current point the algorithm updates it by choosing a neighbor
point according to a determined criterion in search of an extreme value i.e. the al-
gorithm descends following a pick direction, and this step is repeated until some
halting conditions are fulfilled.
Simulated annealing is a substantial variation of that strategy. For most occasions
it proceeds as outlined but in order to avoid falling into local minima, instead of the
sought global minima, at the first stages of the search it jumps with higher probabil-
2.5 Heuristics 37

ities to other parts in the domain in order to reinitiate the descent. At more advanced
steps the jump probability is lower. In “physical terms”, initially the system is rather
hot and as time passes the system is getting colder in a controlled way. This is a
motivation of the given name.
Let C : R+ → [0, 1] be a decreasing function such that limt&0 C(t) = 0 and
limt%+∞ C(t) = 1. The map C is a cooling strategy. For instance, for any constant
k
k > 0 the map t 7→ e− t is a cooling strategy. In Fig. 2.2, there appears the graph of
this map for k = 2 and k13 ≤ t ≤ 20k.

k
Fig. 2.2 The graph of t 7→ e− t for k = 2 and 1
k3
≤ t ≤ 20k.

Let f : D → R be a real map defined on a set D in which for each point x ∈ D

there is associated a finite set N(x) ⊂ D, called the neighborhood of x. The goal
is to find a minimum value of f . Within this context, the objective map f is also
called the energy function and the sought point a minimal energy point. Roughly,
the algorithm is sketched in figure 2.3.
The steps 5.a.iii.A-B are the essence of simulated annealing: if the system is
enough hot the updating may choose a worst current point.

2.5.2 Tabu search

As in simulated annealing, tabu search has an updating rule that sometimes may
opt for worst updatings in the hope to escape from local minima. Let us assume
that D is a domain such that for each x ∈ D there is a neighborhood N(x) ⊂ D, and
that f : D → R is an objective function to be minimized. Now, there is a tabu list T
whose aim is to track the local minima already found, and the paths leading to them,
in order to avoid repetition of any path. The updating rule may be synthesized as:

xnew ∈ N(xold ) & [ f (xnew ) < f (xold ) or xnew 6∈ T ] =⇒ xold := xnew . (2.9)

In a schematic way, the procedure is shown in figure 2.4.

38 2 Algorithmic Strategies

SimulatedAnnealing
Input. A cooling strategy C : R+ → [0, 1].
An energy function f : D → R.
Output. A minimum energy point x0 ∈ D.
1. Initially Pick xold ∈ D ; vold := f (xold ) ;
2. t0 :=(some small threshold) ; % Threshold for the system to get cool
3. t := (enough high value) ; % just to begin with a hot system
4. cool := (t < t0 ) ;
5. While (Not cool) Do
a. Repeat
i. Pick xnew ∈ N(xold ) ; vnew := f (xnew ) ;
ii. ∆ f := vnew − vold ;
iii. If ∆ f < 0 then Update {xold := xnew ; vold := vnew }
Else
A. Pick r ∈ [0, 1] ;
B. If r < C(t) then Update {xold := xnew ; vold := vnew }
Until No Updating has been done in several trials % Thermal equilibrium
is got
b. Decrease t ;
c. cool := (t < t0 ) ;
6. Output x0 = xold

Fig. 2.3 Procedure SimulatedAnnealing.

TabuSearch
Input. An objective function f : D → R.
Output. An intended minimum point x0 ∈ D: f (x0 ) ≤ f (x), ∀x ∈ D.
1. Let xmc ∈ D be a (local) minimum of f ;
2. xc := xmc ;
3. T := [xc ] ;
4. Repeat
a. Pick x ∈ N(xc ) − T ;
b. AppendTo[T, x] ;
c. If f (x) < f (xmc ) then xmc := x ;
d. xc := x ;
e. Update T
Until some ending condition is fulfilled
5. Output xmc

Fig. 2.4 Procedure TabuSearch.

2.5 Heuristics 39

The search at step 1. is done through any conventional procedure. Steps 4.c and
4.d implement the updating rule (2.9). In steps 4.b and 4.e some storage consider-
ations may be implemented. In short term memory, 4.b can be skipped most times,
and in step 4.e there can be stored just selected values of xc . In long term memory,
every tested point can be stored in order to avoid repetitions not just of paths but
even of mere points.

2.5.3 Genetic algorithms

The genetic algorithms treat mainly with optimization problems. Given a real ob-
jective function f : D → R, the problem consists in finding an extreme value x0 ∈ D,
say a minimum, of the function f .
Here, at each stage of a genetic algorithm there is a whole collection of points
with the corresponding function values. Naturally, in the collection those points giv-
ing lesser values are preferred. Hence some points are discarded and the remaining
points are combined in order to increase the number of current points and the whole
process is repeated until improvements in the objective function are either impos-
sible (the minimum has already found) or implausible (a closer approximation to
the minimum is computationally hard). Due to the biological inspiration of the al-
gorithms, the jargon is this domain is quite particular. We will use it since currently
it has become conventional. However the reader should be aware that at any time
we will be dealing with points, values and maps and the biological terms are just
naming (although in a metaphoric way) those objects or particular procedures.
The genetic algorithms involve characteristic subprocesses:
Initialization. Many individual solutions are randomly chosen within the domain
D to form an initial population. These first elements either are uniformly dis-
tributed in order to form a representative and unbiased sample of points or may
be seeds in likely regions to contain solutions of the problem. The size of the
initial populations is chosen ad-hoc.
Selection. The current population is called a generation. A subset of the current
population is selected in terms of good-fitness, in order to breed a new generation.
The selection process may involve either the rating of all the individuals or just a
randomly chosen sample of individuals, and in this case the stochastic procedures
in selecting the individuals to be tested should guarantee the diversity of the
generation in order to avoid premature convergence to poor solutions. Among
well-studied selection procedures there are the so called roulette wheel selection
and tournament selection.
Reproduction. There are to main operators: crossover, and mutation. For each
prospective child in a new generation, a couple of parents is selected and they
are combined in order to breed the child point. Of course, a couple of just two
parents is an arbitrary decision, and couples with more than just two members
may provide better performance. Also, individual children may mutate in order
to get better fitness.
40 2 Algorithmic Strategies

All the combinations are designed with the purpose to improve the average fitness
of the current population.
Termination. Several ending conditions may be stated in terms of the reached
number of generations, or the value of the average fitness, or the value of the best
fitness, or the number of repetitions of some population features.
The sketch of the general procedure is the following:
Genetic algorithm
Input. An objective function f : D → R.
Output. An intended minimum point x0 ∈ D: f (x0 ) ≤ f (x), ∀x ∈ D.
1. ctr := 0 ;
2. Initialize a population P(ctr) ;
3. Evaluate the population P(ctr) ;
4. While Termination conditions fail Do
a. ctr + + ;
b. Select P(ctr) from P(ctr − 1) ;
c. Crossover P(ctr) ;
d. Mutate P(ctr) ;
e. Evaluate P(ctr) ;
5. Output a chosen x0 ∈ P(ctr)

2.5.4 Lagrangian relaxation

Let n, m ∈ N be positive integers, and let f : Rn → R be a real continuous function.

Let us consider the optimization problem

Minimizev∈Rn f (v)
(2.10)
subject to Ev = d & ` ≤ v ≤ u

where E ∈ Rm×n , d ∈ Rm , `, u ∈ Rn . The problem (2.10) is called the primal prob-

lem. Let us consider the map

L : Rn × Rm → R , (v, w) 7→ L(v, w) = f (v) + hw|Ev − di = f (v) + wT (Ev − d),

the variable vector w consists of Lagrange multipliers. Also, let us consider

η : Rm → R , w 7→ min{L(v, w)| ` ≤ v ≤ u}.

Then the so called dual problem is:

Maximizew∈Rm η(w). (2.11)

2.5 Heuristics 41

Remark 2.2. If w0 ∈ Rm is a solution of the dual problem (2.11) and v0 ∈ Rn is a

solution of the primal problem (2.10) then η(w0 ) ≤ f (v0 ).

Proof. From the definitions of maps L and η we have:

w ∈ Rm & v ∈ Rn & ` ≤ v ≤ u =⇒ η(w) ≤ f (v) + wT (Ev − d)

and consequently

w ∈ Rm & v ∈ Rn & ` ≤ v ≤ u & Ev = d =⇒ η(w) ≤ f (v).

The remark follows. u

t

Remark 2.3. The map η is concave:

w0 , w1 ∈ Rm & t ∈ [0, 1] =⇒ (1 − t)η(w0 ) + tη(w1 ) ≤ η ((1 − t)w0 + tw1 ) .

Thus η has an unique maximum. Hence, solving the dual problem is an easier
task than solving the primal problem, and a solution of the dual problem entails a
solution of the primal problem.
A particular case in the use of Lagrange multipliers is Lagrangian relaxation. Let
us consider the Linear Integer Problem:

Minimizev∈Zn cT v
(2.12)
subject to Av = d & Bv = e

where c ∈ Zn , A ∈ Zm0 ×n , d ∈ Zm0 , B ∈ Zm1 ×n and e ∈ Zm1 . We may think that the
condition Bv = e imposes easy restrictions (for instance, if B is the identity matrix
and e = 0 the consition states v ≥ 0), while the condition Av = d imposes hard
restrictions. Then the corresponding dual problem becomes

max η(w) (2.13)

w∈Zm0

where
η : w 7→ min{cT v + wT (Av − d)| v ∈ Zn & Bv = e}. (2.14)
The dual problem (2.13) may be easier to solve, because due to the concavity of
the corresponding objective map, it can be treated by “classical methods” in Rn and
then to round-up to a solution in Zn , while the evaluation of η according to (2.14)
involves just “easy” restrictions.
42 2 Algorithmic Strategies

2.6 Pattern matching and syntactical algorithms

2.6.1 General notions

A language is a set of words and a word is a string of finite length over a given
alphabet A. The zero-length word, i.e. the word containing no symbols, is the empty
word λ . Let A∗ denote the set of words. Thus a language is a set L ⊂ A∗ . For instance,
the empty language 0/ contains no words, while the total language, or dictionary A∗ ,
contains all possible words. A language may be given as an exhaustive list consisting
of the words in that language. Obviously this is feasible, although probably not
possible, only when the language is finite.

Example 2.6. A database is a language. Each register in the data base is properly a
word, which is, in turn a concatenation of “symbols” which are the values taken by
the attributes in the database relational scheme. Thus the alphabet is the union of all
attribute domains.

Example 2.7 (Chess). In chess, a match is a sequence of, let us say, positions, steps
or configurations. Each configuration is a 64-length word over the alphabet P con-
sisting of the figures, or pieces, white and black, and a “blank” figure to point that a
cell may be empty. This alphabet has thus 16 + 16 + 1 = 33 symbols. Let C denote
the collection of configurations, C ⊂ P64 ⊂ P∗ . Since each figure may appear in at
most one cell, the number of configurations is bounded by
32   32
64 64!
card (C) ≤ ∑
j
j! = ∑ (64 − j)! ∼ 1054 .
j=2 j=2

A match, as a finite sequence of configurations, is a word with symbols in C. Let M

be the collection of legal chess matches. Then M is a language over C, M ⊂ C∗ , and
indeed a language over P in which each word has as length a multiple of 64.

If a language L ⊂ A∗ is infinite, then it can be specified by a collection of syntac-

tical rules or productions:
α → β , with α ∈ (V + A)∗V (V + A)∗ and β ∈ (V + A)∗ ,
where V is a set of metasymbols, and the connotation
whenever there appears α, then it can be replaced by β .
Beginning from an initial metasymbol S ∈ V the language generated by the system
of productions (or grammar) is the set of words such that there is a transformation
(or parsing) from the initial symbol to that word.

Example 2.8 (Triplets of equal length). Let L3 = {ak bk ck |k ≥ 1} be the language

consisting of words formed by three strings of constant symbols a’s, b’s and c’s of
equal length.
A set of productions to generate this language is the following:
2.6 Pattern matching and syntactical algorithms 43

1. S → A initialize
2. A → aABC add a’s ad for each a b and a c should be added
3. A → abC stop a’s addition and balance it with a b and a c
4. CB → BC move the c’s to the right of all b’s
5. bB → bb change the provisional B’s for terminal b’s
6. bC → bc begin to change the C’s
7. cC → cc change the provisional C’s for terminal c’s

The language generated by this grammar is L3 , indeed.

Example 2.9 (Chess). Let us get back to example 2.7. There is an initial configura-
tion, the well known starting position in chess. Then a match is obtained through
a finite sequence of legal moves. These are indeed also well known and they are
formalized as productions in the language of configurations. The configurations ob-
tained from the initial one are the ending configurations: checkmate to a king or
draw configurations.

Two grammars are equivalent if they generate the same language.

The grammars can be classified according to the form of their productions, and
this originates the so called Chomsky hierarchy (most conventional texts expose
this hierarchy, e.g. [Hopcroft et al(2001)Hopcroft, Motwani, and Ullman]). Just to
mention a few examples:
Regular. T → βU, or T → β , with T,U ∈ V , β ∈ A∗
Linear. T → βl Uβr , or T → β , with T,U ∈ V , βl , βr ∈ A∗
Context-free. T → β , with T ∈ V , β ∈ (V + A)∗
Unrestricted. No restrictions are imposed.
The levels of the hierarchy are nested. The most general, the unrestricted grammars,
generate all recursive or effectively recognized languages. The languages receive
the name of the grammars that generate them. The most popular programming lan-
guages are context-free languages (they are generated by context-free grammars)
and regular languages are extremely important in communication protocols and in
basic operations of operative systems. Each level in the Chomsky hierarchy has a
particular kind of computing devices able to parse the languages in that level. The
unrestricted grammars are parsed by Turing machines, context-free grammars are
parsed by pushdown automata, linear grammars are parsed by two-state pushdown
automata, and regular grammars are parsed by regular or finite automata.
Several important problems arose within the context of formal languages:
Word problem. Given a language L ⊂ A∗ and a word x ∈ A∗ decide whether x ∈ L.
Synthesis problem. Given a language L ⊂ A∗ find the simplest grammar that gen-
erates L.
Equivalence problem. Given two languages L0 , L1 ⊂ A∗ decide whether L0 = L1 .
Boolean composition problem. Given two languages L0 , L1 ⊂ A∗ characterize the
union L0 ∪ L1 , the meet L0 ∩ L1 , and the complement A∗ − L0 .
44 2 Algorithmic Strategies

The complexity of these problems depend on the level at the Chomsky hierarchy.
The Word Problem is easily solvable within regular languages (given L find the min-
imum automaton recognizing it, and for any query word x apply it to the automaton:
it arrives to a success state if and only if the query word lies indeed in the language),
hence communication protocols as TCP/IP can efficiently be implemented; but the
Word Problem is unsolvable for recursive languages.
The following problems have simple statements, the first has an existential
flavour while the second is of an universal kind:
Substring problem. Given two words, a text and a pattern, decide whether the
pattern occurs at the text.
Occurrences problem. Given a text and a pattern, find all occurrences of the pat-
tern within the text.
In spite of their simple statements, the problems above have important applications
in several areas of Computer Engineering: Image Processing, Word Processing,
Database Management, Bioinformatics, Operative Systems, etc.
As concrete examples of syntactic algorithms, let us sketch some conventional
procedures to solve the Substring Problem.

2.6.2 Research with an automaton

Let y ∈ A∗ be the text and let x ∈ A∗ be the pattern. Let us construct an automaton
R recognizing the language A∗ x consisting of all words that have x as a suffix. Then
the pattern appears in the text if and only if when applying y into R a terminal state
is visited at least once. In figure 2.5 we sketch the procedure.

2.6.3 Knuth-Morris-Pratt algorithm

Let y ∈ A∗ be the text and let x ∈ A∗ be the pattern:

y=
x=

Suppose that a prefix σ has a repetition:

x= σ a σ b

For the purpose of the current algorithm, let us number the symbols at any string
from the index 0: m = len (z) =⇒ z = z0 · · · zm−1 .
Suppose that the repeated prefix σ has length k. If the letter b appears at position
j then it will be useful to remember that k symbols before b form a prefix of the
2.6 Pattern matching and syntactical algorithms 45

ResearchWithAutomaton
Input. The text y ∈ A∗ and the pattern x ∈ A∗ .
Output. A Yes-No decision whether the pattern occurs at the text.
1. n := len (x) ;
2. for each i ∈ [[0, n]] let xi be the prefix of length i of x ;
3. let R := (Q, A,t, x0 ) be the automaton defined as follows:
States: Q = {xi }ni=0 .
Transitions: ∀xi ∈ Q , a ∈ A,

xi a ∈ Q =⇒ t(xi , a) = xi a
xi a 6∈ Q =⇒ t(xi , a) = longest suffix at xi a being a prefix of x

Initial state: x0 = λ (the empty word)

Terminal state: xn = x (the whole pattern)
4. m := len (y) ;
5. keep on := False ;
6. qcurrent := x0 ; j := 1 ;
7. While (Not keep on)&&( j ≤ m) Do
a. y j := j-th symbol of y ;
b. qcurrent := t(qcurrent , y j ) ;
c. keep on := (qcurrent == x) ;
d. j++
8. If keep on Then Output Yes Else Output No

Fig. 2.5 Procedure ResearchWithAutomaton.

pattern, because when matching the pattern with a current segment of the text, if
a discrepancy occurs at b then the pattern x may be shifted in order to align the
prefix σ , say σ (1) , with its second occurrence, say σ (2) , and to continue with the
comparison process.

y: a0
σ a σ b
σ a σ b

Thus, in a Trackx table, let us put Trackx [ j] := k. For instance:

x= GCATGCT
Trackx = −1 0 0 0 0 1 2

The Knuth-Morris-Pratt algorithm (KMP) exploits the table Trackx . The proce-
dure at figure 2.6 outlines the construction of the table.
The algorithm KMP shifts the pattern along the text in order to find an occur-
rence. The shift is not done by moving just one place, instead, whenever there is a
discrepancy, the pattern moves to the first non-tested place in which the matching
46 2 Algorithmic Strategies

KMPTrackingTable
Input. The pattern x ∈ A∗ .
Output. The table Trackx .
1. n := len (x) ;
2. Trackx [0] := −1 ; Trackx [1] := 0 ;
3. j := 2 ; i := 0 ;
4. While j ≤ n Do
a. If x[ j − 1] == x[i] Then
i. Trackx [ j] := i + 1 ;
ii. j + + ; i + +
Else
i. If i > 0 Then i := Trackx [i]
Else {Trackx [ j] := 0 ; j + +} ;
5. Output Trackx .

Fig. 2.6 Procedure KMPTrackingTable.

may continue. This is done with the help of the tracking table. The procedure is
shown at figure 2.7.

KMP
Input. The text y ∈ A∗ and the pattern x ∈ A∗ .
Output. The first occurrence of x in y if it does occur.
1. Trackx := KMPTrackingTable(x) ;
2. ny := len (y) ; nx := len (x) ;
3. jy := 0 ; jx := 0 ;
4. While jx + jy < ny Do
a. If x[ jx ] == y[ jy + jx ] Then
i. jx + + ;
ii. If jx == nx Then Output jy
Else
i. jy := jy + jx − Trackx [ jx ] ;
ii. If jx > 0 Then jx := Trackx [ jx ] ;
5. Output “The pattern x does not appear in the text y”

Fig. 2.7 Procedure KMP: Decides whether a pattern occurs within a text.

2.6.4 Suffix trees

A suffix tree for a given text y of length m is a tree such that:

• Each internal node in the tree has at least two children.
2.6 Pattern matching and syntactical algorithms 47

• Each edge in the tree is labelled by a substring of y.

• No pair of edges exiting from one node may have a non-empty common prefix.
• The tree has m leaves: Each non-empty suffix of y is associated to one leaf in the
tree. The path connecting the leaf to the root produces the associated suffix by
concatenating the labels.
For instance, for y =parangaracatirimicuaro of length 22, table 2.1 dis-
plays its suffix tree (calculated with the Suffix Tree Demo developed by L. Alli-
son1 ): each pair ( j: label) represents the initial position of a suffix in the string and
the label associated to the corresponding edge.

|(1:parangaracatirimicuaro)|leaf
tree:|
| | | |(5:ngaracatirimicuaro)|leaf
| | |(4:a)|
| | | |(10:catirimicuaro)|leaf
| |(3:r)|
| | |(22:o)|leaf
|(2:a)|
| |(5:ngaracatirimicuaro)|leaf
| |
| |(10:catirimicuaro)|leaf
| |
| |(12:tirimicuaro)|leaf
|
| | |(5:ngaracatirimicuaro)|leaf
| |(4:a)|
| | |(10:catirimicuaro)|leaf
|(3:r)|
| |(15:imicuaro)|leaf
| |
| |(22:o)|leaf
|
|(5:ngaracatirimicuaro)|leaf
|
|(6:garacatirimicuaro)|leaf
|
| |(11:atirimicuaro)|leaf
|(10:c)|
| |(19:uaro)|leaf
|
|(12:tirimicuaro)|leaf
|
| |(14:rimicuaro)|leaf
|(13:i)|
| |(16:micuaro)|leaf
| |
| |(18:cuaro)|leaf
|
|(16:micuaro)|leaf
|
|(19:uaro)|leaf
|
|(22:o)|leaf

Table 2.1 Suffix tree for the string parangaracatirimicuaro of length 22. It has 8 branch-
ing nodes.

If the string y is such that a proper prefix coincides with a proper prefix, say
∃x, z ∈ A∗ : y = xzx, with len (x) > 0, then no suffix tree may exist for y. Thus, in
order to avoid this problem, usually it is considered a symbol $ not in A and the
word y$ instead of y.
Let us sketch a very basic procedure to build suffix trees. Let us assume that the
string y = u1 · · · um is given. Let, for each i ≤ m, yi = ui · · · um $ be the suffix ranging

1 http://www.allisons.org/ll/AlgDS/Tree/Suffix/
48 2 Algorithmic Strategies
| | |(7:aedadysiglosdichososaquellosalosquelosantiguos)|leaf
| |(2:ichos)|
| | |(25:osaquellosalosquelosantiguos)|leaf
|(1:d)|
| |(10:adysiglosdichososaquellosalosquelosantiguos)|leaf
| |
| |(12:ysiglosdichososaquellosalosquelosantiguos)|leaf
tree:|
| | |(7:aedadysiglosdichososaquellosalosquelosantiguos)|leaf
| |(3:chos)|
| | |(25:osaquellosalosquelosantiguos)|leaf
|(2:i)|
| | |(16:losdichososaquellosalosquelosantiguos)|leaf
| |(15:g)|
| | |(50:uos)|leaf
|
| |(7:aedadysiglosdichososaquellosalosquelosantiguos)|leaf
|(3:chos)|
| |(25:osaquellosalosquelosantiguos)|leaf
|
| |(7:aedadysiglosdichososaquellosalosquelosantiguos)|leaf
|(4:hos)|
| |(25:osaquellosalosquelosantiguos)|leaf
|
| | |(8:edadysiglosdichososaquellosalosquelosantiguos)|leaf
| |(7:a)|
| | |(28:quellosalosquelosantiguos)|leaf
| | |
| | |(36:losquelosantiguos)|leaf
| | |
| | |(46:ntiguos)|leaf
|(5:os)|
| |(19:dichososaquellosalosquelosantiguos)|leaf
| |
| |(25:osaquellosalosquelosantiguos)|leaf
| |
| |(39:quelosantiguos)|leaf
|
| | |(8:edadysiglosdichososaquellosalosquelosantiguos)|leaf
| |(7:a)|
| | |(28:quellosalosquelosantiguos)|leaf
| | |
| | |(36:losquelosantiguos)|leaf
| | |
| | |(46:ntiguos)|leaf
|(6:s)|
| |(14:iglosdichososaquellosalosquelosantiguos)|leaf
| |
| |(19:dichososaquellosalosquelosantiguos)|leaf
| |
| |(25:osaquellosalosquelosantiguos)|leaf
| |
| |(39:quelosantiguos)|leaf
|
| |(8:edadysiglosdichososaquellosalosquelosantiguos)|leaf
|(7:a)|
| |(11:dysiglosdichososaquellosalosquelosantiguos)|leaf
| |
| |(28:quellosalosquelosantiguos)|leaf
| |
| |(36:losquelosantiguos)|leaf
| |
| |(46:ntiguos)|leaf
|
| |(9:dadysiglosdichososaquellosalosquelosantiguos)|leaf
|(8:e)|
| | |(32:losalosquelosantiguos)|leaf
| |(31:l)|
| | |(43:osantiguos)|leaf
|
|(12:ysiglosdichososaquellosalosquelosantiguos)|leaf
|
| |(16:losdichososaquellosalosquelosantiguos)|leaf
|(15:g)|
| |(50:uos)|leaf
|
| | |(19:dichososaquellosalosquelosantiguos)|leaf
| |(17:os)|
| | | |(36:losquelosantiguos)|leaf
| | |(35:a)|
| | | |(46:ntiguos)|leaf
| | |
| | |(39:quelosantiguos)|leaf
|(16:l)|
| |(32:losalosquelosantiguos)|leaf
|
| |(32:losalosquelosantiguos)|leaf
|(28:quel)|
| |(43:osantiguos)|leaf
|
| | |(32:losalosquelosantiguos)|leaf
| |(30:el)|
| | |(43:osantiguos)|leaf
|(29:u)|
| |(51:os)|leaf
|
|(46:ntiguos)|leaf
|
|(47:tiguos)|leaf

Table 2.2 y =dichosaedadysiglosdichososaquellosalosquelosantiguos of

length 53. The tree has 22 branching nodes.
2.7 Numerical approximation algorithms 49

from the i-th to the last position. Let Ti be the suffix tree codifying just the suffixes
y1 . . . , yi . The goal is to build Tm . Let us proceed recursively.
Clearly, T1 consists of just one edge labelled by y1 = y$. The edge has as ex-
tremes the root vertex and a leaf, marked as 1.
Let us assume that T j has been constructed already. Find the largest prefix z j+1
of y j+1 obtained as concatenation of labels in T j from its root to a vertex in T j . The
right extreme of z j+1 should be between two nodes u and v of T j . In other words,
there is a suffix wl ∈ A∗ of z j+1 and two vertexes u and v in T j such that the label
of the edge (u, v) is wl wr , with wr ∈ A∗ . Then split the edge (u, v) and add a new
leaf. Explicitly, add two vertexes u0 , v0 , and the edges (u, u0 ), (u0 , v) and (u0 , v0 ) with
respective labels wl , wr , and the complementary suffix of z j+1 within y j+1 . Mark v0
as the ( j + 1)-th leaf. The resulting tree is T j+1 .
The reader may check this algorithm with the example displayed at table 2.1.
Since there are m suffixes each of different length between 1 and m, we see that
the mentioned algorithm requires O(m2 ) comparisons of symbols. An alternative
algorithm [Ukkonen(1995)] requires O(m) comparisons, thus the quadratic naive
mentioned algorithm is far from the linear complexity of suffix trees construction.
However, the representation is very economical. Since the suffix tree has m
leaves, then necessarily it should have 2m − 1 vertexes. On the other hand, the orig-
inal string has, for each i ≤ m, m − i + 1 substrings of length i. Thus, there are
1 2
2 m(m − 1) = O(m ) non-empty substrings. Hence a linear space is used to store a
quadratic number of objects.
It is interesting to observe that the suffix trees for words of the form abk a and
a bk are remarkably similar.
k

The Substring Problem can be solved in linear time with suffix trees: Namely,
give a text y and a pattern x, construct the suffix tree Ty of y, then in a breadth-first
way locate the vertex v with an edge labelled with a word having the beginning of
the pattern, and parse the pattern along the descendants of v. Reply in accordance
with the parsing success.
The Occurrences Problem also can be solved in linear time with suffix trees:
Namely, once a pattern has been decided to occur in the text, and the parsing has led
to a vertex w in the suffix tree, then all occurrences of the pattern are realized as the
leaves of the subtree rooted at w.
The longest common subsequence of two sequences x, y can also be efficiently
solved. Namely, let z = x#y$ and construct the suffix tree Tz of z. Then the longest
common subsequence is the longest pattern w of length at most min{ len (x), len (y)}
that occurs in a suffix of x of length greater than len (w) and in a suffix of y of length
greater than len (w).

2.7 Numerical approximation algorithms

We will review here schematic procedures based on numerical approximation. The

reader may get a detailed exposition at the classical book [Stoer and Bulirsch(1980)].
50 2 Algorithmic Strategies

2.7.1 Roots of functions

2.7.1.1 A reminder of Differential Calculus

For any function f : Rn → R and any point x ∈ Rn , the following notions are con-
ventional:
Partial derivatives. For each j ∈ [[1, n]] the partial derivative of f at x is

∂f 1
∂ j f (x) = (x) = lim [ f (x + te j ) − f (x)] , (2.15)
∂xj t→0 t

where e j is the j-th vector at the canonical basis of Rn , i.e. it is the vector that
has the value 0 at each entry, except at the j-th entry in which it has the value 1.
Gradient. The gradient of f at x is
 
∂f ∂f
∇ f (x) = (x) · · · (x) .
∂ x1 ∂ xn

In other words, the gradient is the (1 × n)-matrix whose entries are the partial
derivatives. Naturally, the gradient can be identified with the vector whose com-
ponents are the partial derivatives,
n
∂f
∇ f (x) ≈ ∑ ∂ x j (x)e j .
j=1

Differential. The differential of f at x is the linear map D f (x) : Rn → R that is

represented by the gradient of f at x:
n
∂f
D f (x) : y 7→ ∇ f (x) · y = ∑ ∂ x j (x) y j .
j=1

Higher order derivatives. These are defined recursively. If j1 , j2 , . . . , jm are m ≥ 2

indexes in the integer interval [[1, n]] the corresponding partial derivative of order
m of f at x is
∂ j1 j2 ··· jm f (x) = ∂ j1 (∂ j2 ··· jm f (x)) .
Hessian. The Hessian of f at x is the (n × n)-matrix whose entries are the partial
derivatives of order 2:
 
∂11 f (x) · · · ∂1n f (x)
H f (x) =  .. ..
.
 
. .
∂n1 f (x) · · · ∂nn f (x)

Customarily, it is also written D(2) f (x) = H f (x). The Hessian determines a

quadratic form
2.7 Numerical approximation algorithms 51
n
D(2) f (x) : Rn → R , y 7→ D(2) f (x)(y) = yT D(2) f (x) y = ∑ ∂i j f (x) yi y j .
i, j=1

For any enough smooth function f : Rn → R and any point x0 ∈ Rn a local first order
approximation is obtained as

f (x) = f (x0 ) + D f (x) (x − x0 ) + o (kx − x0 k) (2.16)

and a local second order approximation is obtained as

1
f (x) = f (x0 ) + D f (x) (x − x0 ) + D(2) f (x) (x − x0 ) + o kx − x0 k2 .


(2.17)
2
q
In those formulas k · k : x 7→ kx − x0 k = ∑nj=1 |x j |2 is the Euclidean norm.
For n = 1 and enough smooth functions f : R → R, above relation (2.17), can be
generalized to the Taylor’s Expansion Formula: For any k ≥ 0,
k
1
f (x) = ∑ κ! f (κ) (x0 ) (x − x0 )κ + o (kx − x0 kκ ) . (2.18)
κ=0

As is well known, Taylor’s Expansion is used to approximate within arbitrary pre-

cision the values of transcendent functions.

2.7.1.2 The Fixed Point Theorem

A classical problem consists in finding the roots of a real valued function. If f :

Rn → R is a real valued map defined on the n-dimensional real vector space, a root
of map f is a point x ∈ Rn such that f (x) = 0.
Let us assume along this section that the considered objective maps are enough
differentiable in their (open) domains.

Remark 2.4. The problem to find the roots of a function is closely related to the
problem to find unconstrained function extrema.

Suppose that h : Rn → R is a smooth objective function to be minimized over

its domain Rn . Then any minimum point x0 ∈ Rn is such that the gradient attains a
null value, ∇h(x) = 0. Hence the problem to minimize h reduces to the problem to
calculate a common root of the maps f j = ∂ j h : x 7→ ∂ j h(x), for j ∈ [[1, n]].
Indeed, the converse is also valid. Suppose that f : Rn → R is a function whose
roots are sought. Let h : x 7→ k f (x)k2 , then the roots of f correspond to the points
giving the minimal values of h.

Remark 2.5. The problem to find the simultaneous roots of n real functions is closely
related to the problem to find fixed points of maps Rn → Rn .

Given f1 , . . . , fn : Rn → R, let
52 2 Algorithmic Strategies

g : Rn → Rn , x 7→ g(x) = x + ( f1 (x), . . . , fn (x)).

Evidently, for any x ∈ Rn :

x = g(x) ⇐⇒ ∀ j ∈ [[1, n]] : f j (x) = 0.

A function g : Rn → Rn is a contraction if

∃c ∈ [0, 1[: ∀x, y ∈ Rn k f (x) − f (y)k ≤ ckx − yk. (2.19)

Theorem 2.1 (Banach’s Fixed Point -). Any contraction g : Rn → Rn possesses a

fixed point.

Proof. Let x0 ∈ Rn be an arbitrary point, and for each k ≥ 1 let xk = g(xk−1 ). In

other words, the sequence (xk )+∞ k=0 consists of the consecutive iterations of the map
g starting at an arbitrary point in the domain space.
Then, for any k ∈ N, k ≥ 2:

kxk − xk−1 k = kg(xk−1 ) − g(xk−2 )k ≤ ckxk−1 − xk−2 k ≤ · · · ≤ ck−1 kx1 − x0 k.

Consequently, for any k ≤ `,

`
kx` − xk k = ∑ (xκ − xκ−1 )

κ=k+1
`
≤ ∑ kxκ − xκ−1 k
κ=k+1
`
≤ ∑ cκ−1 kx1 − x0 k
κ=k+1
0
≤ kx1 − x0 k ck ∑ cκ
0
κ ≥0
kx1 − x0 k k
= c (2.20)
1−c

Since 0 ≤ c < 1, we have ck −→k→+∞ 0. The above inequalities entail that (xk )+∞
k=0
is a Cauchy sequence in Rn . Hence there exists a point x∞ such that xk −→k→+∞ x∞ .
Necessarily, x∞ = g(x∞ ), and x∞ is indeed a fixed point of g. u
t

Let us observe that, since the final inequality (2.20) is satisfied for all ` ≥ k, it
will be satisfied by the limit point, e.g.

kx1 − x0 k k
∀k ∈ N : kx∞ − xk k ≤ c. (2.21)
1−c
Thus, given a threshold ε > 0, if the iteration is continued till
2.7 Numerical approximation algorithms 53
 −1 
1−c
ln kx1 −x0 k ε
k≥
ln(c−1 )

then the approximation error done by the iteration process will be bounded by ε.

2.7.1.3 Newton’s method

Let us consider the problem to find a root of an enough smooth function f : Rn → R.

Let us assume that we are proceeding in an approximate way through a sequence
(xk )+∞ n
k=0 of points in R converging to a root. Then at a sufficiently large n, we
must expect, due to the continuity of f , that f (xn+1 ) ∼ 0, and the difference be-
tween two consecutive points in the sequence is negligible. Thus, the first order
approximation (2.16) around point xk becomes 0 = f (xk ) + D f (xk ) (xk+1 − xk ) thus
D f (xk ) (xk+1 − xk ) = − f (xk ).
Hence, as a non-deterministic method, let us consider a sequence (xk )+∞ k=0 built as
follows:
Choose x0 ∈ Rn .

(2.22)
For each k ≥ 0 pick xk+1 ∈ Rn such that D f (xk ) (xk+1 − xk ) = − f (xk ).

Proposition 2.1 (Newton’s method). If the sequence (xk )+∞ k=0 built according to
rules (2.22) converges to a limit point x∞ ∈ Rn , then this point is a root of f .

The special case for n = 1 is well determined and for f : R → R the Newton-
Raphson method is the following:

Choose x0 ∈ R.
)
 −1 (2.23)
∀k ≥ 0 : xk+1 = xk − ddxf (xk ) f (xk ).

Another special case is for n ≥ 1 and f : Rn → Rn . Let us write f = ( f1 , . . . , fn ).

Then at any point x ∈ Rn the differential is represented by the Jacobian matrix

J f (x) = D f (x) = [∂ j fi ]i, j∈[[1,n]] .

The Newton-Raphson method is in this case:

Choose x0 ∈ Rn .

(2.24)
∀k ≥ 0 : xk+1 = xk − (D f (xk ))−1 f (xk ).

Here a common root of all component functions is obtained, or, in an equivalent

form, a root for the system of equations fi = 0, i ∈ [[1, n]], is obtained.
Of course, some additional conditions on the objective functions should be im-
posed in order to guarantee convergence. For instance, it is important to have at any
point that the Jacobian matrix is non-singular, and this is just a minor condition on
54 2 Algorithmic Strategies

the whole context of the problem. However, whenever the method converges, the
convergence is at least quadratic:

kxk − x∞ k = O(h2k ), for some h ∈]0, 1[.

2.7.2 Iterative methods to solve linear systems of equations

Let us recall that a system of linear equations

Ax = b (2.25)

with A ∈ Rn×n , b ∈ Rn , has an unique solution x = A−1 b provided that A is non-

singular, det A 6= 0. In this case the calculation of the inverse matrix A−1 entails a
cost of O(n3 ) real numbers multiplications. If the matrix A has a large order, this
can be very costly.
As an alternative, suppose that B ∈ Rn×n is a non-singular matrix. Then any
solution of the original system of linear equations (2.25) satisfies Bx + (A − B)x = b,
and consequently x = (Idn −B−1 A)x+B−1 b, where Idn is the (n×n)-identity matrix.
Hence the solution of (2.25) is a fixed point of the operator

gB : x 7→ (Idn − B−1 A)x + B−1 b.

−1
If supx∈Rn −{0} k(Idn −B
kxk
A)xk
< 1 then gB is a contraction, and the fixed point can be
approximated by the sequence:

x0 ∈ Rn − {0} , ∀k ≥ 0 : xk+1 = gB (xk ).

The method depends on the chosen matrix B. In practice, this matrix is chosen in
dependence of matrix A. Namely, let D = diag [a11 · · · ann ] be the diagonal (n × n)-
matrix whose non-zero values are those appearing at the diagonal of A, let L =
[`i j ]i, j∈[[1,n]] be the lower triangular matrix consisting of the lower triangular block
of matrix A: (i > j ⇒ `i j = ai j ) ; (i ≤ j ⇒ `i j = 0) , and let U = [ui j ]i, j∈[[1,n]] be
the upper triangular matrix consisting of the upper triangular block of matrix A:
(i ≥ j ⇒ ui j = 0) ; (i < j ⇒ ui j = ai j ) . The following procedures result:
Jacobi method. Let B = D. Then Idn − B−1 A = −D−1 (L +U).
Gauss-Seidel method. Let B = D − L. Then Idn − B−1 A = −(D + L)−1U.
The inversion of diagonal and triangular matrices can be computed efficiently. Thus,
the iterative methods would be more efficient than the calculation of the inverse
matrix.
2.7 Numerical approximation algorithms 55

2.7.3 Discretization of differential equations

A differential equation of order k has the form

 
F x, y(x), y(1) (x), . . . , y(k) (x) = 0 (2.26)

where F : Rk+2 → R is a continuous map and for each κ ≤ k, y(κ) (x) = ∂κ y(x). A
solution in a real interval [a, b] ⊂ R is a map y : [a, b] → R such that for each point
x ∈ [a, b], eq. (2.26) holds.
The Cauchy problem consists in solving the differential equation restricted to an
initial condition:
 
Find a solution y of eq. (2.26) such that y(a), y(1) (a), . . . , y(k) (a) = y0 ∈ Rk+1 .

Instead, a problem with boundary conditions is stated as

Find a solution y of eq. (2.26) such that (y(a), y(b)) = y0 ∈ R2 .

Let us introduce a grid in the domain:

a = x0 < x1 < · · · < xm−1 < xm = b

consisting of m + 1 equally spaced points, in increasing order, with extremes in the

real interval. ∀i ∈ [[0, n]], xi = a + i ∆ x, with ∆ x = m1 (b − a). Then, according to
relation (2.15), the derivative of a solution y at a point xi has as approximation:
yi+1 − yi 1
dy(xi ) ≈ δ1 (y)(i) = = (yi+1 − yi ). (2.27)
xi+1 − xi ∆x

Consecutively, if δκ−1 (y)(i) is an approximation of the (κ − 1)-th derivative of y at

the i-th point, then an approximation of the κ-th derivative is

y(κ) (xi ) = (y(κ−1) )(1) (xi ) ≈ δ1 (y(κ−1) (i) ≈ δ1 ◦ δκ−1 (y)(i).

Thus δκ = δ1 ◦ δκ−1 and δκ = δ1κ , which entails

κ  
1 κ
δκ (y)(i) =
(∆ x)κ ∑ ι (−1)κ−ι yi+ι . (2.28)
ι=0

We see that, for each κ, each point (x j , y j ) appears in κ + 1 expressions of the

form (2.28). These possibilities give rise to several modifications in the discretiza-
tion process.
A substitution of the approximations (2.28) into eq. (2.26) poses a linear system
of equations, and the solution of that transformed system is indeed a method to
approximate the solution of the original differential equation.
56 2 Algorithmic Strategies

Problems

2.1. Write a program to solve the Eight Queens puzzle by Brute-Force as explained
in example 2.4.

2.2. Write a program implementing Activities selection as explained in

section 2.2.1.

2.3. Write a program implementing Greedy Fractional Knapsack as ex-

plained in section 2.2.2.

2.4. Write a program implementing Closest Pair as explained in section 2.3.3.

2.5. Write a program implementing Convex hull as explained in section 2.3.4.

2.6. Write a program implementing Depth First traversal as explained in sec-

tion 2.4.1.

2.7. Write a program to solve the Eight Queens puzzle by a backtracking procedure
as explained in section 2.4.4.

2.8. Write a program implementing Graham scan as explained in section 2.4.5.

2.9. Write a program implementing the Strassen method for matrix multipli-
cation as explained in section 2.3.5.

2.10. Write a program implementing the Traveling salesman problem as

explained in section 2.4.3.

2.11. Write a program implementing Simulated annealing as explained in

section 2.5.1. Test the implementation with D = [0, 2π] and the map f : x 7→
sin(x) cos(15x).

2.12. Write a program implementing Tabu search as explained in section 2.5.2.

Test the implementation with D = [0, 2π] and the map f : x 7→ sin(x) cos(35x).

2.13. Write a program implementing the research with an automaton as explained

in section 2.6.2.

2.14. Write a program implementing the Knuth-Morris-Pratt algorithm

as explained in section 2.6.3.

2.15. Write a program implementing the Suffix-Tree Construction as ex-

plained in section 2.6.4.

2.16. Write a program implementing the search of a fixed point according to Ba-
nach’s Theorem (see theorem 2.1). Test the implementation with [0, π2 ] and the map
f : x 7→ cos(x).
2.7 Numerical approximation algorithms 57

2.17. Write a program implementing the Newton Raphson method as explained

in section 2.7.1.3. Test the implementation with D = [0, 2π] and the map f : x 7→
sin(x) cos(15x).

2.18. Write a program implementing the iterative method to solve systems of linear
equations as explained in section 2.7.2 using the Jacobi method.

2.19. Write a program implementing the iterative method to solve systems of linear
equations as explained in section 2.7.2 using the Gauss-Seidel method.
Chapter 3
Fundamental Algorithms

Abstract In this chapter we expose succinctly classical algorithm that serve as pro-
totypes of the general design lines in the previous chapter. In numerical algorithms
we pose the interpolation problem and the solution given by Legendre polynomials
and by solving systems of linear equations. The queries to databases entail local-
ization of given registers. The sequential search solve this problem in linear time
while the binary search works in logarithmic time. Typical sorting algorithms work
on quadratic time, but by using a divide-and-conquer approach sorting can be done
in n lg n time through Quicksort. Hash tables help to store information in compact
forms, and binary trees are also storage devices structuring the information in such
a way to reduce, to logarithmic costs, the most basic relational operations. The final
part of the chapter is devoted to Graph Theory problems with wide applications.
The final part of the chapter is devoted to Graph Theory problems with wide ap-
plications. Here, the breadth-first and depth-first traversal help to solve important
problems as calculation of shortest-paths and of minimum spanning trees.

3.1 Simple numerical algorithms

The interpolation problem is the following:

Problem Polynomial interpolation.
Instance: A set of n + 1 points in the plane {(xi , yi )}ni=0 , with pairwise distinct
abscissae.
Solution: The n degree polynomial p(X) = ∑nj=0 a j X j such that p(xi ) = yi for
i ∈ [[0, n]].
For instance, for n = 1, the straight line joining (x0 , y0 ) and (x1 , y1 ) has para-
metric equation r(t) = t(x1 , y1 ) + (1 − t)(x0 , y0 ) = (x0 + t(x1 − x0 ), y0 + t(y1 − y0 )).
Thus, any point (x, y) = r(t) in that line should satisfy xx−x 0
1 −x0
= yy−y 0
1 −y0
, or

y1 − y0
y= (x − x0 ) + y0 .
x1 − x0

59
60 3 Fundamental Algorithms

Thus the expression at the right hand side is the linear polynomial interpolating the
two given points.
In order to solve the Interpolation Problem let us consider the
Lagrange method. Suppose that `i (X) is a n degree polynomial such that

∀ j ∈ [[0, n]] : [ j 6= i =⇒ `i (x j ) = 0] & [ j = i =⇒ `i (xi ) = 1] . (3.1)

Then clearly the sought polynomial is

n
p(X) = ∑ yi `i (X) (3.2)
i=0

Relations (3.1) and (3.2) determine an algorithm to solve the Interpolation Problem.
The polynomials satisfying relation (3.1) are

∏ j∈[[0,n]]−{i} (X − x j ) X −xj
`i (X) = = ∏
∏ j∈[[0,n]]−{i} (xi − x j ) x −xj
j∈[[0,n]]−{i} i

Vandermonde method. Since it is required

∀ j ∈ [[0, n]] : y j = p(x j ) (3.3)

then, as an equation system, these equations can be stated jointly as

   n j
 
∑ j=0 a j x0 1 x0 · · · x0n
 
y0 a0
 j
 y1   ∑n a j x   1 x1 · · · xn   a1 

   j=0 1   1 
 ..  =  =. .

. . . ..   .. 
 .   .
.  . .

.  . 
yn n j 1 x · · · x n an
∑ j=0 a j xn n n

thus we have a system u = Va, with solution a = V −1 u, provided that the matrix V
is non-singular. One can see that the matrix V is indeed non- singular because

detV = ∏ (x j − xi ).
0≤i< j≤n

However this value could be very close to zero if the distance among consecutive
abscissae is smaller than 1. This renders the problem rather ill-conditioned.
In section 2.7 some other numerical algorithms have been discussed.
3.2 Sequential and binary search algorithms 61

3.2 Sequential and binary search algorithms

For a given finite sequence A of objects of the same type, and a query object x it is
asked to decide whether x appears in A and if it appears indeed, then to locate its
position in the sequence.
The sequential search consists in checking consecutively whether each element
in the array coincides with the query object x. The procedure, shown at figure 3.1,
is quite direct.

SequentialSearch
Input. An array A[1 : n], and a query value x.
Output. The position at A in which x occurs, or an indication that it does not
occur, otherwise.
1. pcurr := 0 ;
2. Stop := False ;
3. While Not(Stop) && pcurr < len (A) Do
a. pcurr + + ;
b. Stop := (x == A[pcurr ]) ;
4. If Stop Then Output pcurr
Else Output ’x does not occur in A’

Fig. 3.1 Procedure SequentialSearch.

Let us assume that for each entry i ∈ [[1, n]] at the array, pi is the probability that
the query object x coincides with A[i]. Hence W = (pi )ni=1 is a probability distribu-
tion on A. Let χ be the random variable χ : x 7→ i, that for each query object gives
the number of tests in SequentialSearch realized to find it in the array. The
expected number of tests is thus
n
E(χ) = ∑ i pi ∈ [1, n],
i=1

which is an average of the indexes weighted by the probabilities. If we perform a

permutation of the entries in A, let E(χ, π) = ∑ni=1 i pπ(i) be the corresponding ex-
pected number of tests for the permuted array. Then the minimum value corresponds
to the permutation that sorts the probability distribution in a decreasing way, while
the permutation arranging the probability increasingly gives the greatest expected
values. This means that if the array is arranged with the most probable elements for
query at the beginning then the expected number of tests will be diminished.
On the other side, if the objects in the array are in an ordered set and the array is
sorted monotonically with respect to that order, then the search of a query element
can be done in a manner much more efficient.
Indeed, binary search is a divide-and-conquer algorithm. Suppose that the array
is sorted in an increasing way. Given a query object x, initially let us consider the
62 3 Fundamental Algorithms

whole array A with inferior extreme index im = 1 and superior index iM = n. At

any stage of the algorithm, compare x with the element at the middle of the array in
order to decide whether the element should appear at the inferior or at the superior
segment of A and continue the search
 with
 that subsegment. The rule is thus the
im +iM
following: Let imid := Ceiling 2 ,

x < A[imid ] =⇒ let iM := imid

x = A[imid ] =⇒ Output “Found x at entry imid ”
x > A[imid ] =⇒ let im := imid

Thus according with the time estimations performed at section 2.3 the expected
number of tests in finding x is O(log n).

3.3 Quadratic sorting algorithms

When sorting an array of items in an ordered set, the most intuitive algorithm is a
round-robin comparison among its entries as shown in figure 3.2.

Round-robinSort
Input. An array A = A[1 : n].
Output. The sorted array A.
1. For i = 1 to n − 1 do
a. For j = i + 1 to n do
i. If A[i] > A[ j] Then switch A[i] and A[ j] ;
2. Output A

Fig. 3.2 Procedure Round-robinSort.

The number of comparisons, realized at step 1.a.i., is

n−1 n−1
n(n − 1)
∑ (n − i) = ∑ k = = O(n2 ).
i=1 k=1 2

Thus it has a quadratic time complexity.

InsertionSort discussed at section 1.1.3 has also quadratic time complexity.

3.4 Quicksort type algorithms

Quicksort is the typical example of a divide-and conquer algorithm:

3.5 Hash tables, including collision-avoidance strategies 63

Divide Given an array A split it into two subarrays A = A1 ∪ A2 such that each
entry at A1 is not greater than any entry at A2 .
Conquer Sort each subarray Ai , i = 1, 2, using the current procedure.
No merge is required due to the “divide” condition. The procedure is sketched at
figure 3.3.

Quicksort
Input. An array A and extreme indexes 1 ≤ p ≤ r ≤ n
Output. The subarray A[p · · · r] sorted.
1. If p < r Then
a. q := SplittingIndex(A, p, r) ;
b. Quicksort(A, p, q) ;
c. Quicksort(A, q + 1, r) ;

Fig. 3.3 Procedure Quicksort.

There are several criteria to choose the splitting index, at step 1.a., and they de-
pend on the selection of a pivot. Namely, given a pivot x in the range of A, let
J1 = { j| A[ j] < x} and J2 = { j| A[ j] ≥ x}. Clearly, when recognizing J1 , the ele-
ments of A may be switched if necessary in order to guarantee the condition in the
“divide” step. The splitting index is thus q = p + card (J1 ).
Each criterion for pivot selection produces a variant of Quicksort. Among the
criteria, there are the following:
• the first entry A[1],
• the middle entry A p+r

2 ,
√
• the entry in “golden ratio” proportion A [dp + γ(r − p)e], where γ = 5−1
2 ,
A[p]+A[r]
• the middle value of extremes 2 ,
• the value in “golden ratio” proportion of the extremes A[p] + γ(A[r] − A[p]) if
A[r] ≥ A[p], or A[r] + γ(A[p] − A[r]) otherwise,
• etc.
Quicksort will show better performances whenever the splitting index is closer
to the middle number of current entries p+r
2 and the probabilistic distribution of the
values at the input array will determine whether this condition is fulfilled. In best
and average cases, the time complexity of Quicksort is Θ (n log n).

3.5 Hash tables, including collision-avoidance strategies

A hashing is a map h : A → B from a set A of strings of large length into a set B

of strings of short length. For instance, let A be the set of full names of citizens
in a country, and let B be the set of social security numbers in the USA or the
64 3 Fundamental Algorithms

certificado único de registro de población (CURP) in Mexico. Usually the names of

persons are long strings, while the social security numbers or CURP’s are short. The
correspondence among citizens and these identifiers is a hash function. In database
management, the key attributes in a database define a hash function: The whole
register can be identified by its value in a key attribute. Conversely, whenever there
is a hash function h : R → B, from a database into a set B, then B can be seen as the
domain of a key attribute defined by the values of h.
Although there cannot be a one-to-one correspondence A → B, the maps that
“seem” to be injective can be very useful. A collision in a hashing consists of a pair
a1 , a2 ∈ A such that they have the same hash value, h(a1 ) = h(a2 ).
For purposes of efficient information storage or digital signatures, hash functions
with low collision probability in a well defined domain are very important. Let us re-
call some general constructions of hash functions and later some collision-avoidance
strategies.
Suppose card (A) = n and card (B) = m. We look up for a map h : A → B satis-
fying ∀b ∈ B : Pr {a ∈ A| h(a) = b} ≈ 1/m.
An elementary method is the following: let (r j )nj=1 be a sequence of random
numbers with uniform distribution in [0, 1[ and let h : a j 7→ br j · mc.
As a second elementary example, let us assume A, B ⊂ N. Let p be a prime num-
ber close to m and let h : k 7→ k mod p.
As a last elementary example, let · mod 1 : x 7→ x − bxc be the fractionary part
map, and√ let γ ∈]0, 1[ be a fixed real number, mostly it is taken the golden ratio
γ = 21 ( 5 − 1). In this case, let h : k 7→ bm((kγ) mod 1)c. Observe here, that m does
not essentially influence in the definition of h. It appears just as a parameter, hence
h actually may be considered as a whole family of hash functions.
Let H ⊂ [[0, m − 1]]A be a finite family of hash functions. H is called an univer-
sal hash family if
 
card (H )
∀a1 , a2 ∈ A a1 6= a2 ⇒ card {h ∈ H |h(a1 ) = h(a2 )} = (3.4)
m

Within an universal hash family, one can have a hash map h : x 7→ hx (x), where
hx ∈ H is chosen randomly for each x ∈ A. Obviously, it is necessary to record the
used map hx for each x.

Remark 3.1. If H is universal, then for each a ∈ A the expected value of the number
of collisions involving a is n−1
m .

Proof. For each u, v ∈ A, u 6= v, let cuv = 1 if h(u) = h(v) , or cuv = 0 otherwise; and
let Cu = ∑v6=u cuv . Then E(cuv ) = m1 and E(Cu ) = ∑v6=u E(cuv ) = n−1m . u t

A direct method for collision-avoidance is the following: Suppose an a ∈ A is

given and let b = h(a). If there has appeared a1 ∈ A such that b = h(a1 ) then consider
b1 = b + 1. If there has appeared a2 ∈ A such that b1 = h(a2 ) then consider b2 =
b1 + 1. And so on. The first time that a bk is found such that it has not been taken by
previous a’s then let us associate this value to the current a. In a more general way,
3.6 Binary search trees 65

let f : N → Z be a one-to-one map. Then if the value b = h(a) has been taken by
a previous a1 , the associate to a the value b + f (k), where k is the minimum index
such that b + f (k) has not been assumed formerly.
As another direct possibility, a tagging could be used in order to distinguish dif-
ferent objects in A colliding into the same hash values.

3.6 Binary search trees

A binary tree is either an empty tree or a leaf or a root which is the parent of a left
binary tree and a right binary tree. In this last case, the root of the left subtree is the
left child and the root of the right subtree is the right child of the parent root. Thus
in any binary tree any internal node is the parent of two children. For any node x let
Left(x) denote the left subtree and Right(x) the right subtree that have x as parent.
If n is the number of vertexes in a binary tree and k is its height then k = O(log n)
and n = O(2k ), i.e. the number of vertexes is exponential with the height and the
height is logarithmic with the number of vertexes. Besides in a binary tree, the num-
ber of leaves is around half the total number of vertexes.
An inorder labeling is a map L : {vertexes} → Z such that

y ∈ Left(x) ⇒ L(y) < L(x)
∀x, ∀y : (3.5)
y ∈ Right(x) ⇒ L(y) > L(x)

The labels may be the values of a hashing function, thus a binary tree may be
used as a data structure containing the keys of the database. The elementary access
functions can be efficiently performed:
Find the minimum key. Just go to the leftmost leaf.
Find the maximum key. Just go to the rightmost leaf.
Find successor. Given a vertex x, if Right(x) is not empty, the successor of x
is the minimum element in Right(x); otherwise, ascend in order to find the first
ancestor in the left line, the successor of x is this ancestor.
Find predecessor. Given a vertex x, if Left(x) is not empty, the predecessor
of x is the maximum element in Left(x); otherwise, ascend in order to find the
first ancestor in the right line, the predecessor of x is this ancestor.
Search a query key. Compare the query key z with the root x. If z < L(x) then
search z in Left(x), if z = L(x) then exit successfully the search, and if z > L(x)
then search z in Right(x). A search procedure in an empty tree is always a failure
process.
The time complexity is proportional with the height, thus it is logarithmic with
the length of the input array.
Insertion of a labelled node. Given a key z and a binary tree T , consider z as a tree
consisting just of one leaf. Find the position, using tree search, at which z should
be at T . If x is the node to act as parent of z then define the pointers as it was
necessary.
66 3 Fundamental Algorithms

Deletion of a labelled node. Given a vertex x for deletion in a binary tree T , let us
consider three cases according to the number of children of x in T . If x has no
children, i.e. it is a leaf, just suppress it and redefine the pointer to x of its parent.
If x has one child, just overskip it, i.e. suppress it and redefine the pointers of its
parent and its child. If x has two children then its right son y should be the left
son of successor(y).
The binary trees entail thus procedures with expected logarithmic time complex-
ities. However, the worst cases may maintain linear complexities.
An elaborated technique is given by the red-black trees that maintains all trees
balanced, and consequently even the worst case cases have logarithmic complexity,
but this technique is out of the scope of the current presentation.

3.7 Representations of graphs

A graph is a structure G = (V, E) where V is a non-empty set of vertexes or nodes

and E is a set of edges. If E ⊂ V (2) , i.e. the edges are 2-sets of vertexes, the graph
G is called undirected, while ifE ⊂ V 2 , i.e. the edges are ordered pairs of vertexes,
the graph G is called directed or a digraph. In this case, if (x, y) ∈ E is an edge, x is
called the tail of the edge and y its arrow. In general, the sole term graph will refer
to an undirected graph.
A weighted graph is G = (V, E, w) where V is a set of vertexes, E is a set of edges
and w : E → R associates a weight wa ∈ R to each edge a ∈ E.
In a graph G = (V, E), the degree or valency, ∂ (v), of a vertex v ∈ V is the number
of edges incident to v. A vertex of degree zero is an isolated vertex.
In a digraph G = (V, E), the in-degree ∂ + (v) of a vertex v ∈ V is the number of
edges in which v acts as arrow, while the out-degree ∂ − (v) of v is the number of
edges in which v acts as tail.
A path in a graph G = (V, E) is a sequence v0 , . . . , vk such that ∀i ≤ k: {vi−1 , vi } ∈
E, i.e. any pair of consecutive vertexes is an edge. A path is a cycle if its extreme
points coincide vk = v0 .
For any vertex v ∈ V , the connected component of v is the set ConG (v) consisting
of the extreme vertexes of paths starting at v. The graph is connected if for any
pair of vertexes there is a path connecting them. The graph is connected if it is the
connected component of any of its vertexes.

3.7.1 Adjacency-lists

Let G = (V, E) be a graph. For each vertex v ∈ V , its adjacency-list is Adj(v) =

[u]{v,u}∈E . Thus Adj(v) ∈ V ∗ . If G is weighted, then the adjacency-list is Adj(v) =
3.8 Depth- and breadth-first traversals 67

[(u, w({v, u}))]{v,u}∈E and has type (V ×R)∗ . The length of Adj(v) is the degree ∂ (v)
of v. Thus the whole graph can be represented by a structure of type (V ∗ )∗ .
For a directed graph G = (V, E) its adjacency-list is Adj(v) = [u](u,v)∈E . The
length of Adj(v) is the in-degree ∂ + (v). The calculation of out-degrees is rather
expensive since it is necessary to check all adjacency-lists.

3.7.2 Adjacency-matrices

Let G = (V, E) be a graph. The adjacency-matrix of G is


1 if {v, u} ∈ E
MG = [mvu ]v,u∈V , mvu =
0 if {v, u} 6∈ E

Thus, MG ∈ {0, 1}n×n , where n = card (V ) is the number of vertexes in the graph,
and the number of 1’s appearing in MG is 2m where m = card (E) is the number of
edges in the graph. Let us remark that Mg is a symmetric matrix and the entries at
the diagonal are all zero. Thus, instead of storing the whole matrix MG it is worth to
store just the 12 n(n − 1) entries above the main diagonal.
The adjacency matrix of a digraph is defined similarly, but it is not necessarily
symmetric.

3.8 Depth- and breadth-first traversals

Let G = (V, E) be a graph. Given a vertex v ∈ V it is required to calculate ConG (v).

In order to solve this problem, a representation by adjacency-lists will be used,
as well as a coloring of vertexes in three colors: white, gray and black.
Initially all vertex are painted in white. A vertex is discovered the first time it is
painted non-white. The calculation process will keep an invariant condition:
If (u, v) ∈ E is an edge and u is black, then v must be either gray or black, i.e. no pair white-
black is allowed to be adjacent. In other words, the adjacent vertexes to black colors shall
be discovered already.

The invariant condition states that at any time the white part has not yet been dis-
covered, the black region has been discovered and the gray region separates those
former regions.
The procedure uses a queue working in a principle “First-In–First-Out”. It is
sketched in figure 3.4.
The steps 3. and 4.d essentially just paste the adjacency-lists of s and u.
The procedure determines a tree structure in the connected component of the
source vertex s and it is indeed traversing the tree in a breadth-first traversal.
68 3 Fundamental Algorithms

BreadthFirst
Input. A graph G = (V, E) as a list of adjacency-lists and a source vertex s ∈ V .
Output. The connected component ConG (v).
1. Initially paint the source point gray ;
2. let d(s) := 0 ;
3. enqueue all pairs (s, v) ∈ E ;
4. Repeat
a. dequeue an edge {v, u} painted (gray,white) ;
b. paint it (black, gray) ;
c. declare v as the parent of u ; let du := dv + 1 ;
d. enqueue all pairs (u, w) ∈ E with w white ;
until the queue is empty ;
5. Output the black region

Fig. 3.4 Procedure BreadthFirst.

In contrast, a depth-first traversal proceeds using backtracking and a stack, work-

ing in a principle “First-In–Last-Out”, as an auxiliary storage device. In section 2.4
we have seen several examples of depth-first traversals.
Let G = (V, E) be a digraph. A precedence map is a function π : V → V such that

∀v ∈ V : π(v) 6= nil ⇒ (π(v), v) ∈ E.

For a precedence map π, let Gπ = (V, Eπ ) be the digraph with edges (π(v), v), v ∈ V .
Gπ is a subgraph of G, probably not-connected, hence it is a forest, and it is called a
depth-first forest.
In order to build such a forest, let us use, as before, a vertex coloring with three
colors: white, gray and black. Besides, as map d before, let us use two time marks:
dv is the discovering time of vertex v, while fv is the moment when its adjacency-list
has been exhausted and v is turning black. In figure 3.5 we show an algorithm for
calculation of a depth-first forest.

DepthFirst
Input. A digraph G = (V, E) as a list of adjacency-lists.
Output. A predecessor map and the corresponding depth-first forest.
1. For each v ∈ V Do {color(v) := white ; πv := nil} ;
2. time := 0 ;
3. For each v ∈ V Do If color(v) := white Then TraverseDepthFirst(v)
;
4. Output maps π, d and f

Fig. 3.5 Procedure DepthFirst.

The algorithm DepthFirst has an initialization process and thereafter it calls

the recursive program TraverseDepthFirst displayed in figure 3.6.
3.9 Topological sort 69

TraverseDepthFirst
Input. A digraph G = (V, E) and a vertex v ∈ V .
Output. Depth-first traversal of the tree rooted at v.
1. color(v) := gray ;
2. time + + ; dv := time ;
3. For each u ∈ Adj(v) Do
a. If color(u) := white Then {πu := v ; TraverseDepthFirst(u)} ;
b. color(u) := black ;
4. time + + ; fv := time

Fig. 3.6 Procedure TraverseDepthFirst.

Observe that steps 3. at DepthFirst and 3. at TraverseDepthFirst are

realizing a backtracking procedure. In TraverseDepthFirst, the steps 2. and
4., the maps d and f are defined, while the predecessor map π is actually defined at
step 3.a.
The whole process has time complexity Θ ( card (V ) + card (E)).

3.9 Topological sort

Let G = (V, E) be a directed acyclic graph, a dag for short. A topological sort is an
enumeration e : V → [[1, n]] such that

∀(u, v) ∈ E : e(u) < e(v).

If we write vi = v whenever i = e(v), then above condition means

∀i, j ∈ [[1, n]] : [(vi , v j ) ∈ E =⇒ i < j.] (3.6)

This condition entails that the graph may be drawn in a horizontal line and all ver-
tices are going form left to right. Obviously, a digraph with cycles cannot posses a
topological sort.
In order to construct a topological sort we may use the DepthFirst procedure
shown at figure 3.5. Namely, apply this algorithm on G = (V, E) in order to obtain
the predecessor map π, the initial-times map d and the ending-times map f of a
depth-first traversal of the graph G. Then a sorting of the set of vertexes V according
to f defines a topological sort in G.
Consequently, the topological sort can be obtained in time complexity Θ ( card (V )+
card (E)).
70 3 Fundamental Algorithms

3.10 Shortest-path algorithms

Let G = (V, A) be a graph, and for any two vertexes v, u ∈ V let

δ (v, u) = min{n|∃c = [u0 , . . . , un ] path : u0 = v, un = u}.

δ (v, u) is the distance of the shortest path connecting v with u in G. δ (v, u) = ∞ if

there is no a path from v to u.
Lemma 3.1. The following propositions hold in a graph:
1. If (v, u) ∈ E the for any vertex w: δ (v, w) ≤ δ (u, w) + 1.
2. If du is the value calculated at BreadthFirst(G, v), then

∀u ∈ V − {v} : du ≥ δ (v, u).

3. Let us assume that at a given moment, the queue is [u1 , . . . , uk ] in BreadthFirst.

Then,

duk ≤ du1 + 1
∀i < k : dui ≤ dui+1

From here, we have:

Theorem 3.1 (BreadthFirst correctness). In any graph G = (V, E), the algo-
rithm BreadthFirst(G, v) discovers all vertexes connected with v in G, for each
u ∈ ConG (v), the value du will equal δ (v, u) and the shortest path from v to u can be
recovered from the tree structure introduced in ConG (v).
The most used procedure for shortest paths calculation is Dijkstra’s algorithm.
Given a directed weighted graph G = (V, E, w), with non-negative weights, and a
source vertex s ∈ V it gives the distance of the shortest path from s to any other
vertex in V (if a vertex is not connected with s by a path then this distance is ∞).
The algorithm proceeds inductively: if a vertex x is not yet reached from s, then
the shortest path connecting x with s is the path passing through an already reached
intermediate point y minimizing the addition of the shortest path from s to y and the
weight of the connection between y and x. The algorithm is displayed in figure 3.7,
which has quadratic time complexity.

3.11 Transitive closure

The Floyd-Warshall algorithm, in contrast with Dijkstra’s algorithm, produces the

shortest path among every pair of vertexes.
In a weighted digraph G = (V, E, w), with non-negative weights, n = card (V ),
for a (n × n)-matrix D = [di j ]i, j≤n , representing current distances among pairs of
points, a triangle operation involving three vertexes is
3.11 Transitive closure 71

DijkstraAlgorithm
Input. A weighted digraph G = (V, E, w), with non-negative weights, and a
source vertex s ∈ V .
Output. The distance of the shortest path from s to any other vertex in V .
1. Reached := {s} ; ρ(s) := 0 ;
2. For each x ∈ V − {s} Do ρ(x) := w(s, x) ;
3. While Reached 6= V Do
a. let x := arg min{ρ(y)| y ∈ V − Reached} ;
b. Add x into Reached ;
c. For each y ∈ V − Reached Do ρ(y) := min{ρ(y), ρ(x) + w(x, y) ;
4. Output the array ρ

Fig. 3.7 Procedure DijkstraAlgorithm.

Triangle(D; i, j, k) ≡ (dik := min{dik , di j + d jk }) whenever i, k 6= j, (3.7)

with the connotation: if the weight diminishes, replace the segment from vi to vk by
two segments through the intermediate point v j .
If the triangle operation is iterated over all j’s and realized for all other pairs i, k,
beginning with the weights at the graph, then the distances will take the values of the
shortest paths. The algorithm is displayed in figure 3.8, which has time complexity
O(n(n − 1)2 ).

FloydWarshallAlgorithm
Input. A weighted digraph G = (V, E, w), with non-negative weights.
Output. A matrix D ∈ Rn×n giving the distances of the shortest paths among
vertex pairs in V .
1. For i = 1 To n Do For k = 1 To n Do
If i == k Then dii := ∞ Else dik := w(vi , vk ) ;
2. For each j = 1 To n Do
a. For each i ∈ [[1, n]] − { j} Do
i. For each k ∈ [[1, n]] − { j} Do Triangle(D; i, j, k) ;
3. Output the array D

Fig. 3.8 Procedure FloydWarshallAlgorithm.

By keeping track of the changes made by the triangle operation at step 2.a.i, as
defined by relation (3.7), then it is possible to recover the actual shortest path among
any pair of vertexes.

Given a digraph G = (V, E), its transitive closure is the graph G∗ = (V, E ∗ ) such
that

∀i, j ∈ V : [(vi , v j ) ∈ E ∗ ⇐⇒ ∃[u0 , . . . , uk ] path : u0 = vi & uk = v j ] (3.8)

72 3 Fundamental Algorithms

The transitive closure is thus the graph of paths in G.

If MG is the adjacency matrix of G, then the Boolean power MG2 = MG · MG
represents the graph of paths of length 2, MG3 = MG2 · MG represents the graph of
paths of length 3, and so on. Since there exists a power k ∈ N such that MGk+1 = MGk
we have MG∗ = MGk . This provides an effective, although expensive, procedure to
compute the transitive closure of G.
The Floyd-Warshall algorithm can be used as an alternative to compute G∗ .
Namely, let us assign a weight w(vi , v j ) = 1 to each edge (vi , v j ) ∈ E and an in-
finite weight to pairs not belonging to E. Then, after applying the Floyd-Warshall
algorithm, we have:

∀i, j ∈ V : [(vi , v j ) ∈ E ∗ ⇐⇒ di j < n] .

3.12 Minimum spanning tree

Let G = (V, E, w) be a weighted graph. A tree T = (U, F) within G is a subgraph,

U ⊆ V , F ⊆ E, connected without cycles. A spanning forest of G is a collection of
trees {(Ui , Fi )}ki=1 within G such that {Ui }ki=1 is a partition of V , i.e. V = ki=1 Ui
S

and [i 6= j ⇒ Ui ∩ U j = 0]./ A spanning tree of G is a subtree T = (U, F) such that

U = V , i.e. {T } is a spanning tree. The weight of a spanning tree is the sum of the
weight of edges in F, w(T ) = ∑{v,u}∈E w(v, u).
The minimum spanning tree problem (MST) consists in finding the spanning tree
of minimum weight:

Find Tm = arg min{w(T )| T spanning tree in G}.

2
This is a combinatorial problem whose search space is extremely huge, O(nn ). Thus
any brute-force approach is unrealistic.
However, a clever remark allows to solve this problem efficiently:

Proposition 3.1. If {(Ui , Fi )}ki=1 is a spanning forest in G and {v, u} ∈ E is a lowest

weight edge withSonly one extreme in the first tree U1 , then among the spanning trees
containing F = ki=1 Fi there is one optimal, i.e. with least weight, containing the
{v, u}.

Although it is not hard to prove this proposition, we skip the proof (the interested
reader may consult [Papadimitriou and Steiglitz(1998)]).
The resulting algorithm is direct: Starting with the forest consisting of trees
formed by the vertices with no edges, {({v}, 0)} / v∈V , choose the edge with mini-
mum weight and initiate a tree with this edge. Thereafter to having the current tree,
find the minimum weight edge with just an extreme in this tree and extend the tree.
The algorithm is displayed in figure 3.9, which has time complexity O(n2 ).
3.12 Minimum spanning tree 73

PrimMST
Input. A weighted graph G = (V, E, w), as a distance matrix W = [wi j ]i j .
Output. A spanning tree of minimal weight.
1. let {v, u} ∈ E be the minimal weight edge ;
2. U := {v} ; F := 0/ ;
3. For each u ∈ V −U Do closestU (u) := v ;
4. While U 6= V Do
a. let v := arg min{d(u, closestU (u))| u ∈ V −U} ;
b. U := U ∪ {v} ; F := F ∪ {{v, closestU (v)}} ;
c. For each u ∈ V −U Do
If d(u, closestU (u)) > d(u, v) Then closestU (u) := v ;
5. Output (U, F)

Fig. 3.9 Procedure PrimMST.

Chapter 4
Distributed Algorithms

Abstract This is a short introduction to distributive algorithms. The area is a topic

that requires a complete textbook by itself. However we would illustrate some basic
procedures for analysis and design of distributive algorithms. First, we sketch the
most used connectivity topologies and the formal tools to analyze distributive pro-
cedures. Then we proceed to examine concurrency and the typical access for data
input and output into a shared memory, and we compare several models. Finally we
overview routing algorithms and the solution of Graph Theory and Flowing prob-
lems within the context of distribution.

4.1 Distributed systems

A distributed system consists of a set of processors with a communication platform

forming a network. The connection may be physical or just logical: A set of pro-
cesses are connected through a message passing mechanism.
In this chapter we will sketch basic notions of distributed computing, and we
refer the interested reader to extensive texts in this area, e.g. [Schnitger(2007)].

4.1.1 Networks

A network is properly a graph G = (V, E). The graph is undirected if the commu-
nications along the edges is bidirectional and the graph is directed if just one-way
communications are allowed. Let us recall that the degree of a node is the num-
ber of its adjacent neighbors. The degree of a graph is the maximum degree of its
vertexes, the diameter of G is the maximum distance among any two vertexes on
the graph and the bisection width is the minimal number of edges to be removed in
order to split G into two disjoint subgraphs G0 = (V0 , E0 ), G1 = (V1 , E1 ) such that
V = V0 ∪V1 and | card (V1 ) − card (V0 )| ≤ 1.

75
76 4 Distributed Algorithms

4.1.1.1 Tree networks

Let T0 be the tree consisting of just one vertex. This vertex is the root as well as the
leaf of T0 and it has no edges.
For k ≥ 1, let Tk be the tree consisting of a root with exactly two children: each
being the root of a tree isomorphic to Tk−1 .
Thus Tk is the complete binary tree network. In Tk each vertex is able to com-
municate with any neighbor vertex: either its parent or its children. It has height k,
2k+1 − 1 vertexes and 2k leaves. Each vertex is either a leaf or it has exactly two chil-
dren. As a graph, it has diameter 2k and its bisection width is 1. As a network, the
root of Tk is a bottleneck since in order to connect two vertexes in different copies
of Tk−1 the root necessarily should be visited.
For instance, in order to select the minimum value among an array of n values
using a circuit Tk let us assign 2nk values to each leaf. Initially, in parallel, each leaf
selects the minimum value among its assigned values. Consecutively, each node
receives the minima of its two children and then it sends the minimum resulting
value to its parent. The root will have the minimum value of the whole array.
This procedure entails k communication steps.

4.1.1.2 Mesh networks

Let us consider the lattice Zdm with the neighboring notion given as differences lying
in the canonical basis of Rd : x and y are neighbors if and only if ∃i ∈ [[0, d − 1]],
ε ∈ {−1, +1}: x − y = εei . As a graph, Zdm has md vertexes, the degree of each
vertex is at most 2d and its diameter is d(m − 1). The bisection width is upperly
bounded by md−1 : along any coordinate i ∈[[0, d − 1]] let Ex , x ∈ Zm d−1 , be the edge
m
{y, z} such that the i-th entry of y is yi = 2 , the i-th entry of z is zi = m2 − 1, and
 

the remaining entries of y and z are given by x. For even m, the removal of all edges
d−1 , bisects the graph.
Ex , x ∈ Zm
For instance, let us consider the odd-even transposition sort (OETS) within a
linear (d = 1) array of m processors. Given an array of n numerical values initially
each processor is provided with a mn -length subarray. Then, in parallel, each proces-
sor sorts its subsequence using Quicksort. Consecutively in m steps, let us pro-
ceed as follows: at odd indexed steps, each even numbered processor p2i sends its
sorted sequence to its left neighbor p2i−1 , which merges both subsequences, keeps
the smaller half and sends back the larger half to p2i ; at even indexed steps, each
even numbered processor p2i sends its sorted sequence to its right neighbor p2i+1 ,
which merges both subsequences, keeps the larger half and sends back the smaller
half to p2i .
OETS is accomplishing its task in O mn log mn + n computing steps and m com-

munication steps involving messages of length mn .

4.1 Distributed systems 77

4.1.1.3 Hypercubes

Let Zd2 be the d-dimensional hypercube. Its edges are of the form {x, x + ei } where
w ∈ Zd2 and ei is a vector in the canonical basis and addition is assumed modulo 2.
Zd2 is a regular graph of degree d and its diameter is also d.
For each k ≤ d, there are dk 2d−k subgraphs isomorphic to Zk2 within Zd2 . Hence,

there are d 2d−1 edges in Zd2 .

If A ⊂ Zd2 let C(A) be the set of edges to be suppressed in order to disconnect A
and its complement Ac = Zd2 − A. Let us observe:

card (A) ≤ card (Ac ) =⇒ card (A) ≤ card (C(A)). (4.1)

Namely, let us prove implication (4.1) by induction on d.

For d = 1, the hypothesis in (4.1) entails card (A) ≤ 1, hence the implication is
obvious.
Let d > 1. Let us assume (4.1) for d − 1. Let A ⊂ Zd2 be such that
card (A) ≤ card (Ac ). Let A0 consist of those points in A whose first coordinate has
value 0, and let A1 consist of those points in A whose first coordinate has value 1.
Clearly, {A0 , A1 } is a partition of A and each subset Aε in an own copy, say Zε , of
the (d − 1)-dimensional hypercube Z2d−1 , for ε ∈ {0, 1}.
If card (A0 ) ≤ 2d−2 and card (A1 ) ≤ 2d−2 then, by the induction hypothesis,
card (A0 ) ≤ card (C(A0 )) and card (A1 ) ≤ card (C(A1 )). Obviously,
card (C(A0 )) + card (C(A1 )) ≤ card (C(A)). Hence card (A) ≤ card (C(A)).
Otherwise, let us assume that card (A0 ) > 2d−2 . Then card (A1 ) < 2d−2 . By the
induction hypothesis, in Z2d−1 or properly Z1 , card (A1 ) ≤ card (C(A1 )).
For each x ∈ Z2d−1 , (0x, 1x) is an edge in the hypercube, thus in order to split A
and its complement in Zd2 we must exclude the edges such that

[0x ∈ A0 & 1x 6∈ A1 ] & [0x 6∈ A0 & 1x ∈ A1 ]

as well as the splitting edges of A1 in Z1 . Hence

C(A) ≥ ( card (A0 ) − card (A0 ∩ A1 )) + ( card (Ac0 ) − card (Ac0 ∩ A1 )) + card (A1 )
= 2d−1 − card (A1 ) + card (A1 ).
= 2d−1

Thus, card (A) ≤ card (C(A)). u

t
As a consequence of the implication (4.1) we have:

Proposition 4.1. The bisection width of the hypercube Zd2 is 2d−1 .

Thus any pair among the 2d nodes in Zd2 can communicate within a route of
length at most d, in other words, the communicating length is logarithmic with re-
spect to the number of nodes, as is the case in the complete binary networks. How-
ever here there are no bottle-necks. Instead, the “connectedness degree” is rather
high in the hypercube, as seen in proposition 4.1.
78 4 Distributed Algorithms

Definition 4.1 (Routing procedures). Let G = (V, E) be a graph. For any two nodes
u, v ∈ V let Puv be the class of paths in G beginning in u and ending in v. A routing is
a map R : (u, v) 7→ R(u, v) ∈ Puv that associates to each pair (u, v) a path in G going
from u to v. For any permutation π ∈ SV a π-routing is a map ρπ : u 7→ ρπ (u) ∈ Pu,π(u)
giving a path, for each vertex u, going from u into π(u). If µu is the message that
node u should send to π(u) then ρπ is the route that message µu will follow.
In the hypercube G = Zd2 a direct routing procedure acts according to the “Bit-
Fixing Strategy”, and it is sketched in figure 4.1.

BitFixingRouting
Input. A permutation π ∈ SV
Output. A π-routing ρπ : u 7→ ρπ (u) ∈ Pu,π(u)

1. For each u ∈ Zd2 do (in parallel)

a. let ρπ (u) := BitFixingStrategy(u, π(u))
2. Output ρπ

Fig. 4.1 Procedure BitFixingRouting.

In the Bit Fixing Strategy, in order to go from a node u into another node v in
the hypercube Zd2 , the bits of u and v are compared from left to right: if there is a
discrepancy then the current point is moved along that discrepancy’s direction.

BitFixingStrategy
Input. Two nodes u, v ∈ V
Output. A path joining u with v
1. let crpt := u ; path := [crpt] ;
2. For each i ∈ [[1, d]] do
a. If crpt[[i]] 6= v[[i]] then
i. crpt := crpt + ei ;
ii. AppendTo[path, crpt] ;
3. Output path

Fig. 4.2 Procedure BitFixingStrategy.

Let (u0 , v0 ), (u1 , v1 ) be two pairs of vertexes. Let ρ0 ∈ Pu0 v0 , ρ1 ∈ Pu1 v1 be paths
joining the pairs. We say that there is a path crossing at step i ≤ d if ρ0 [[i]] = ρ1 [[i]].
Clearly BitFixingStrategy will produce paths with a crossing if the pairs
can be expressed as u0 = u0p wus0 , v0 = v0p wvs0 , u1 = u1p wus1 , v1 = v1p wvs1 , for some
non-empty subwords u’s and v’s, and v0p wus0 = v1p wus1 . In this case, this common
value determines a path crossing. At this crossing, BitFixingStrategy will
produce as many skips as the length of the infix w. From here it follows that any two
paths leaving a coincidence will never meet again.
4.1 Distributed systems 79

BitFixingRouting may be quite expensive since it should manage many

concurrences. For instance, let us assume that d is an even integer, d = 2 f , and
let us consider the permutation π : (u, v) 7→ (v, u). In the hypercube Zd2 there are
d √
2 f = 2 2 = 2d nodes of the form (u, 0). Thus when BitFixingRouting is
applied to them, at the f -th step they will pass through (0, 0), the origin is thus a
path crossing for all the paths and this vertex is becoming a bottleneck.
In order to avoid this difficulty, let us proceed with the random algorithm
sketched in figure 4.3.

RandomMiddle
Input. A permutation π ∈ SV
Output. A π-routing ρπ : u 7→ ρπ (u) ∈ Pu,π(u)

1. For each u ∈ Zd2 do (in parallel)

a. select randomly a node α(u) ;
b. as in BitFixingStrategy choose a path L0 from u to α(u) ;
c. as in BitFixingStrategy choose a path L1 from α(u) to π(u) ;
d. let us take the concatenation ρπ (u) := L0 ∗ L1 ;
2. Output ρπ

Fig. 4.3 Procedure RandomMiddle.

Since each vertex v in the hypercube has nk neighbors at distance k, k ∈ [[0, d]],

the expected length of any path connecting two points is

d d

1 d d! d d−1 (d − 1)! d
∑ k 2kd = 2d ∑ k k!(d − k)! = 2d ∑ k!(d − 1 − k)! = 2 .
k=0 k=1 k=0

Let R be a routing strategy. For each edge a = {u, v}, let Wa be the number of paths,
connecting pairs {w, π(w)}, that pass through the edge a. Then, for each edge a, the
expected value of Wa is
d
2 d 2d
E(Wa ) = ∑ = = 1.
w d2d−1 2 d2d−1

Now, let for each two vertexes u, v ∈ Zd2 ,


1 if R(u) and R(v) share an edge
Huv =
0 otherwise

Then, for each u, E(∑v Huv ) ≤ d2 E(Wa ) = d2 . Clearly, ∑v Huv is the number of “de-
lays” that the message µu should wait due to “edge collisions”. According to Cher-
nov’s inequalities [Schnitger(2007)], for any β > 0,
80 4 Distributed Algorithms
 
d β2 d
Pr ∑ Huv ) ≥ (1 + β ) ≤ e− 3 2 .
v 2

Hence, for β = 3, the probability that
a message µu is delayed at least 2d time

units is upper bounded by exp − 32 d . Consequently, the probability that some µu
requires at least 2d time units is upper bounded by
      
3 3 d
2d exp − d = exp d ln 2 − ≤ exp − .
2 2 2

Thus, with probability at least 1 − 2 exp − d2 the first stage 1.b of procedure

RandomMiddle is done in 3d time units and the whole procedure is done in 6d

time units.

4.1.1.4 Butterflies

Let Bd = Zd2 × [[0, n]] be the Cartesian product of the hypercube Zd2 and the set con-
sisting of the first n + 1 natural numbers. Let

Rx = {x} × [[0, n]] , C j = Zd2 × { j}

be the x-th row and the j-th column, or level, of Bd . Let

Ehd = {{(x, j), (x, j + 1)}| j ∈ [[0, n − 1]], x ∈ Zd2 } : horizontal edges
Ecd = {{(x, j), (y, j + 1)}| j ∈ [[0, n − 1]], x ∈ Zd2 , y = x + e j } : crossing edges

The graph (Bd , Ehd ∪Ecd ) is called the d-dimensional butterfly network. In figure 4.4
we sketch the graphs of the butterflies for d ∈ [[1, 4]].
The butterfly can be realized as transitions in the hypercube made in parallel.
Also, it can be seen that a d-dimensional butterfly contains two subgraphs isomor-
phic to the (d − 1)-dimensional butterfly.
A wrapped butterfly is obtained by substituting [[0, n]] by Zn and letting addition
of indexes modulo n (thus the last level j = n in the butterfly is identified with the
initial level j = 0). Let WBd denote the d-dimensional wrapped butterfly network.
We have:
• The butterfly network Bd is a graph of order 4 (each inner node is the extreme of
4 edges, and the nodes at the extreme levels have 2 neighbors). The number of
nodes in Bd is (d + 1) 2d and the number of edges is d 2d+1 . The diameter of Bd
is 2d and its bisection width is Θ (2d ).
• The wrapped butterfly network WBd is a regular graph of order 4. The number of
nodes in WBd is d 2d and the number of edges is d 2d+1 . The diameter of WBd is
b 3d d
2 c and its bisection width is 2 .
4.1 Distributed systems 81

d=1 d=2

d=3 d=4

Fig. 4.4 d-dimensional butterfly networks for d ∈ [[1, 4]].

4.1.2 Transition systems

The communication protocols involve proof methods for safety and liveness prop-
erties, as well as event dependency, message transmission and network routing as
well as avoidance of store-and-forward packet deadlocks.
A transition system is a triplet T = (C, T,C0 ) where C is a non-empty set of
configurations, T ⊂ C ×C is a transition relation (usually, if (cb , cc ) ∈ T it is written
cb → cc ) and C0 ⊂ C is a subset of initial configurations. A partial execution is a
sequence [c0 , c1 , . . . , ck ] of configurations such that c0 ∈ C0 and ∀i > 0, ci → ci+1 . In
∗ ∗
this case, we write c0 → ck . An execution is a maximal partial execution. If c0 → ck
through an execution then ck is called an ending configuration. A configuration c ∈ C
∗
is reachable if ∃c0 ∈ C0 : c0 → c.
A property Φ on states is a safety requirement if it holds on any reachable state
at an execution and it is a liveness requirement if it holds on some reachable state at
an execution.
For two properties Φ,Ψ on states, we write {Φ} → {Ψ } if the following impli-
cation holds:
[(cb → cc ) & Φ(cb )] =⇒ Ψ (cc ).
82 4 Distributed Algorithms

An invariant requirement is a property Φ that holds in the initial configurations

and holds in any configuration in a partial execution, ∀c ∈ C0 , Φ(c), and besides,
{Φ} → {Φ}. Thus an invariant property holds in each reachable configuration at
any execution.

4.1.3 Distributed systems

A local algorithm, or process, is a transition system whose configurations are called

states and whose transitions are called events. The events are of three types: internal,
send and receive. Thus, if S p is the set of states and M the set of messages, then

→i,p ⊂ S p × S p
∀µ ∈ M : →s,µ,p ⊂ S p × S p
∀µ ∈ M : →r,µ,p ⊂ S p × S p


Let → p =→i,p ∪ →s,µ,p ∪ →r,µ,p , i. e.,
S
µ∈M


∀c, d ∈ S p : (c → d) ⇐⇒ (c →i,p d) or ∃µ ∈ M c →s,µ,p d or c →s,µ,p d .

A distributed system is a collection of process. The collection of states is the Carte-

sian product of the processes states. The communication subsystem is a multiset of
messages: each pair of processes has associated, at each step, a message. Formally,
let P be a collection of processes:
States. C = ∏ p∈P C p × ∏ p∈P M p , where for each p ∈ P, M p = M ∗ is the collection
of finite sequences of messages.
Transitions. These are defined according to the type of events at each process.
Namely, 
 

(cq )q∈P , µq∗ q∈P → (dq )q∈P , νq∗ q∈P (4.2)

where

(∀q ∈ P − {p} : dq = c
q ) &
c p →i,p d p =⇒ (4.3)
∀q ∈ P : νq∗ = µq∗

 (∀q ∈ P − {p} : dq = cq )
&
c p →s,µ,p d p =⇒ ∀q ∈ P − {p} : νq∗ =
µq∗ & (4.4)
ν p∗ = µ p∗ − {µ}


 (∀q ∈ P − {p} : dq = cq )
&
c p →r,µ,p d p =⇒ ∀q ∈ P − {p} : νq∗ =
µq∗ & (4.5)
ν p∗ = µ p∗ ∪ {µ}

4.1 Distributed systems 83

It is said that the events at the left sides of (4.3-4.5) are applicable at the configura-
tion on the left side of (4.2), and each produces the configuration at the right side.
As described above, the distributed system is called asynchronous.

Proposition 4.2 (Asynchronous message passing). Let γ be a configuration and

let e p , eq be local events at different processes, both applicable in γ. Then eq is
applicable in e p (γ), e p is applicable in eq (γ) and eq (e p (γ)) = e p (eq (γ)).

Instead, in a synchronous distributed system the set of states is C = ∏ p∈P C p and

the transitions have the form (cq )q∈P → (dq )q∈P where

c p →i,p d p =⇒ ∀q ∈ P − {p} : dq = cq (4.6)


c ps →s,µ,ps d ps
=⇒ ∀q ∈ P − {ps , pr } : dq = cq . (4.7)
c pr →r,µ,pr d pr

In event (4.7) the local events at the left side are called corresponding.
Let Γ = (γκ )kκ=0 be an execution in a synchronous distributed system. For each
κ > 0 let eκ be the event such that γκ = eκ (γκ−1 ). Let Γ = (eκ )kκ=1 be the corre-
sponding sequence of events.
Let Γ be the smallest ordering relation in the collection of local events satisfy-
ing the following conditions:
• If e0 and e00 are events in the same process and e0 is applied before than e00 in the
execution Γ , then e0 Γ e00 .
• If e0 is a send event and e00 is the corresponding receive event in the execution Γ ,
then e0 Γ e00 .
“≺Γ ” is called the causal order determined by Γ .
Two events e0 and e00 are concurrent, e0 ke00 , if neither e0 Γ e00 nor e00 Γ e0 .
Let π : [[1, k]] → [[1, k]] be a permutation consistent with the clausal order:
 
∀i, j ∈ [[1, k]] : eπ(i) Γ eπ(i) =⇒ i ≤ j .

k
Let ∆ = dκ = eπ(κ) κ=1
. Then ∆ determines an execution ∆ in the distribution
system.

Proposition 4.3. ∆ and Γ have the same length and the last configuration of ∆
coincides with the last configuration of Γ .

∆ and Γ are thus equivalent executions. This notion defines, indeed, an equiva-
lent relation on the whole collection of executions.
A logical clock is a map ω : {events} → N that is increasingly monotone with
respect to a clausal ordering  and the usual ordering in N.
Let us see some examples of distributed algorithms.

Example 4.1 (Unidirectional leader election). Let us assume a ring of processors.

There are n processors, p0 , . . . , pn−1 , and each, pi , is able to send a message to its
successor, pi+1 (where addition in the indexes is taken modulus n).
84 4 Distributed Algorithms

The goal of the algorithm is that some processor eventually will claim ’I
am the leader’, and, if required, any other will claim ’I am not the
leader’. The setting is the following:
Messages. Processors UID’s (these are labels in N) and an empty message, to
represent that no message is sent.
States. {⊥, leader}.
Initial states. Each processor knows its own UID and is in state ⊥.
Each process acts as follows:
1. Let v := Receive ;
2. If v >UID Then Send(v)
Else
a. If v =UID Then leader
Else
i. Do Nothing
Clearly, the elected leader would be that with greatest UID and the elected proces-
sor will realize that within n rounds. This process involves O(n2 ) message transmis-
sions.

Example 4.2 (Bidirectional leader election). Also in this case, let us assume a ring
of n processors, p0 , . . . , pn−1 , and each, pi , is now able to send a message to both its
successor, pi+1 , and its predecessor, pi−1 (indexes are taken modulus n).
As before, the elected leader would be that with greatest UID. The messages
should be pairs
1. For each round k ≥ 1 Do
a. each processor sends its UID to both sides with the aim to reach processors at
distance 2k and return to the original processor,
i. If the traveling UID is less than the UID of a visited processor, the traveling
UID is lost,
ii. Once the traveling UID returns to its original processor, a new round be-
gins.
b. If a traveling UID arrives to its original processor in the “on-going” direction,
then that node claims itself as the leader.
Each message consists of the UID and two indexes: The first tracks the number of
visited processors in the “on-going” direction, while the second tracks the number
of visited processors in the “returning” direction.
We may see that the leader will be elected within 1 + dlog2 xe rounds and the
total number of sent messages is 8n(1 + dlog2 xe) = O(n log n). Indeed initially each
processor sends at most 4 messages, and at each new round k, the processors thats
end messages are those that have not been defeated, and at each group of 2k−1 + 1
only one goes to the next round, hence they will remain just b 2k−1n +1 c processors,
hence the number of messages in that round will be 4 · 2k · b 2k−1n +1 c ≤ 8n.
4.2 Concurrency 85

Example 4.3 (Breadth-first search). Let G = (V, E) be a directed graph and let s ∈ V
be a source vertex. A breadth-first traversal of G is a spanning tree rooted at s such
that any node at distance d from s is at height d in the tree.
Initially the source vertex is declared as the only marked node and it sends a
search message to its neighbors. Thereafter, when a non-marked node receives a
search message then it is declared as a marked node, it declares the transmitting
node as its own parent, and it sends a search message to its neighbors.
Clearly, the algorithm produces a tree which is indeed a breadth-first traversal
similar as the traversal produced by the procedure at figure 3.4. At each round,
any vertex with distance (i.e. the length of the shortest path connecting it with s) not
exceeding the round index has been marked and an assigned parental. The algorithm
has time complexity O(k) where k is the diameter of the graph and sends O(e)
messages, where e is the number or edges in the graph.

4.2 Concurrency

A distributed system, consisting of several processors, may involve some shared

resources, as memory or input-output devices. The concurrency problems are related
to (simultaneous) access to shared resources.

4.2.1 Exclusive and concurrent access

Let C be a class of processors, e.g. PRAM’s, Turing machines or while-programs,

see for instance, [Kfoury et al(1991)Kfoury, Arbib, and Moll]. Let us consider now
as the only shared resource a shared memory. Thus, besides the common read-write
instructions into local memory devices, let us consider also general instructions, al-
lowing processors to access the shared memory, read* and write* into registers
of the shared memory. A conflict appears when two processors try to access the
same register in order to commit a read or write action. Let us define the following
program classes:
EREW-C (Exclusive Read-Exclusive Write) : Programs in C with neither read
nor write conflicts.
CREW-C (Concurrent Read-Exclusive Write) : Programs in C with no write
conflicts.
ERCW-C (Exclusive Read-Concurrent Write) : Programs in C with no read con-
flicts.
CRCW-C (Concurrent Read-Concurrent Write) : Programs in C with no read-
write restrictions.
For any subclass D ⊆ C and any two functions p,t : N → N, let D(p(n),t(n)) denote
the class of problems in D running in time O(t(n)) requiring O(p(n)) processors and
86 4 Distributed Algorithms

an unbounded shared memory. Thus, for instance k,`∈N CRCW-C (nk , [log(n)]` ) is
S

the class of programs running in polylogarithmic time in a distributive system in-

volving a shared memory and a polynomial number of processors. Due to definition,

EREW-C (p(n),t(n)) ⊆ CREW-C (p(n),t(n)) ⊆ CRCW-C (p(n),t(n)) and

EREW-C (p(n),t(n)) ⊆ ERCW-C (p(n),t(n)) ⊆ CRCW-C (p(n),t(n)).

A program P ∈ C recognizes a word σ ∈ (0 + 1)∗ if on input σ the corresponding

computation of P ends with the value 1 in the first shared register. The recognized
language of P is L(P) ⊂ (0 + 1)∗ , obviously, consisting of all words that are recog-
nized by P. In this way, any program class D ⊆ C can be seen as a language class:
[L(P) ∈ D ⇐⇒ P ∈ D] .
A distributive system with shared memory can be simulated with a hypercube
using message-passing. Namely, suppose P ∈ CRCW-C (p(n),t(n)), and let k(n) =
dlog2 p(n)e. Then the processors for P can be included as vertexes of the k(n)-
dimensional hypercube. Using a hash function let us map the shared memory as the
union of extended local memories of processors in the hypercube. Then the program
P can be simulated by this hypercube. Hence:
An algorithm in CRCW-C (p(n),t(n)) can be simulated by a dlog2 p(n)e-dimen-
sional hypercube running in time O (t(n) · dlog2 p(n)e).
In order to provide some examples and to compare the introduced subclasses let
us consider three elementary problems:

Problem ZeroCounting.
Instance: A word σ = s0 · · · sn−1 ∈ 0+ 1+ .
Solution: The position i such that si = 0 and si+1 = 1.

Problem Boolean-OR.
Instance: A word σ ∈ (0 + 1)∗ .
Solution: max1≤i≤ len (σ ) si .

Problem Parity.
Instance: hA word σ i∈ (0 + 1)∗ .
len (σ )
Solution: ∑i=1 si mod 2 .

A recursive EREW distributive algorithm, EREWZeroCount, to solve the prob-

lem ZeroCounting is outlined in fig. 4.5.
The time complexity of EREWZeroCount is determined by the recurrence
t(n) = t( n2 ) + 1, thus t(n) = O(log n). The number of required processors is linear
with n. But it is a rough measure.
As a new complexity measure, let w : n 7→ ∑i ti (n)pi (n) be the work of the dis-
tributed algorithm, where i runs over the steps of the algorithm, ti is the time required
at the i-th step and pi is the number of involved processors.
The work complexity of EREWZeroCount is determined by the recurrence
w(n) = w( 2n ) + n2 , thus w(n) = O(n).
4.2 Concurrency 87
EREWZeroCount
Input. A word σ = s0 · · · sn−1 ∈ 0+ 1+ .
Output. The position ZC = i such that si = 0 and si+1 = 1.
1. n := len (σ ) ;
2. If n ≤ 2 then output 0
Else
a. For i = 1 To d n−1
2 e − 1 Do (in parallel)
i. ti := ’0’
ii. If s2i == ’1’ && s2i+1 == ’1’ Then ti := ’1’ ;
b. j0 := EREWZeroCount(τ) ;
c. If s2 j0 +1 == ’1’ Then i := 2 j0 Else i := 2 j0 + 1 ;
3. Output i

Fig. 4.5 Procedure EREWZeroCount.

A recursive EREW distributive algorithm to solve in a more general setting the

problem Boolean-OR is sketched in fig. 4.6, with the procedure EREWMaxArray.

EREWMaxArray
Input. A numeric array S
Output. The max-array [max j∈[[0,i]] S[[ j]]]i∈[[0, len (S)−1]]
1. n := len (S) ;
2. If n == 1 then output S
Else
a. For i = 0 To d n−12 e − 1 Do (in parallel)
{ T [[i]] := max{S[[2i]], S[[2i + 1]]} } ;
b. N := MaxArray(T ) ;
c. For i = 0 To n − 1 Do (in parallel)
i. Case odd i: M[[i]] := max{N[[(i − 1)/2]], S[[i]]} ;
ii. Case even i: M[[i]] := N[[i/2]] ;
3. Output M

Fig. 4.6 Procedure EREWMaxArray.

The time complexity of EREWMaxArray is determined by the recurrence t0 (n) =

t0 (n/2) + 1, thus t0 (n) = O(log n), while the number of required processors is deter-
mined by the recurrence p0 (n) = p0 (n/2) + n, hence p0 (n) = O(n).
It is worth to remark here that the same algorithm may be used to compute partial
sums in an array. Indeed, by changing each max operation by addition, we obtain the
partial sums. This modification may be used to compute the multiplication of two
(n×n)-matrices with n3 processors. Namely, for each index triplet (i, j, k) let an own
processor compute A[[i, k]] ∗ B[[k, j]]; then use the above mentioned modification to
calculate, for each pair (i, j), C[[i, j]] := ∑k A[[i, k]] ∗ B[[k, j]].
88 4 Distributed Algorithms

Instead, a first CREW distributive algorithm to calculate the maximum element

in an array of non-negative real numbers is sketched in fig. 4.7.

CREWFirstMax
Input. A numeric array S
Output. The pairs such that (i, S[[i]]) such that S[[i]] = max j∈[[0, len (S)−1]] S[[ j]]
1. n := len (S) ;
2. If n == 1 Then Output (0, S[[0]])
Else
a. For i = 0 To n − 1 Do (in parallel) A[[i]] := 1 ;
b. For Each pair (i, j) with 0 ≤ i < j ≤ n − 1 Do (in parallel)
{ If S[[i]] < S[[ j]] Then GT[[i, j]] := 0 Else GT[[i, j]] := 1 } ;
c. For Each pair (i, j) with 0 ≤ i < j ≤ n − 1 Do (in parallel)
{ A[[i]] := min(A[[i]], GT[[i, j]]) } ;
d. For i = 0 To n − 1 Do (in parallel)
{ If A[[i]] == 1 Then Output (i, S[[i]])

Fig. 4.7 Procedure CREWFirstMax.

The time complexity of CREWFirstMax is constant,

t1 (n) = O(1), while the

number of required processors is quadratic, p1 (n) = O n2 = O(n2 ).
As a second CREW algorithm, let us proceed recursively as in fig. 4.8.

CREWSecondMax
Input. A numeric array S
Output. The pairs such that (i, S[[i]]) such that S[[i]] = max j∈[[0, len (S)−1]] S[[ j]]
√
1. For i = 0 To n − 1 Do (in parallel)
√ √
a. Si := S[[ j| n i ≤ j < n (i + 1)]] ;
b. T [[i]] := CREWSecondMax(Si ) ;
2. mx := CREWFirstMax(T ) ;
3. Output mx

Fig. 4.8 Procedure CREWSecondMax.

The time √ complexity of CREWSecondMax is determined by the recurrence

t2 (n) = t2 ( n) + O(1), thus t2 (n) = O(log log n). Here we see that the cycles 1. and
2. in the algorithm may use the same processors. Thus, it is rather arbitrary to count
the number of involved processors.
√ √For CREWSecondMax the work is determined
by the recurrence w2 (n) = n w2 ( n) + O(n), hence w2 (n) = O(n log log n).
As a third CREW algorithm, let us section the given numeric array into pieces,
let us select the maxima of the pieces by EREWMaxArray and then let us select the
maximum in the maxima array by CREWSecondMax. See fig. 4.9.
The time complexity of CREWThirdMax is
4.2 Concurrency 89

CREWThirdMax
Input. A numeric array S
Output. The pairs such that (i, S[[i]]) such that S[[i]] = max j∈[[0, len (S)−1]] S[[ j]]
n
1. For i = 0 To log log n − 1 Do (in parallel)

a. Si := S[[ j| (log log n) i ≤ j < (log log n) (i + 1)]] ;

b. T [[i]] := EREWMaxArray(Si ) ;
2. mx := CREWSecondMax(T ) ;
3. Output mx

Fig. 4.9 Procedure CREWThirdMax.

 
n
t3 (n) = t0 (log log n) + t2
log log n
= O(log log log n) + O(log log n)
= O(log log n).

The work is
 
n
w3 (n) = w0 (log log n) + w2
log log n
= O(log log log n) + O(n)
= O(n).

Given an increasing numeric array (X[[i]])i∈[[1,n]] let us add as extreme values

X[[0]] = −∞ and X[[n + 1]] = +∞. For any value y ∈ R let R(y, X) = i if and only if
X[[i]] ≤ y < X[[i + 1]]. Clearly X[[0]] < y < X[[n + 1]].
Let us consider the following problem:
Problem Rank.
Instance: An increasing array (X[[i]])i∈[[1,n]] and a value y ∈ R.
Solution: The rank R(y, X).
A CREW distributed algorithm to solve Rank is displayed at pseudocode 4.10.
Let us remark here that the step 3. at pseudocode 4.10 is typical in the simulation
of EREW procedures by CREW procedures.

Remark 4.1. p values on a p-processor EREW procedure can be reduced to a CREW

procedure within O(log p) time.
p−1
Suppose that the array of values is [si ]i=0 . Let us construct a binary tree as fol-
lows: Initially let us correspond each leaf with a value, v0i ↔ {si }, for i ∈ [[0, `0 − 1]],
with `0 = p. Consecutively, for each k > 0 let vki be the parent of the left child vk−1,2i
`
and the right child vk−1,2i+1 , for i ∈ [[0, `k − 1]], with `k = k−1
2 , and let us associate
as label of the parent the concatenation of the labels of its children. Clearly the
constructed tree has height log2 p and the label of the root is the whole data array.
90 4 Distributed Algorithms

CREWRank
Input. An increasing numeric array X, a value y ∈ R and a fixed number of
processors p ∈ Z+ .
Output. R(y, X)
1. n p := n+1
p ;
2. ic := 0 ;
3. Repeat dlog p (n + 2)e times
a. For i = 1 To p Do (in parallel)
i. If X[[ic + (i − 1)n p ]] ≤ y ≤ X[[ic + in p − 1]] Then
n
{ ic := ic + (i − 1)n p ; n p := pp } ;
b. For i = 1 To p Do (in parallel)
i. If X[[ic + in p − 1]] < y < X[[ic + in p ]] Then
Output (ic + in p − 1)

Fig. 4.10 Procedure CREWRank.

Thus, for instance, the distributive algorithm, EREWZeroCount, in fig. 4.5

gives rise to a corresponding CREWZeroCount with time complexity at least
Ω (log2 p) and O(p) processors.
log n
We have that CREWRank runs in time O( log p ) with p processors, while in any
n
EREW scheme its time is at least Ω (log p ). Thus CREW is more powerful, in gen-
eral, than EREW.

4.2.2 Dining philosophers problem

This is a typical problem illustrating the appearances of deadlocks when some re-
sources are shared in a distributed environment.
Dining philosophers problem
At any moment, a dining philosopher is either thinking or eating. Since the menu consists
just of spaghetti, any philosopher should be provided with two forks in order to begin to eat.
Let us assume that p philosophers are sit around a circular table with the spaghetti bowl at
the center. In between two adjacent philosopher lies a fork. Thus in order to begin to eat,
each philosopher should use the two forks adjacent to him.

A deadlock situation appears when each philosopher is granted to use his left fork.
Each is waiting for his right fork to be free, and no philosopher is able to begin to
eat!
The deadlock may be broken by introducing a waiter. A waiter may schedule
an eating roll, thus he may assign fork pairs (and eventually to take care of forks
cleaning after any use!) to dining philosophers.
A very well known algorithm for deadlock breaking without the participation of
any external agents is the following:
4.3 Routing 91

1. Let us associate ID’s to the philosophers in a one-to-one fashion with ID’s in a

totally ordered set.
2. Let us put a dirty fork in between two neighbor philosophers and let us associate
it to the philosopher with least ID.
3. Whenever a philosopher wants to eat he sends requests for clean forks to both of
his neighbors.
4. Whenever a philosopher receives a request for a clean fork:
a. If he has a proper dirty fork, then he cleans it and allows the requesting neigh-
bor to use the cleaned fork.
b. If the proper fork is clean, then he keeps it and maintains his requesting neigh-
bor in a waiting state.
c. If he has no forks, he keeps on waiting.
5. When a philosopher gets two clean forks he begins to eat. When finishing, he
remains with two dirty forks.
In general, in order to break deadlocks, within a distributed system any processor
is provided with marked tokens and it may decide its access to shared resources by
the current configuration of its own tokens.

4.3 Routing

4.3.1 Byzantine Generals problem

In a battle field, a General gives an order to either attack or retreat. The order is
transmitted by the subordinates and all participants should agree on a value of the
order: Either to attack or to retreat. The General may be a participant and all subor-
dinates are participants as well. Among participants there are traitors that may alter,
when transmitting it, the sense of the order. Even the General may be a traitor. The
goal of the problem is thus to make agree all the non-traitor subordinates on the
General’s order.
In the language of distributed systems the problem is posed as follows: A source
processor transmits a bit. Among the processors there are correct and faulty proces-
sors. The faulty processors may change the value of the message bit. The goal of the
correct processors is to convene in the value of the original message bit emitted by
the source processor.
Let S denote the source processor. For any processors X,Y let

ΨS (X,Y ) : value claimed by Y , up to the knowledge of X, from S

ΦS (X) : received value by X from S
ΞS (X) : transmitted value by X allegedly arrived from S

Necessarily, ΦS (X) = ΨS (X, S). Clearly,

92 4 Distributed Algorithms

ΦS (X) 6= ΞS (X) =⇒ X ∈ Faulty ,

and for any Y 6= X, S:

ΦS (X) 6= ΨS (X,Y ) =⇒ Y ∈ Faulty or S ∈ Faulty .

In the above situation, for just three participants X,Y, S, it is not possible for X to
decide which of the other two participants is faulty.
Let p denote the number of processors and let q be the number of faulty proces-
sors. If 3q < p then an agreement protocol does exist.
Let us consider the following subprocedures:
OM(0)
Input. S source processor and a bit bS .
Output. bS transmitted to neighbors of S
1. For each neighbor Y of S Do bs is sent form S to Y ;
2. Y assumes bS
OM(m)
Input. S source processor and a bit bS .
Output. An agreed value of bS transmitted from S
1. S sends bS to all processors ;
2. For each processor X Do (in parallel)
a. bX = ΞS (X) ;
b. OM(m − 1) (X, bX ) (X acts as source and resends its bit value
to all processors different than S and X itself)
3. For each processor X Do (in parallel)
X assumes the most frequent value among (ΦY (X)| Y 6= X, S).

4.3.2 Routing algorithms

The Routing problem consists in select the “best” pat connecting two nodes in
a network. Any solving procedure should be correct (any path selected among two
nodes, should indeed connect them), efficient (the selected paths are optimal among
the paths connecting the nodes) and robust (the paths should be selected without
prior knowledge of the, possible varying, networks topology).
Any network may be realized as a graph G = (V, E) (either undirected or di-
rected). A cycle is a path whose extreme points coincide. A simple path is a path
without internal cycles.
In static networks a general strategy may be the computation of spanning trees.
For any node v ∈ V , let Tv be a spanning tree of the network rooted at v. Then for
each node u ∈ V in the connected component of v, let the branch connecting v with
u be the selected route.
4.3 Routing 93

At any modification of the network, the spanning tree should be recomputed.

In case of a weighted network, whose edge weights are either costs of traffic
loads, the Floyd-Warshall algorithm (shown in figure 3.8) computes the shortest dis-
tance path among any pair source-destination nodes. Let us sketch here a distributed
version of the Floyd-Warshall algorithm.
For each node v ∈ V let us consider the following data arrays:
Nv . Neighbors of v: u ∈ Nv if and only if {v, u} ∈ E.
In this case, let wvu ∈ R+ be the weight of the edge {v, u}.
Rv . Subset of nodes whose route from v has been decided.
Dv . Distances from v: Dv [[u]] will be the distance of the route from v to w.
Iv . Initial visited nodes from v: Iv [[u]] = w if and only if the route connecting v
with u shall begin going through node w.
The distributive algorithm to build arrays Dv and Iv , v ∈ V is sketched at pseu-
docode 4.11.

DistributedFloydWarshall
Input. A weighted graph G = (V, E) with non negative weights modeling a net-
work.
Output. The arrays Dv and Iv , v ∈ V .
1. For each v ∈ V Do (in parallel)
a. Rv := {v} ;
b. Dv [[v]] := 0 ; Iv [[v]] :=⊥ ;
c. For each u ∈ V − {v} Do (in parallel)
i. If u ∈ Nv Then { Iv [[u]] := u ; Dv [[u]] := wvu }
Else { Iv [[u]] :=⊥ ; Dv [[u]] := +∞ } ;
2. For each v ∈ V Do (in parallel)
a. While ¬(Rv == V ) do
i. Broadcast (Iv , Dv ) ;
ii. For each w ∈ V − Rv Do (in parallel)
A. Broadcast (Iw , Dw ) ;
B. For each u ∈ V − {v, w} Do (in parallel)
{ pnw := Dv [[w]] + Dw [[u]] ;
If pnw < Dv [[u]] Then
{ Iv [[u]] := Iv [[w]] ; Dv [[u]] := pnw }
};
iii. Rv := Rv ∪ {w} ;
3. Output {(Iv , Dv )| v ∈ V }.

Fig. 4.11 Procedure DistributedFloydWarshall.

Alternatively we may proceed according to the pseudocode 4.12.

94 4 Distributed Algorithms

ChandyMisra
Input. A weighted graph G = (V, E) with non negative weights modeling a net-
work.
Output. The arrays Dv and Iv , v ∈ V .
1. For each v ∈ V Do (in parallel)
a. Dv [[v]] := 0 ;
b. For each u ∈ V − {v} Do (in parallel)
i. If u ∈ Nv Then { Iv [[u]] := u ; Dv [[u]] := wvu }
Else { Iv [[u]] :=⊥ ; Dv [[u]] := +∞ } ;
2. For each v ∈ V Do (in parallel) For each u ∈ V − {v} Do (in parallel)
a. For each w ∈ V − {v, u} Do (in parallel)
{ pnw := Dv [[w]] + Dw [[u]] ;
If pnw < Dv [[u]] Then
{ Iv [[u]] := Iv [[w]] ; Dv [[u]] := pnw }
};
3. Output {(Iv , Dv )| v ∈ V }.

Fig. 4.12 Procedure ChandyMisra.

References 95

References

[Aho et al(1974)Aho, Hopcroft, and Ullman] Aho AV, Hopcroft JE, Ullman J (1974) The De-
sign and Analysis of Computer Algorithms. Addison-Wesley Longman Publishing Co., Inc.,
Boston, MA, USA, URL http://portal.acm.org/citation.cfm?id=578775
[Brassard and Bratley(1988)] Brassard G, Bratley P (1988) Algorithmics: theory & practice.
Prentice-Hall, Inc., Upper Saddle River, NJ, USA
[Brassard and Bratley(1990)] Brassard G, Bratley P (1990) Algor´ı́tmica: Concepción y Análisis.
Editorial Masson
[Cattaneo and Italiano(1999)] Cattaneo G, Italiano G (1999) Algorithm engineering. ACM Com-
put Surv p 3, DOI http://doi.acm.org/10.1145/333580.333582
[Cormen et al(2003)Cormen, Leiserson, Rivest, and Stein] Cormen TH, Leiserson CE, Rivest
RL, Stein C (2003) Introduction to Algorithms, 2nd edn. McGraw-Hill Science /
Engineering / Math, URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-
20&path=ASIN/0072970545
[Fernandez et al(2003)Fernandez, Garcia, and Garzon] Fernandez JJ, Garcia I, Garzon E (2003)
Educational issues on number representation and arithmetic in computers: an undergraduate
laboratory. IEEE Transactions on Education 46(4):477–485, DOI 10.1109/TE.2003.815237
[Garey and Johnson(1979)] Garey MR, Johnson DS (1979) Computers and Intractability : A
Guide to the Theory of NP-Completeness. Series of Books in the Mathematical Sciences, W.H.
Freeman & Company, URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-
20&path=ASIN/0716710455
[Gonnet and Baeza-Yates(1991)] Gonnet GH, Baeza-Yates R (1991) Handbook of algorithms and
data structures: in Pascal and C (2nd ed.). Addison-Wesley Longman Publishing Co., Inc.,
Boston, MA, USA
[Graham et al(1994)Graham, Knuth, and Patashnik] Graham RL, Knuth DE, Patashnik O (1994)
Concrete Mathematics: A Foundation for Computer Science (2nd Edition), 2nd edn. Addison-
Wesley Professional, URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-
20&path=ASIN/0201558025
[Hopcroft et al(2001)Hopcroft, Motwani, and Ullman] Hopcroft JE, Motwani R, Ullman JD
(2001) Introduction to automata theory, languages, and computation, 2nd edition. SIGACT
News 32(1):60–65, DOI http://doi.acm.org/10.1145/568438.568455
[Kfoury et al(1991)Kfoury, Arbib, and Moll] Kfoury AJ, Arbib MA, Moll RN (1991) Program-
ming Approach to Computability. Springer-Verlag New York, Inc., Secaucus, NJ, USA
[Knuth(1999)] Knuth DE (1999) Mathematics for the Analysis of Algorithms. Birkhauser Boston
[Levitin(2007)] Levitin AV (2007) Introduction to the Design and Analysis of Algorithms, 2-nd
Edition. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA
[Manber(1989)] Manber U (1989) Introduction to Algorithms: A Creative Approach. Addison-
Wesley Longman Publishing Co., Inc. Boston, MA, USA
[Michalewicz and Fogel(2004)] Michalewicz Z, Fogel DB (2004) How to Solve It: Mod-
ern Heuristics. Springer, URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-
20&path=ASIN/3540224947
[Papadimitriou and Steiglitz(1998)] Papadimitriou CH, Steiglitz K (1998) Combi-
natorial Optimization : Algorithms and Complexity. Dover Publications, URL
http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/0486402584
[Schnitger(2007)] Schnitger G (2007) Parallel and Distributed Algorithms. Institut f
ür Informatik, Johann Wolfgang Goethe-Universität, Frankfurt am Main, URL
http://www.thi.informatik.uni-frankfurt.de/Parallele/index.html
[Sedgewick and Flajolet(1996)] Sedgewick R, Flajolet P (1996) An introduction to the analysis
of algorithms. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA
[Stoer and Bulirsch(1980)] Stoer J, Bulirsch R (1980) Introduction to Numerical Analysis.
Springer-Verlag, New York and Berlin
[Ukkonen(1995)] Ukkonen E (1995) On-line construction of suffix trees. Algorithmica
14(3):249–260
96 4 Distributed Algorithms

[Wood(1993)] Wood D (1993) Data Structures, Algorithms, and Performance. Addison-Wesley

Publishing Co., Reading, MA, USA
Index

adjacency-list, 62 depth-first forest, 64

adjacency-matrix, 63 depth-first traversing, 30
algorithm, 1 diameter, 71
asynchronous, 76 dictionary, 1
differential, 48
backtracking procedure, 31 differential equation, 53
base of natural logarithms, 8 digraph, 62
binary search, 57 Dijkstra’s algorithm, 66
binary tree, 61 directed, 62
bisection width, 71 distributed system, 71, 76
breadth-first traversal, 63 diverges, 1
brute-force algorithm, 22 Divide and conquer, 15
domain, 1
Cauchy problem, 53 dual problem, 38
causal order, 77
characteristic, 10 empty word, 1
Chomsky hierarchy, 41 ending state, 2
class of functions bounded polylogarithmically, equivalence relation, 7
10 equivalent executions, 77
class of functions polynomially bounded, 8 events, 76
coefficients, 7 exponential function, 8
collision, 60
first order approximation, 49
collision-avoidance, 60
Floyd-Warshall algorithm, 66
communication protocols, 75
fractional knapsack, 25
complexity measures, 2
fractionary part map, 60
concurrent, 77
functional connotation, 1
connected component, 62
contraction, 50 Gauss-Seidel method, 52
convex hull, 28 generate-and-test algorithms, 22
cooling strategy, 35 genetic algorithms, 37
cubic polynomials, 7 gradient, 48
cycle, 62 Graham’s scan, 32
graph, 62
dag, 65 greedy algorithm, 23
decision problem, 21 growth order, 7
decreasing function, 7
degree, 7, 62 hash tables, 14

97
98 Index

hashing, 59 primal problem, 38

Hessian, 48 procedural connotation, 1
heuristic algorithm, 33 process, 76
hypercube, 72 productions, 40

in-degree, 62 quadratic polynomials, 7

increasing function, 7 Quicksort, 58
inorder labeling, 61
input, 1 root of map, 49
integer knapsack, 25 routing, 74
interpolation problem, 55 running space, 2
invariant requirement, 75 running time, 2
isolated vertex, 62
safety requirement, 75
Jacobi method, 52 search problem, 21
second order approximation, 49
Karatsouba’s multiplication method, 27 sequential search, 57
Knuth-Morris-Pratt algorithm, 43 several-variables polynomial, 7
sign, 2
Lagrange method, 56
Simulated annealing, 34
Lagrange multipliers, 38
size, 2
Lagrangian relaxation, 39
space complexity, 2
leading coefficient, 7
spanning forest, 68
length, 1, 2
spanning tree, 68
linear polynomials, 7
states, 76
liveness requirement, 75
Strassen method, 29
local algorithm, 76
suffix tree, 44
logarithm in base, 9
synchronous distributed system, 76
logical clock, 77
syntactical rules, 40
lookup table, 14

mantissa, 10 tabu list, 35

Mercator series, 10 tabu search, 35
metaheuristics, 34 Taylor’s Expansion Formula, 49
minimum spanning tree, 68 telescopic sum, 11
monotone function, 7 the growth order of g is strictly greater than
that of f , 7
nanosecond, 5 the growth order of g is strictly lower than that
natural logarithm, 9 of f , 7
network, 71 time complexity, 2
Newton-Raphson method, 51 time-space tradeoff, 14
non-decreasing function, 7 topological sort, 65
non-increasing function, 7 transition system, 75
transitive closure, 67
odd-even transposition sort, 72 traveling salesman problem, 31
one-variable polynomial, 7 tree, 30
ordered tree, 30 triangle operation, 66
out-degree, 62
output, 1 undirected, 62
universal hash family, 60
partial derivative, 48
partial derivative of order, 48 valency, 62
path, 62 Vandermonde method, 56
pivot, 59
power rules, 8 weighted graph, 62