Wavelets Made Easy – Some Linear Algebra

August 20, 2009

ii
Contents

1 Vector Spaces over R 1

1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Subspaces 7
2.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Lines through the origin as subspaces of R2 . . . . . . . . . . . 8
2.3 A subset of R2 that is not a subspace . . . . . . . . . . . . . . 9
2.4 Subspaces of R3 . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4.1 Planes containing the origin . . . . . . . . . . . . . . . 10
2.5 Summary of subspaces of R3 . . . . . . . . . . . . . . . . . . . 11
2.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 Vector Spaces of Functions 13

3.1 Space of Continuous Functions C(I) . . . . . . . . . . . . . . . 13
3.2 Space of continuously differentiable functions C 1 (I) . . . . . . 14
3.2.1 A continuous but not differentiable function . . . . . . 14
3.3 Space of r-times continuously differentiable functions C r (I) . . 15
3.3.1 C r (I) 6= C r−1 (I) . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Piecewise-continuous functions PC(I) . . . . . . . . . . . . . . 16
3.5 The indicator function χA . . . . . . . . . . . . . . . . . . . . 17
3.5.1 Some properties of χA . . . . . . . . . . . . . . . . . . 17
3.5.2 χA∪B with A, B disjoint . . . . . . . . . . . . . . . . 18
3.5.3 χA∪B with A, B arbitrary . . . . . . . . . . . . . . . . 18
3.6 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

iii
iv CONTENTS

4 Linear Maps 21
4.1 Two Important Examples . . . . . . . . . . . . . . . . . . . . 21
4.1.1 The Integral . . . . . . . . . . . . . . . . . . . . . . . . 21
4.1.2 The Derivative . . . . . . . . . . . . . . . . . . . . . . 22
4.2 Linear Maps from Rn to Rm . . . . . . . . . . . . . . . . . . . 22
4.2.1 Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.2.2 A Counterexample . . . . . . . . . . . . . . . . . . . . 24
4.2.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3 Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5 Inner Product 27

6 Generating Sets and Bases 35

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.2 In general . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.3 Possibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.3.1 Case C . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.4 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
6.5 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

7 Gram-Schmidt Orthogonalization 43
7.1 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
7.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
7.3 Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

8 Orthogonal Projections 47
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
8.2 w ∈ W is closest to v iff v − w ⊥ W . . . . . . . . . . . . . . 47
8.2.1 Construction of w . . . . . . . . . . . . . . . . . . . . . 49
8.3 The main theorem . . . . . . . . . . . . . . . . . . . . . . . . 50
8.4 Orthogonal projections . . . . . . . . . . . . . . . . . . . . . . 51
8.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
8.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
8.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

9 2-D Haar Wavelet Transform 55

9.1 Tensor Product of Functions . . . . . . . . . . . . . . . . . . . 55
9.2 The two-dim. Haar Wavelet Transform . . . . . . . . . . . . . 57
CONTENTS v

9.3 The Inverse Transform . . . . . . . . . . . . . . . . . . . . . . 59

9.3.1 Bigger Matrices . . . . . . . . . . . . . . . . . . . . . . 60

10 Complex Vector Spaces 63

10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
10.2 Complex numbers . . . . . . . . . . . . . . . . . . . . . . . . . 64
10.2.1 Addition and multiplication on C . . . . . . . . . . . . 64
10.2.2 Conjugate and absolute value of z . . . . . . . . . . . . 65
10.2.3 Reciprocal of z . . . . . . . . . . . . . . . . . . . . . . 65
10.2.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . 65
10.3 The complex exponential function . . . . . . . . . . . . . . . . 66
10.3.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . 66
10.4 Complex-valued functions . . . . . . . . . . . . . . . . . . . . 67
10.4.1 Integration and differentiation . . . . . . . . . . . . . . 67
10.5 Complex vector spaces . . . . . . . . . . . . . . . . . . . . . . 69
10.5.1 Complex Inner Product . . . . . . . . . . . . . . . . . 70
10.5.2 Norm in a complex vector space . . . . . . . . . . . . . 71

11 Discrete and Fast Fourier Transforms 75

11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
11.2 The Discrete Fourier Transform (DFT) . . . . . . . . . . . . . 75
11.2.1 Definition and Inversion . . . . . . . . . . . . . . . . . 75
11.2.2 The Fourier matrix . . . . . . . . . . . . . . . . . . . . 77
11.3 Discrete Fourier Transform . . . . . . . . . . . . . . . . . . . . 77
11.3.1 Two Results . . . . . . . . . . . . . . . . . . . . . . . . 77
11.4 Unitary Operators . . . . . . . . . . . . . . . . . . . . . . . . 79
11.5 The Fast Fourier Transform . . . . . . . . . . . . . . . . . . . 79
11.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 79
11.5.2 The Forward FFT . . . . . . . . . . . . . . . . . . . . 81
11.5.3 The Inverse FFT (IFFT) . . . . . . . . . . . . . . . . . 82
11.5.4 Interpolation by the IFFT . . . . . . . . . . . . . . . . 83
11.5.5 Bit Reversal . . . . . . . . . . . . . . . . . . . . . . . . 83
11.5.6 Applications of the FFT . . . . . . . . . . . . . . . . . 85
11.5.7 Multidimensional DFT and FFT . . . . . . . . . . . . 85
vi CONTENTS
Lecture 1

Vector Spaces over R

1.1 Definition
Definition 1. A vector space over R is a nonempty set V of objects,
called vectors, on which are defined two operations, called addition + and
multiplication by scalars · , satisfying the following properties:

A1 (Closure of addition)
For all u, v ∈ V, u + v is defined and u + v ∈ V .

A2 (Commutativity for addition)

u + v = v + u for all u, v ∈ V .

A3 (Associativity for addition)

u + (v + w) = (u + v) + w for all u, v, w ∈ V .

A4 (Existence of additive identity)

There exists an element ~0 such that u + ~0 = u for all u ∈ V .

A5 (Existence of additive inverse)

For each u ∈ V , there exists an element -denoted by −u- such that
u + (−u) = ~0.

M1 (Closure for scalar multiplication)

For each number r ∈ R and each u ∈ V , r · u is defined and r · u ∈ V .

M2 (Multiplication by 1)
1 · u = u for all u ∈ V .

1
2 LECTURE 1. VECTOR SPACES OVER R

M3 (Associativity for multiplication)

r · (s · u) = (r · s) · u for r, s ∈ R and all u ∈ V .
D1 (First distributive property)
r · (u + v) = r · u + r · v for all r ∈ R and all u, v ∈ V .
D2 (Second distributive property)
(r + s) · u = r · u + s · u for all r, s ∈ R and all u ∈ V .
Remark. The zero element ~0 is unique, i.e., if 0~1 , 0~2 ∈ V are such that
u + 0~1 = u + 0~2 = u, ∀u ∈ V
then 0~1 = 0~2 .
Proof. We have 0~1 = 0~1 + 0~2 = 0~2 + 0~1 = 0~2
Lemma. Let u ∈ V , then 0 · u = ~0.
Proof.
u+0·u = 1·u+0·u
= (1 + 0) · u
= 1·u
= u

Thus ~0 = u + (−u) = (0 · u + u) + (−u)

= 0 · u + (u + (−u))
= 0 · u + ~0
= 0·u
Lemma. a) The element −u is unique.
b) −u = (−1) · u.
Proof of part (b).
u + (−1) · u = 1 · u + (−1) · u
= (1 + (−1)) · u
= 0·u
= ~0
1.2. EXAMPLES 3

1.2 Examples
Before examining the axioms in more detail, let us discuss two examples.
Example.
 LetV = Rn ,considered as column vectors
x1
 x2 
Rn = { ..  |x1 , x2 , . . . , xn ∈ R} Then for
 
 . 
xn
   
x1 y1
u =  ...  , v =  ..  ∈ Rn and r ∈ R :
  
. 
xn yn

Define
   
x1 + y1 rx1
u+v =
 .. 
and r · u =  ... 
 
. 
xn + yn rxn

Note
 that
 the zero 
vector and
 the additive inverse of u are given by:
0 −x1
~0 =  .  −u =  ... 
 .. ,
 
0 −x2
Remark. Rn can also be considered as the space of all row vectors.

Rn = {(x1 , . . . , xn ) | x1 , . . . , xn ∈ R}

The addition and scalar multiplication is again given coordinate wise

(x1 , . . . , xn ) + (y1 , . . . , yn ) = (x1 + y1 , . . . , xn + yn )

r · (x1 , . . . , xn ) = (rx1 , . . . , rxn )

Example. If ~x = (2, 1, 3), ~y = (−1, 2, −2) and r = −4 find ~x + ~y and r · ~x.

4 LECTURE 1. VECTOR SPACES OVER R

Solution.

~x + ~y = (2, 1, 3) + (−1, 2, −2)

= (2 − 1, 1 + 2, 3 − 2)
= (1, 3, 1)

r · ~x = −4 · (2, 1, 3) = (−8, −4, −12).

Remark.

(x1 , . . . , xn ) + (0, . . . , 0) = (x1 + 0, . . . , xn + 0)

= (x1 , . . . , xn )

So the additive identity is ~0 = (0, . . . , 0).

Note also that

0 · (x1 , . . . , xn ) = (0x1 , . . . , 0xn )

= (0, . . . , 0)

for all (x1 , . . . , xn ) ∈ Rn .

Example. Let A be the interval [0, 1) and V be the space of functions
f : A −→ R, i.e.,
V = {f : [0, 1) −→ R}
Define addition and scalar multiplication by

(f + g)(x) = f (x) + g(x)

(r · f )(x) = rf (x)

For instance, the function f (x) = x4 is an element of V and so are

g(x) = x + 2x2 , h(x) = cos x, k(x) = ex

We have (f + g)(x) = x + 2x2 + x4 .

 1.2. EXAMPLES 5

Remark. (a) The zero element is the function ~0 which associates to each x
the number 0:
~0(x) = 0 for all x ∈ [0, 1)

Proof. (f + ~0)(x) = f (x) + ~0(x) = f (x) + 0 = f (x).

(b) The additive inverse is the function −f : x 7→ −f (x).

Proof. (f + (−f ))(x) = f (x) − f (x) = 0 for all x.

Example. Instead of A = [0, 1) we can take any set A 6= ∅, and we can replace
R by any vector space V . We set
V A = {f : A −→ V }
and set addition and scalar multiplication by
(f + g)(x) = f (x) + g(x)
(r · f )(x) = r · f (x)
Remark. (a) The zero element is the function which associates to each x
the vector ~0:

0 : x 7→ ~0

Proof
(f + 0)(x) = f (x) + 0(x)
= f (x) + ~0 = f (x)
Remark.
(b) Here we prove that + is associative:
Proof. Let f, g, h ∈ V A . Then
[(f + g) + h](x) = (f + g)(x) + h(x)
= (f (x) + g(x)) + h(x)
= f (x) + (g(x) + h(x)) associativity in V
= f (x) + (g + h)(x)
= [f + (g + h)](x)
6 LECTURE 1. VECTOR SPACES OVER R

1.3 Exercises
Let V = R4 . Evaluate the following:

a) (2, −1, 3, 1) + (3, −1, 1, −1).

b) (2, 1, 5, −1) − (3, 1, 2, −2).

c) 10 · (2, 0, −1, 1).

d) (1, −2, 3, 1) + 10 · (1, −1, 0, 1) − 3 · (0, 2, 1, −2).

e) x1 · (1, 0, 0, 0) + x2 · (0, 1, 0, 0) + x3 · (0, 0, 1, 0) + x4 · (0, 0, 0, 1).

Lecture 2

Subspaces

In most applications we will be working with a subset W of a vector space

V such that W itself is a vector space.

Question: Do we have to test all the axioms to find out if W is a vector

space?
The answer is NO.
Theorem. Let W 6= ∅ be a subset of a vector space V . Then W , with the
same addition and scalar multiplication as V , is a vector space if and only if
the following two conditions hold:

1. u + v ∈ W for all u, v ∈ W (or W + W ⊆ W )

2. r · u ∈ W for all r ∈ R and all u ∈ W (or RW ⊆ W ).

In this case we say that W is a subspace of V .

Proof. Assume that W + W ⊆ W and RW ⊆ W .

To show that W is a vector space we have to show that all the 10 axioms of
Definition 1.1 hold for W . But that follows because the axioms hold for V
and W is a subset of V :

A1 (Commutativity of addition)
For u, v ∈ W , we have u + v = v + u. This is because u, v are also in
V and commutativity holds in V .

7
8 LECTURE 2. SUBSPACES

A4 (Existence of additive identity)

Take any vector u ∈ W . Then by assumption 0 · u = ~0 ∈ W . Hence
~0 ∈ W .

A5 (Existence of additive inverse)

If u ∈ W then −u = (−1) · u ∈ W .

One can check that the other axioms follow in the same way.

2.1 Examples
Usually the situation is that we are given a vector space V and a subset of
vectors W satisfying some conditions and we need to see if W is a subspace
of V .

W = {v ∈ V : some conditions on v}

We will then have to show that


u, v ∈ W u+v
Satisfy the same conditions.
r∈R r·u

2.2 Lines through the origin as subspaces of

R2
Example.

V = R2 ,
W = {(x, y)|y = kx} for a given k
= line through (0, 0) with slope k.

To see that W is in fact a subspace of R2 :

Let u = (x1 , y1 ), v = (x2 , y2 ) ∈ W . Then y1 = kx1 and y2 = kx2
2.3. A SUBSET OF R2 THAT IS NOT A SUBSPACE 9

and

u + v = (x1 + x2 , y1 + y2 )
= (x1 + x2 , kx1 + kx2 )
= (x1 + x2 , k(x1 + x2 )) ∈ W

Similarly, r · u = (rx1 , ry1 ) = (rx1 , krx1 ) ∈ W

So what are the subspaces of R2 ?

1. {0}

2. Lines. But only those that contain (0, 0). Why?

3. R2

Remark (First test). If W is a subspace, then ~0 ∈ W .

Thus: If ~0 ∈
/ W , then W is not a subspace.

This is why a line not passing through (0, 0) can not be a subspace of R2 .

2.3 A subset of R2 that is not a subspace

Warning. We can not conclude from the fact that ~0 ∈ W , that W is a subspace.

Example. Lets consider the following subset of R2 :

W = {(x, y)|x2 − y 2 = 0}

Is W a subspace of R2 ? Why?

The answer is NO.

We have (1, 1) and (1, −1) ∈ W but (1, 1) + (1, −1) = (2, 0) ∈
/ W . i.e., W is
not closed under addition.

Notice that (0, 0) ∈ W and W is closed under multiplication by scalars.

10 LECTURE 2. SUBSPACES

2.4 Subspaces of R3
What are the subspaces of R3 ?
1. {0} and R3 .

2. Planes: A plane W ⊆ R3 is given by a normal vector (a, b, c) and its distance

from (0, 0, 0) or
W = {(x, y, z)| ax + by + cz = p }
| {z }
condition on (x,y,z)

For W to be a subspace, (0, 0, 0) must be in W by the first

test. Thus
p=a·0+b·0+c·0=0

or

p=0

2.4.1 Planes containing the origin

A plane containing (0, 0, 0) is indeed a subspace of R3 .

Proof. Let (x1 , y1 , z1 ) and (x2 , y2 , z2 ) ∈ W . Then

ax1 + by1 + cz1 = 0

ax2 + by2 + cz2 = 0

Then we have

a(x1 + x2 ) + b(y1 + y2 ) + c(z1 + z2 ) = (ax1 + by1 + cz1 ) + (ax2 + by2 + cz2 )

| {z } | {z }
0 0
= 0

and

a(rx1 ) + b(ry1 ) + c(rz1 ) = r(ax1 + by1 + cz1 )

= 0
2.5. SUMMARY OF SUBSPACES OF R3 11

2.5 Summary of subspaces of R3

1. {0} and R3 .

2. Planes containing (0, 0, 0).

3. Lines containing (0, 0, 0).

(Intersection of two planes containing (0, 0, 0))

2.6 Exercises
Determine whether the given subset of Rn is a subspace or not (Explain):

a) W = {(x, y) ∈ R2 | xy = 0}.

b) W = {(x, y, z) ∈ R3 | 3x + 2y 2 + z = 0}.

c) W = {(x, y, z) ∈ R3 | 2x + 3y − z = 0}.

d) The set of all vectors (x1 , x2 , x3 ) satisfying

2x3 = x1 − 10x2 .

e) The set of all vectors in R4 satisfying the system of linear equations

2x1 + 3x2 + 5x4 = 0

x1 + x2 − 3x3 = 0

f) The set of all points (x1 , x2 , x3 , x4 ) ∈ R4 satisfying

x1 + 2x2 + 3x3 + x4 = −1.

12 LECTURE 2. SUBSPACES
Lecture 3

Vector Spaces of Functions

3.1 Space of Continuous Functions C(I)

Let I ⊆ R be an interval. Then I is of the form (for some a < b)

 {x ∈ R| a < x < b}, an open interval;

{x ∈ R| a ≤ x ≤ b}, a closed interval;

I=

 {x ∈ R| a ≤ x < b}
{x ∈ R| a < x ≤ b}.


Recall that the space of all functions f : I −→ R is a vector space. We will

now list some important subspaces:
Example (1). Let C(I) be the space of all continuous functions on I. If f and g
are continuous, so are the functions f + g and rf (r ∈ R). Hence C(I) is a vector
space.
Recall, that a function is continuous, if the graph has no gaps. This can be
formulated in different ways:
a) Let x0 ∈ I and let  > 0 . Then there exists a δ > 0 such that for all
x ∈ I ∩ (x0 − δ, x0 + δ) we have
|f (x) − f (x0 )| < 
This tells us that the value of f at nearby points is arbitrarily close to the
value of f at x0 .

b) A reformulation of (a) is:

lim f (x) = f (x0 )
x→x0

13
14 LECTURE 3. VECTOR SPACES OF FUNCTIONS

3.2 Space of continuously differentiable func-

tions C 1(I)
Example (2). The space C 1 (I). Here we assume that I is open. Recall that f is
differentiable at x0 if
f (x) − f (x0 ) f (x0 + h) − f (x0 )
lim = lim =: f 0 (x0 )
x→x0 x − x0 h→0 h

exists. If f 0 (x0 ) exists for all x0 ∈ I, then we say that f is differentiable on I. In

this case we get a new function on I

x 7→ f 0 (x)

We say that f is continuously differentiable on I if f 0 exists and is continuous

on I.
Recall that if f and g are differentiable, then so are
f + g and rf (r ∈ R)
moreover
(f + g)0 = f 0 + g 0 ; (rf )0 = rf 0
As f 0 + g 0 and rf 0 are continuous by Example (1), it follows that C 1 (I) is a
vector space.

3.2.1 A continuous but not differentiable function

Let f (x) = |x| for x ∈ R. Then f is continuous on R but it is not differentiable on
R. We will show that f is not differentiable at x0 = 0. For h > 0 we have

f (x0 + h) − f (x0 ) |h| − 0 h

= = =1
h h h
hence
f (x0 + h) − f (x0 )
lim =1
h→0+ h
But if h < 0, then

f (x0 + h) − f (x0 ) |h| − 0 −h

= = = −1
h h h
hence
f (x0 + h) − f (x0 )
lim = −1
h→0− h
3.3. SPACE OF R-TIMES CONTINUOUSLY DIFFERENTIABLE FUNCTIONS C R (I)15

Therefore,
f (x0 + h) − f (x0 )
lim
h→0 h
does not exist.

3.3 Space of r-times continuously differentiable

functions C r (I)
Example (3). The space C r (I)
Let I = (a, b) be an open interval. and let r ∈ N = {1, 2, 3, · · · }.

Definition. The function f : I −→ R is said to be r-times continuously differentiable

if all the derivatives f 0 , f 00 , · · · , f (r) exist and f (r) : I −→ R is continuous.
We denote by C r (I) the space of r-times continuously differentiable functions
on I. C r (I) is a subspace of C(I).

We have
C r (I) $ C r−1 (I) ( · · · $ C 1 (I) $ C(I).

3.3.1 C r (I) 6= C r−1 (I)

We have seen that C 1 (I) 6= C(I). Let us try to find a function that is in C 1 (I) but
not in C 2 (I).
5
Assume 0 ∈ I and let f (x) = x 3 . Then f is differentiable and

2
f 0 (x) = 35 x 3

which is continuous.

If x 6= 0, then f 0 is differentiable and

10 − 31
f 00 (x) = 3 x
16 LECTURE 3. VECTOR SPACES OF FUNCTIONS

But for x = 0 we have

2
f 0 (h) − 0 5 h3 5 1
lim = lim = lim h− 3
h→0 h h→0 3 h h→0 3

which does not exist.

Remark. One can show that the function

3r−1
f (x) = x 3

is in C r−1 (R), but not in C r (R).

Thus, as stated before, we have

C r (I) $ C r−1 (I) · · · $ C 1 (I) $ C(I).

3.4 Piecewise-continuous functions PC(I)

Example (4). Piecewise-continuous functions on I

Definition. Let I = [a, b). A function f : I −→ R is called piecewise-continuous if

there exists finitely many points

a = x0 < x1 < · · · < xn = b

such that f is continuous on each of the sub-intervals

(xi , xi+1 ) for i = 0, 1, · · · , n − 1.

Remark. If f and g are both piecewise-continuous, then

f + g and rf (r ∈ R)

are piecewise-continuous.

Hence the space of piecewise-continuous functions is a vector space. We denote

this vector space by PC(I).
3.5. THE INDICATOR FUNCTION χA 17

3.5 The indicator function χA

Important elements of PC(I) are the indicator functions χA , where A ⊆ I is a
sub-interval.
Let A ⊆ R be a set. Define

1, if x ∈ A
χA (x) =
0, if x ∈
/A

So the values of χA tell us whether x is in A or not.

If x ∈ A, then χA (x) = 1 and if x ∈
/ A, then χA (x) = 0.

We will work a lot with indicator functions so let us look at some of their
properties.

3.5.1 Some properties of χA

Lemma. Let A, B ⊆ I. Then

χA∩B (x) = χA (x) χB (x) (∗)

Proof. We have to show that the two functions

x 7→ χA∩B (x) and x 7→ χA (x)χB (x)

take the same values at every point x ∈ I. So lets evaluate both functions:
If x ∈ A and x ∈ B, that is x ∈ A ∩ B, then
χA∩B (x) = 1 and,
since χA (x) = 1 and χB (x) = 1, we also have χA (x)χB (x) = 1.
Thus, the left and the right hand sides of (∗) agree.
On the other hand, if x ∈/ A ∩ B, then there are two possibilities:

• x∈
/ A, then χA (x) = 0, so χA (x)χB (x) = 0.

• x∈
/ B, then χB (x) = 0, so χA (x)χB (x) = 0.

It follows that
0 = χA∩B (x) = χA (x)χB (x)
18 LECTURE 3. VECTOR SPACES OF FUNCTIONS

3.5.2 χA∪B with A, B disjoint

What about χA∪B ? Can we express it in terms of χA and χB ?
If A and B are disjoint, that is A ∩ B = ∅ then

χA∪B (x) = χA (x) + χB (x).

Let us prove this:

• If x ∈
/ A ∪ B, then x ∈
/ A and x ∈
/ B. Thus the LHS (left hand side) and the
RHS (right hand side) are both zero.

• If x ∈ A ∪ B then either

x is in A but not in B. In this case

χA∪B (x) = 1 and χA (x) + χB (x) = 1 + 0 = 1
or
x is in B but not in A. In this case
χA∪B (x) = 1 and χA (x) + χB (x) = 0 + 1 = 1

3.5.3 χA∪B with A, B arbitrary

Thus we have,

If A ∩ B = ∅, then χA∪B (x) = χA (x) + χB (x).

Now, what if A ∩ B 6= ∅?
Lemma. χA∪B (x) = χA (x) + χB (x) − χA∩B (x).
Proof.

• If x ∈
/ A ∪ B, then both of the LHS and the RHS take the value 0.

• If x ∈ A ∪ B, then we have the following possibilities:

1. If x ∈ A, x ∈/ B, then
χA∪B (x) = 1
χA (x) + χB (x) − χA∩B = 1 + 0 − 0 = 1
2. Similarly for the case x ∈ B, x ∈
/ A: LHS equals the RHS.
3.6. EXERCISE 19

3. If x ∈ A ∩ B, then
χA∪B (x) = 1
χA (x) + χB (x) − χA∩B = 1 + 1 − 1 = 1

As we have checked all possibilities, we have shown that the statement in the
lemma is correct

3.6 Exercise
Write the product

(2χ[0,3] − 3χ[3,6] )(χ[0,2] + 5χ[2,4] − 7χ[4,6] )

as a linear combination of the indicator functions of intervals.

20 LECTURE 3. VECTOR SPACES OF FUNCTIONS
Lecture 4

Linear Maps

We have all seen linear maps before. In fact, most of the maps we have been
using in Calculus are linear.

4.1 Two Important Examples

4.1.1 The Integral
To integrate the function f (x) = x2 + 3x − cos x over the interval [a, b], we
first find the antiderivative of x2 , that is 13 x3 , then the antiderivative of x,
which is 21 x2 , and then multiply that by 3 to get 32 x2 . Finally, we find the
antiderivative of cos x, which is sin x, and then multiply that by −1 to get
− sin x. To finish the problem we insert the endpoints. Thus,

Z 1 Z 1 Z 1
2 2
x + 3x − cos x dx = x dx + 3 x dx
−1 −1 −1
Z 1
− cos x dx
−1
 1  1
1 3 3
= x + x2 − [sin x]1−1
3 −1 2 −1
2
= − sin 1 + sin(−1).
3

What we have used is the fact that the integral is a linear map C([a, b]) −→ R
and that

21
22 LECTURE 4. LINEAR MAPS

Z b Z b Z b
rf (x) + sg(x) dx = r f (x) dx + s g(x) dx.
a a a

4.1.2 The Derivative

Another example is differentiation Df = f 0 . To differentiate the function f (x) =
x4 − 3x + ex − cos x, we first differentiate each term of the function and then add:

D(x4 − 3x + ex − cos x) = Dx4 − 3Dx + Dex

−D cos x
= 4x3 − 3 + ex + sin x.

Definition. Let V and W be two vector spaces. A map T : V −→ W is said to

be linear if for all v, u ∈ V and all r, s ∈ R we have:

T (rv + su) = rT (v) + sT (u).

Remark: This can also be written by using two equations:

T (v + u) = T (v) + T (u)

T (rv) = rT (v).
Lemma. Suppose that T : V −→ W is linear. Then T (~0) = ~0.

Proof. We can write ~0 = 0v, where v is any vector in V . But then T (~0) = T (0v) =
0T (v) = 0

4.2 Linear Maps from Rn to Rm

Example. Let us find all the linear maps from R2 −→ R2 . Any arbitrary vector
(x1 , x2 ) ∈ R2 can be written as:

(x1 , x2 ) = x1 (1, 0) + x2 (0, 1).

Hence,
T (x1 , x2 ) = x1 T (1, 0) + x2 T (0, 1).
Write T (1, 0) and T (0, 1) as:

T (1, 0) = (a11 , a12 ), T (0, 1) = (a21 , a22 ), where aij ∈ R.

4.2. LINEAR MAPS FROM RN TO RM 23

T (x1 , x2 ) = x1 (a11 , a12 ) + x2 (a21 , a22 )

= (x1 a11 + x2 a21 , x1 a12 + x2 a22 )
 
a11 a12
= (x1 , x2 ) .
a21 a22

Thus,
 all the information
 about T is given by the matrix:
a11 a12
.
a21 a22
Example. Next, let us find all the linear maps T : R3 −→ R3 . As before we write
(x1 , x2 , x3 ) ∈ R3 as:

(x1 , x2 , x3 ) = x1 (1, 0, 0) + x2 (0, 1, 0) + x3 (0, 0, 1)

where,

T (1, 0, 0) = (a11 , a12 , a13 )

T (0, 1, 0) = (a21 , a22 , a23 )
T (0, 0, 1) = (a31 , a32 , a33 ).

Then,

T (x1 , x2 , x3 ) = x1 (a11 , a12 , a13 ) + x2 (a21 , a22 , a23 )

+ x3 (a31 , a32 , a33 )
= (x1 a11 + x2 a21 + x3 a31 , x1 a12 + x2 a22
+ x3 a32 , x1 a13 + x2 a23 + x3 a33 )
 
a11 a12 a13
= (x1 , x2 , x3 )  a21 a22 a23  .
a31 a32 a33

Example. All the linear maps from R3 −→ R. Notice that R is also a vector space,
so we can consider all the linear maps Rn to R. We have :

T (x1 , x2 , ..., xn ) = x1 T (1, 0, ..., 0) + x2 T (0, 1, ..., 0)

+... + xn T (0, 0, ..., 1)
= x1 a1 + x2 a2 + ... + xn an

where,
T (1, 0, ..., 0) = a1 , T (0, 1, ..., 0) = a2 , . . . T (0, 0, ..., 1) = an .
24 LECTURE 4. LINEAR MAPS

4.2.1 Lemma
Lemma. A map Rn −→ Rm is linear if and only if there exists numbers aij , i =
1, ..., n, j = 1, ..., m, such that:

T (x1 , x2 , ..., xn ) = (x1 a11 + x2 a21 + ... + xn an1 , ...,

x1 a1m + x2 a2m + ... + xn anm )

This can also be written as:

n
X n
X n
X
T (x1 , x2 , ..., xn ) = ( xi ai1 , xi ai2 , xi aim )
i=1 i=1 i=1

or by using matrix multiplication:

 
a11 a12 . . . a1m

 a21 a22 . . . a2m 

T (x1 , x2 , ..., xn ) = (x1 , x2 , ..., xn ) 
 . . . . . . .

 . . . . . . 
an1 an2 . . . anm

4.2.2 A Counterexample
Example. The map T (x, y, z) = (2x + 3xy, z + y) is not linear because of the factor
xy. Notice that:
T (1, 1, 0) = (5, 0)
but
T (2(1, 1, 0)) = T (2, 2, 0) = (16, 0)
and
2T (1, 1, 0) = (10, 0) 6= (16, 0)

4.2.3 Examples
Example. Evaluate the given following maps at a given point:

T (x, y) = (3x + y, 3y), (x, y) = (1, −1)

T (1, −1) = (3 · 1 − 1, 3(−1)) = (2, −3)

T (x, y, z) = (2x − y + 3z, 2x + z), (x, z, y) = (2, −1, 1)

T (2, −1, 1) = (4 + 1 + 3, 4 + 1) = (8, 5)
4.3. KERNEL 25

Example. Some examples involving differentiation and integration:

D(3x2 + 4x − 1) = 6x + 4

Z 2  2
2 x 1 3 x
x − e dx = x −e
1 3 1
8 1
= − e2 − + e
3 3
7 2
= −e +e
3

4.3 Kernel
Definition. Let V and W be two vector spaces and T : V −→ W a linear map.
• A1 The set Ker(T ) = {v ∈ V : T (v) = 0} is called the kernel of T .

• A2 The set Im(T ) = {w ∈ W : there exists av ∈ V : T (v) = w} is called

the image of T .
Remark: Notice that Ker(T ) ⊆ V and Im(T ) ⊆ W .
Theorem. The kernel of T is a vector space.

Proof. Let u, v ∈ Ker(T ) and r, s ∈ R We have to show that ru + sv ∈ Ker(T ).

Now,u, v ∈ Ker(T ) if and only if T (u) = T (v) = 0. Hence,

T (ru + sv) = rT (u) + sT (v) (T is linear)

= r·0+s·0 (u, v ∈ Ker(T ))
= 0

This shows that ru + sv ∈ Ker(T ).

Remark: Let R2 −→ R2 be the map:

T (x, y) = (x2 + y, x + y).

Then,
T (1, −1) = (1 − 1, 1 − 1) = (0, 0).
But T (2(1, −1)) = T (2, −2) = (4 − 2, 2 − 2) = (2, 0) 6= (0, 0).
So if T is not linear, then the set v ∈ V : T (v) = 0 is in general not a vector
space.
26 LECTURE 4. LINEAR MAPS

Example. Let R2 −→ R be the map: T (x, y) = 2x − y. Describe the kernel of T .

We know that (x, y) is in the kernel of T if and only if T (x, y) = 2x − y = 0.
Hence, y = 2x. Thus, the kernel of T is a line through (0, 0) with slope 2.
Example. Let R3 −→ R2 be the map: T (x, y, z) = (2x−3y+z, x+2y−z). Describe
the kernel of T .
We have that (x, y, z) ∈ Ker(T ) if and only if

2x − 3y + z = 0 and x + 2y − z = 0.

The equations describe planes through (0, 0, 0) with normal vectors (2, −3, 1) and
(1, 2, −1) respectively. The normal vectors are not parallel and therefore the planes
are different. It follows that the intersection is a line.
Let us describe this line. Adding the equations we get:

3x − y = 0 or y = 3x.

Plugging this into the second equation we get:

0 = x + 2(3x) − z = 7x − z or z = 7x.

Hence, the line is given by: x · (1, 3, 7).

Theorem. Let V and W be vector spaces, and T : V −→ W linear. Then, Im(T ) j
W is a vector space.

Proof. Let w1 , w2 ∈ Im(T ). Then we can find u1 , u2 ∈ V such that T (u1 ) =

w1 , T (u2 ) = w2 . Let r, s ∈ R. Then,

rw1 + sw2 = rT (u1 ) + sT (u2 )

= T (ru1 + su2 ) ∈ Im(T )).
Lecture 5

Inner Product

Let us start with the following problem. Given a point P ∈ R2 and a line L j R2 ,
how can we find the point on the line closest to P ?
Answer: Draw a line segment from P meeting the line in a right angle. Then,
the point of intersection is the point on the line closest to P .
Let us now take a plane L j R3 and a point outside the plane. How can we
find the point u ∈ L closest to P ?
The answer is the same as before, go from P so that you meet the plane in a
right angle.

Observation
In each of the above examples we needed two things:

A1 We have to be able to say what the length of a vector is.

B1 Say what a right angle is.

Both of these things can be done by using the dot-product (or inner product) in
Rn .

Definition. Let (x1 , x2 , ..., xn ), (y1 , x2 , ..., yn ) ∈ Rn . Then, the dot-product of

these vectors is given by the number:

((x1 , x2 , ..., xn ), (y1 , x2 , ..., yn )) = x1 y1 + x2 y2 + ... + xn yn .

27
28 LECTURE 5. INNER PRODUCT

The norm (or length) of the vector ~u = (x1 , x2 , ..., xn ) ∈ Rn is the non-negative
number: p q
kuk = (u, u) = x21 + x22 + ... + x2n .

Examples

Example. (a) ((1, 2, −3), (1, 1, 1)) = 1 + 2 − 3 = 0

(b) ((1, −2, 1), (2, −1, 3)) = 2 + 2 + 3 = 7

Perpendicular
Because,
q q
|x1 y1 + x2 y2 + ... + xn yn | ≤ x21 + x22 + ... + x2n y12 + y22 + ... + yn2

or
|(u, v)| ≤ kuk · kvk
we have that (for u, v 6= 0)

(u, v)
−1 ≤ ≤ 1.
kuk · kvk
Hence we can define:
(u, v)
cos(∠(u, v)) = .
kuk · kvk
In particular, u⊥v (u is perpendicular to v) if and only if (u, v) = 0.

Questions
Example. Let L be the line in R2 given by y = 2x. Thus,

L = {r(1, 2) : r ∈ R} .

Let P = (2, 1). Consider the following questions.

Question 1: What is the point on L closest to P ?
Answer: Because u ∈ L, we can write ~u = (r, 2r). Furthermore, v − u =
(2 − r, 1 − 2r) is perpendicular to L. Hence,

0 = ((1, 2), (2 − r, 1 − 2r)) = 2 − r + 2 − 4r = 4 − 5r.

 29

4
Hence, r = 5 and ~v = ( 54 , 85 ). Question 2: What is the distance of P from the line?
Answer: The length of the vector v − u, i.e. kv − uk. First we have to find out
what v − u is. We have done almost all the work:
4 8 6 −3
v − u = (2, 1) − ( , ) = ( , ).
5 5 5 5
The distance therefore is: r √
36 9 3 5
+ = .
25 25 5

Properties of the Inner Product

1. (positivity)To be able to define the norm, we used that (u, u) ≥ 0.
2. (zero length)All non-zero vectors should have a non-zero length. Thus,
(u, u) = 0 only if u = 0.
3. (linearity)If the vector v ∈ Rn is fixed, then a map u 7→ (u, v) from Rn to
R is linear. That is,
(ru + sw, v) = r(u, v) + s(w, v).

4. (symmetry) For all u, v ∈ Rn we have: (u, v) = (v, u).

We will use the properties above to define an inner product on arbitrary vector
spaces.

Definition
Let V be a vector space. An inner product on V is a map (., .) : V × V −→ R
satisfying the following properties:
1. (positivity) (u, u) ≥ 0, for all v ∈ V .
2. (zero length) (u, u) = 0 only if u = 0.
3. (linearity)If v ∈ V is fixed, then a map u 7→ (u, v) from V to R is linear.
4. (symmetry) (u, v) = (v, u), for all u, v ∈ V .
Definition. We say that u and v are perpendicular if (u, v) = 0.
Definition. If (.,.) is an inner product on the vector space V , then the norm of
a vector v ∈ V is given by:
p
kuk = (u, u).
30 LECTURE 5. INNER PRODUCT

Properties of the Norm

Lemma. The norm satisfies the following properties:

1. kuk ≥ 0, and kuk = 0 only if u = 0.

2. kruk = |r| · kuk.

Proof. We have that

p
kruk = (ru, rv)
p
= (r2 (u, v)
p
= |r| (u, v) = |r| · kuk .

Examples
Example. Let a < b, I = [a, b], and V = P C([a, b]). Define:
Z b
(f, g) = f (t)g(t) dt
a

Then, (.,.) is an inner product on V .

Proof. Let r, s ∈ R, f, g, h ∈ V . Then:

Rb Rb
1. (f, f ) = a f (t)2 dt. As f (t)2 ≥ 0, it follows that a f (t)2 dt ≥ 0.

2. If (f, f ) = 0, then f (t)2 = 0 for all t, i.e f = 0.

Rb Rb
3. a (rf sg)(t)h(t) dt = a rf (t)h(t) + sg(t)h(t) dt
Z b Z b
= r f (t)h(t) dt + s g(t)h(t) dt
a a
= r(f, h) + s(g, h).

Hence, linear in the first factor.

4. As f (t)g(t) = g(t)f (t), it follows that (f, g) = (g, f ).

Notice that the norm is: s
Z b
kf k = f (t)2 dt.
a
 31

Example. Let a = 0, b = 1 in the previous example. That is, f (t) = t2 and

g(t) = t − 3t2 . Then:
Z 1
(f, g) = t2 (t − 3t2 ) dt
0
Z 1
= t3 − 3t4 ) dt
0
1 3
= −
4 5
7
= − .
20
Also, the norms are: s
Z 1
1
kf k = t4 dt = √ .
0 5
s
Z 1
kgk = (t − 3t2 )2 dt
0
s
Z 1
= t2 − 6t3 + 9t4 dt
0
r
1 3 9
= − +
3 2 5
r
19
= .
30
Example. Let f (t) = cos 2πt and g(t) = sin 2πt. Then:
Z 1
(f, g) = cos 2πt sin 2πt dt
0
1  1
= (sin 2πt)2 0 = 0.
4π
So, cos 2πt is perpendicular to sin 2πt on the interval [0, 1].
Example. Let f (t) = χ[0,1/2) − χ[1/2,1) and g(t) = χ[0,1) . Then:
Z 1
(f, g) = (χ[0,1/2) (t) − χ[1/2,1) (t))(χ[0,1) ) dt
0
Z 1 Z 1
= χ[0,1/2) (t) dt − χ[1/2,1) (t) dt
0 0
Z 1/2 Z 1
1 1
= dt − dt = = = 0.
0 1/2 2 2
32 LECTURE 5. INNER PRODUCT

One can also easily show that kf k = kgk = 1.

Problem
Problem: Find a polynomial f (t) = a + bt that is perpendicular to the polynomial
g(t) = 1 − t.

Answer: We are looking for numbers a and b such that:

Z 1
0 = (f, g) = (a + bt)(1 − t) dt
0
Z 1
= a + bt − at − bt2 dt
0
b a b
= a+ − −
2 2 3
a b
= + .
2 6

Thus, 3a + b = 0. So, we can take f (t) = 1 − 3t.

Important Facts
We state now two important facts about the inner product on a vector space V .
Recall that in R2 we have:
(u, v)
cos(θ) = .
kuk · kvk

where u, v are two non-zero vectors in R2 and θ is the angle between u and v. In
particular, because −1 ≤ cos θ ≤ 1, we must have:

k(u, v)k ≤ kuk · kvk.

We will show now that this comes from the positivity and linearity of the inner
product.

Theorem. Let V be a vector space with inner product (.,.). Then:

|(u, v)| ≤ kuk · kvk

for all u, v ∈ V .
 33

Proof. We can assume that u, v 6= 0 because otherwise both the LHS and the RHS
will be zero. By the positivity of the inner product we get:

(v, u) (v, u)
0 ≤ (v − 2 u, v − u) (positivity)
kuk kuk2
(v, u) (v, u) (v, u)2
= (v, v) − 2 (u, v) − 2 (v, u) + (u, u) (linearity)
kuk kuk kuk4
(u, v)2 (v, u)2
= kvk2 − 2 + (symmetry)
kuk2 kuk2
(u, v)2
= kvk2 − .
kuk2

Thus,
(u, v)2 2
2 ≤ kvk or k(u, v)k ≤ kuk · kvk.
kuk

Note
Notice that:
(u, v) (u, v)
0 = (v − 2 u, v − u)
kuk kuk2
only if
(u, v)
v− u=0
kuk2
i.e.
(u, v)
v= u.
kuk2
Thus, v and u have to be on the same line through 0.

A Lemma
We can therefore conclude:

Lemma. k(u, v)k = kuk · kvk if and only if u and v are on the same line through 0.
34 LECTURE 5. INNER PRODUCT

Theorem
The following statement is generalization of Pythagoras Theorem.

Theorem. Let V be a vector space with inner product (.,.). Then:

ku + vk ≤ kuk · kvk

for all u, v ∈ V . Furthermore, ku + vk2 = kuk2 + kvk2 if and only if (u, v) = 0.

Proof.

ku + vk2 = (u + v, u + v)
= (u, u) + 2(u, v) + (v, v) (∗)
2 2
≤ kuk + 2 kuk · kvk + kvk
= (kuk + kvk)2 .

If (u, v) = 0, then (*) reads:

ku + vk2 = kuk2 + kvk2

On the other hand, if ku + vk2 = kuk2 + kvk2 , we see from (*) that (u, v) = 0.

Example. Let u = (1, 2, −1), v = (0, 2, 4). Then:

(u, v) = 4 − 4 = 0

and
kuk2 = 1 + 4 + 1 = 6, kvk2 = 4 + 16 = 20.
Also, u + v = (1, 4, 3) and finally:

ku + vk2 = 1 + 16 + 9 = 26 = 6 + 20 = kuk2 + kvk2 .

Lecture 6

Generating Sets and Bases

Let V be the vector space R2 and consider the vectors (1, 0), (0, 1). Then, every
vector (x, y) ∈ R2 can be written as a combination of those vectors. That is:

(x, y) = x(1, 0) + y(0, 1).

Similarly, the two vectors (1, 1) and (1, 2) do not belong to the same line, and
every vector in R2 can be written as a combination of those two vectors.

6.1 Introduction
In particular:
(x, y) = a(1, 1) + (1, 2)
gives us two equations

a + b = x and a + 2b = y

Thus, by substituting the first equation to the second, we get

b = −x + y

Inserting this into the first equation we get

a = 2x − y

Take for example the point (4, 3). Then:

(4, 3) = 5(1, 1) + (−1)(1, 2)

= 5(1, 1) − (1, 2)

35
36 LECTURE 6. GENERATING SETS AND BASES

We have similar situation for R3 and all of the spaces Rn .

In the case of R3 , for example, every vector can be written as combinations of

(1, 0, 0), (0, 1, 0) and (0, 0, 1), i.e.,

(x, y, z) = x(1, 0, 0) + y(0, 1, 0) + z(0, 0, 1).

Or, as a combination of (1, −1, 0), (1, 1, 1) and (0, 1, −1), that is:

(x, y, z) = a(1, −1, 0) + b(1, 1, 1) + c(0, 1, −1).

The latter gives three equation:

a + b = x (1)
−a + b + c = y (2)
b−c = z (3).

(2) + (3) gives:

−a + 2b = y + z (4)

(4) + (1) gives:

x+y+z
3b = x + y + zorb = .
3
Then (1) gives:

a = x−b
x+y+z
= x−
3
2x − y − z
= .
3

Finally, (3) gives:

x + y − 2z
c=b−z =
3
Hence, we get:

2x − y − z x+y+z x + y − 2z
(x, y, z) = (1, −1, 0) + (1, 1, 1) + (0, 1, −1).
3 3 3
6.2. IN GENERAL 37

6.2 In general
Notice that we get only one solution, so there is only one way that we can write a
vector in R3 as a combination of those vectors. In general, if we have k vectors in
R3 , then the equation:
x = (x1 , x2 , ..., xn ) = c1 v1 + c2 v2 + ... + ck vk (∗)
gives n-equations involving the n-coordinates of v1 , v2 , ..., vk and the unknowns
c1 , c2 , ..., ck . There are three possibilities:

6.3 Possibilities
A The equation(*) has no solution. Thus, x can not be written as a combina-
tion of the vectors v1 , v2 , ..., vk .
B The equation (*) has only one solution, so x can be written in exactly one
way as a combination of v1 , v2 , ..., vk .
C The system of equations has infinitely many solutions, so there are more than one
way to write x as a combination of v1 , v2 , ..., vk .

6.3.1 Case C
Let us look at the last case a little closer. If we write x in two different ways:
x = c1 v1 + c2 v2 + ... + ck vk
x = d1 v1 + d2 v2 + ... + dk vk
Then, by subtracting, we get:
0 = (c1 − d1 )v1 + (c2 − d2 )v2 + ... + (ck − dk )vk
where some of the numbers ci − di are non-zero.
Similarly, since we can write:
0 = a1 v1 + a2 v2 + ... + ak vk
and
x = c1 v1 + c2 v2 + ... + ck vk
then we also have:
x = (c1 + a1 )v1 + (c2 + a2 )v2 + ... + (ck + ak )vk .
Thus, we can write x as a combination of the vectors v1 , v2 , ..., vk in several different
ways (in fact ∞-many ways).
38 LECTURE 6. GENERATING SETS AND BASES

6.4 Definitions
We will now use this as a motivation for the following definitions.

Definition. Let V be a vector space and v1 , v2 , ..., vn ∈ V .

1. Let W ⊆ V be a subspace. We say that W is spanned by the vectors

v1 , v2 , ..., vn if every vector in W can be written as a linear combination of
v1 , v2 , ..., vn . Thus, if w ∈ W , then there exist numbers c1 , c2 , ..., cn ∈ R such
that w = c1 v1 + c2 v2 + ... + cn vn .

2. The set of vectors v1 , v2 , ..., vn is linearly dependent if there exist c1 , c2 , ..., cn ,

not all equal to zero, such that c1 v1 + c2 v2 + ... + cn vn = 0.

Definition. 1. The set of vectors v1 , v2 , ..., vn is linearly independent if the set

is not linearly dependent(if and only if we can only write c1 v1 + c2 v2 + ... +
cn vn = 0 with all ci = 0).

2. The set of vectors v1 , v2 , ..., vn is a basis for W , if v1 , v2 , ..., vn is linearly

independent and spans W .

Before we show some examples, let us make the following observations:

Lemma. Let V be a vector space with an inner product (., .). Assume that
6 j).
v1 , v2 , ..., vn is an orthogonal subset of vectors in V (thus (vi , vj ) = 0 if i =
(v,vi )
If v = c1 v1 + c2 v2 + ... + cn vn , then ci = kv k2 , i = 1, ..., n.
i

Proof. Assume that v = c1 v1 + c2 v2 + ... + cn vn . Take the inner product with v1

in both sides of the equation. The LHS is (v, v1 ). The RHS is:

(c1 v1 + c2 v2 + ... + cn vn , v1 ) = c1 (v1 , v1 ) + c2 (v2 , v1 )

+... + cn (vn , v1 )
= c1 (v1 , v1 )
= c1 kv1 k2 .

(v,v1 )
Thus, (v, v1 ) = c1 kv1 k2 , or c1 = kv1 k2
. Repeat this for v2 , ..., vn .

Corollary. If the vectors v1 , v2 , ..., vn are orthogonal, then they are linearly inde-
pendent.
6.5. EXAMPLES 39

6.5 Examples
Example. Let V = R2 . The vectors (1, 2) and (−2, −4) are linearly dependent
because:

(−2)(1, 2) + 1(−2, −4) = 0.

The vectors (1, 2), (1, 1) are linearly independent. In fact, (1, 2), (1, 1) is a basis
for R2 .
Indeed, let (x, y) ∈ R2 . Then,

(x, y) = c1 (1, 2) + c2 (1, 1)

= (c1 + c2 , 2c1 + c2 ).

Thus,

x = c1 + c2
y = 2c1 + c2 .

Subtracting we get: x − y = −c1 , or c1 = y − x. Plugging this into the first

equation we get:
c2 = x − c1 = x − (y − x) = 2x − y.
Thus, we can write any vector in R2 as a combination of those two. In particular,
for (0, 0) we get c1 = c2 = 0. The vectors (1, 2), (−2, 1) are orthogonal and hence
linearly independent, and in fact a basis. Hence,

(x, y) = c1 (1, 2) + c2 (−2, 1).

x+2y x+2y −2x+y
Taking the inner product we get: c1 = kv1 k2
= 5 and c2 = 5 .

Example. Let V = R3 . One vector can only generate a line, two vectors can at
most span a plane, so we need at least three vectors to span R3 . The vectors
(1, 2, 1), (1, −1, 1) are orthogonal but not a basis. In fact, those two vectors span
the plane:
W = (x, y, z) ∈ R3 : x − z = 0
(explain why).

On the other hand, the vectors: (1, 2, 1), (1, −1, 1) and (1, 0, −1) are orthogonal,
and hence a basis.
40 LECTURE 6. GENERATING SETS AND BASES

We have, for example:

(4, 3, 1) = c1 (1, 2, 1) + c2 (1, −1, 1) + c3 (1, 0, −1)

with
4+6+1
c1 =
1+4+1
4−3+1
c2 =
3
4−1
c3 =
2
In general, we have:
x + 2y + z x−y+z x−z
(x, y, z) = (1, 2, 1) + (1, −1, 1) + (1, 0, −1)
6 3 2
Let us now discuss some spaces of functions:

a) Let v0 (x) = 1, v1 (x) = x and v2 (x) = x2 . Then, vo , v1 and v2 are linearly

independent.

0 = c0 v0 (x) + c1 v1 (x) + c2 v2 (x) for all x

= c0 + c1 x + c2 x2

Take x = 0, then we get c0 = 0

Differentiate both sides to get:

0 = c1 + 2c2 x

Take again x = 0 to find c1 = 0. Differentiate one more time to get that c2 = 0.

Notice that the span of v0 , v1 , v2 is in the space of polynomials of degree ≤ 2.
Hence, the functions 1, x, x2 form a basis for this space. Notice that the functions
1 + x, 1 − 2x, x2 are also a basis.
b) Are the functions v0 (x) = x, v1 (x) = xex linearly independent/dependent
on R? Answer: No.

Assume that 0 = c0 x + c1 xex . It does not help to put x = 0 now, but let us
first differentiate both sides and get:

0 = c0 + c1 ex + c1 xex

Now, x = 0 gives:
0 = c0 + c1 (1)
6.5. EXAMPLES 41

Differentiating again, we get: 0 = c1 ex + c1 ex + c1 xex . Now, x = 0 gives 0 = 2c1 ,

or c1 = 0. Hence, (1) gives c1 = 0.
c) The functions χ[0, 1 ) , χ[ 1 ,1) are orthogonal and hence linearly independent.
2 2
Let us show this directly. Assume that

0 = c1 χ[0, 1 ) + c2 χ[ 1 ,1) .
2 2

An equation like this means that every x we put into the function on the RHS,
the result is always 0.

1
Let us take x = 4. Then: χ[0, 1 ) ( 14 ) = 1, but χ[ 1 ,1) ( 41 ) = 0. Hence, 0 =
2 2
c1 · 1 + c2 · 0, or c1 = 0. Taking x = 34 shows that c2 = 0.
d) The functions χ[0, 1 ) , χ[0,1) are not orthogonal, but linearly independent.
2

0 = c1 χ[0,1) + c2 χ[0, 1 ) .
2

Take x so that χ[0, 1 ) (x) = 0 but χ[0,1) (x) = 1. Thus, any x ∈ [0, 1) \ [0, 21 ) = [ 12 , 1)
2
will do the job. So, take x = 43 . Then, we see that:

0 = c1 · 1 + c2 · 0, orc1 = 0.
1
Then take x = 4 to see that c2 = 0.
42 LECTURE 6. GENERATING SETS AND BASES
Lecture 7

Gram-Schmidt
Orthogonalization

The ”best” basis we can have for a vector space is an orthogonal basis. That is
because we can most easily find the coefficients that are needed to express a vector
as a linear combination of the basis vectors v1 , ..., vn :
(v, v1 ) (v, vn )
v= 2 v1 + ... + vn .
kv1 k kvn k2
But usually we are not given an orthogonal basis. In this section we will show how
to find an orthogonal basis starting from an arbitrary basis.

7.1 Procedure
Let us start with two linear independent vectors v1 and v2 (i.e. not on the same line
through zero). Let u1 = v1 . How can we find a vector u2 which is perpendicular
to u1 and that the span of u1 and u2 is the same as the span of v1 and v2 ? We try
to find a number a ∈ R such that:
u2 = au1 + v2 , u 2 ⊥ u1
Take the inner product with u1 to get:
0 = (u2 , u1 ) = a(u1 , u1 ) + (v2 , u1 )
= a ku1 k2 + (v2 , u1 )
or
(v2 , u1 )
a=−
ku1 k2

43
44 LECTURE 7. GRAM-SCHMIDT ORTHOGONALIZATION

What if we have a third vector v3 ? Then, after choosing u1 , u2 as above, we

would look for u3 of the form:
u3 = a1 u1 + a2 u2 + v3
Take the inner product with u1 to find a1 :
0 = (u3 , u1 ) = a1 ku1 k2 + (v3 , u1 )
or
(v3 , u1 )
a1 = −
ku1 k2
and the inner product with u2 to find a2 :
0 = (u3 , u2 ) = a2 ku2 k2 + (v3 , u2 )
or
(v3 , u2 )
a2 = −
ku2 k2
Thus:
u1 = v1
(v2 , u1 )
u2 = v2 − u1
ku1 k2
(v3 , u1 ) (v3 , u2 )
u3 = v3 − 2 u1 − u2
ku1 k ku2 k2

7.2 Examples
Example. Let v1 = (1, 1), v2 = (2, −1). Then, we set u1 = (1, 1) and
(v2 , u1 )
u2 = (2, −1) − u1
ku1 k2
2−1
= (2, −1) − (1, 1)
2
3
= (1, −1)
2
Example. Let v1 = (2, −1), v2 = (0, 1). Then, we set u1 = (2, −1) and
(v2 , u1 )
u2 = (0, 1) − u1
ku1 k2
−1
= (0, 1) − (2, −1)
5
2
= (1, 2)
5
7.2. EXAMPLES 45

Note We could have also started with v2 = (0, 1), and get first basis vector to
be (0, 1) and second vector to be:
(2, −1) · (0, 1)
(2, −1) − (0, 1) = (2, 0)
k(0, 1)k2
Example. Let v1 = (0, 1, 2), v2 = (1, 1, 2), v3 = (1, 0, 1). Then, we set u1 = (0, 1, 2)
and
(0, 1, 2) · (1, 1, 2)
u2 = (1, 1, 2) − (0, 1, 2)
k(0, 1, 2)k2
5
= (1, 1, 2) − (0, 1, 2)
2
= (1, 0, 0)

2
u3 = (1, 0, 1) − (0, 1, 2) − (1, 0, 0)
5
1
= (0, −2, 1)
5
Example. Let v0 = 1, v1 = x, v2 = x2 . Then, v0 , v1 , v2 is a basis for the space of
polynomials of degree ≤. But they are not orthogonal, so we start with u0 = v0
and u1 = v1 − (vku1 ,uk02) u0 . So we need to find:
0
Z 1
1 1
(v1 , u0 ) = x dx = [ x2 ]10 =
0 2 2
Z 1
ku0 k2 = 1 dx = [x]10 = 1
0

Hence, u1 = x − 12 . Then:
(v2 , u0 ) (v2 , u1 )
u1 = v2 − 2 u0 − u1 .
ku0 k ku1 k2
We also find that:
Z 1
1
(v2 , u0 ) = x2 dx =
0 3
Z 1
1 1
(v2 , u1 ) = x2 (x − ) dx =
0 2 12
Z 1
1 1
ku1 k2 = (x − )2 dx = .
0 2 12
1
Hence, u2 = x2 − 3 − (x − 12 ) = x2 − x + 16 .
46 LECTURE 7. GRAM-SCHMIDT ORTHOGONALIZATION

7.3 Theorem
Theorem. (Gram-Schmidt Orthogonalization)Let V be a vector space with inner
product (., .). Let v1 , ..., vk be a linearly independent set in V . Then, there
exists an orthogonal set u1 , ..., uk such that (vi , ui ) > 0 and span{vi , ..., vi } =
span{u1 , ..., ui } for all i = 1, ..., k.

Proof. See the book, p.129 − 131.

Lecture 8

Orthogonal Projections

8.1 Introduction
We will now come back to our original aim: Given a vector space V , a
subspace W , and a vector v ∈ V , find the vector w ∈ W which is closest to
v.
First let us clarify what the ”closest” means. The tool to measure distance
is the norm, so we want kv − wk to be as small as possible.
Thus our problem is:
Find a vector w ∈ W such that

kv − wk ≤ kv − uk

for all u ∈ W .
Now let us recall that if W = Rw1 is a line, then the vector w on the line
W is the one with the property that v − w ⊥ W .
We will start by showing that this is always the case.

8.2 w ∈ W is closest to v iff v − w ⊥ W

Theorem. Let V be a vector space with inner product (·, ·). Let W ⊂ V be
a subspace and v ∈ V . If v − w ⊥ W , then kv − wk ≤ kv − uk for all u ∈ W
and kv − wk = kv − uk if and only if w = u. Thus w is the member of W
closest to v.

47
48 LECTURE 8. ORTHOGONAL PROJECTIONS

Proof. First we remark that kv − wk ≤ kv − uk if and only if kv − wk2 ≤

kv − uk2 . Now we simply calculate

kv − uk2 = k(v − w) + (w − u)k2

= kv − wk2 + kw − uk2
because v − w ⊥ W and w − u ∈ W
(∗) ≥ kv − wk2 because kw − uk2 ≥ 0

So kv − uk ≥ kv − wk. If kv − uk2 = kv − wk2 , then we see - using (∗)- that

kw − uk2 = 0, or w = u.
As kv − wk = kv − uk if u = w, we have shown that the statement is
correct.

Theorem. Let V be a vector space with inner product (·, ·). Let W ⊂ V be
a subspace and v ∈ V . If w ∈ W is the closest to v, then v − w ⊥ W .

Proof. We know that kv − wk2 ≤ kv − uk2 for all u ∈ W . Therefore the

function f : R −→ R

F (t) := kv − w + txk2 (x ∈ W )

has a minimum at t = 0. We have

F (t) = (v − w + tx, v − w + tx)

= (v − w, v − w) + t(v − w, x)
+ t(x, v − w) + t2 (x, x)
= kv − wk2 + 2t(v − w, x) + t2 kxk2

Therefore

0 = F 0 (0) = 2(v − w, x).

As x ∈ W was arbitrary, it follows that v − w ⊥ W .

8.2. W ∈ W IS CLOSEST TO V IFF V − W ⊥ W 49

8.2.1 Construction of w
Our task now is to construct the vector w such that v − w ⊥ W . The idea
is to use Gram-Schmidt orthogonalization.
Let W = Ru and v ∈ V . Applying Gram-Schmidt to u and v gives:

(v, u)
v− u⊥W
kuk2

(v,u)
So that w = kuk2
u is the vector (point) on the line W closest to v.
What if the dimension of W is greater than one? Let v1 , . . . , vn be an or-
thogonal basis for W . Applying the Gram-Schmidt to the vectors v1 , . . . , vn , v
shows that
n
X (v, vj )
v− v
2 j
j=1
kv j k

is orthogonal to each one of the vectors v1 , . . . , vn . Since

n
X (v, vj )
v− vj
j=1
kvj k2

is orthogonal to vj forPall j, it is orthogonal to any linear combination of them

c1 v1 + . . . + cn vn = nj=1 cj vj , and hence it is orthogonal to W . Therefore
our vector w closest to v is given by

n
X (v, vj )
w= vj .
j=1
kvj k2

Let us look at another motivation; Let w ∈ W be the closest to v and let

v1 , . . . , vn be a basis for W . Then there are scalars c1 , . . . , cn ∈ R such that

n
X
w= ck vk .
k=1

So what are these scalars? As v − w ⊥ vj for j = 1, . . . , n and vk ⊥ vj for

50 LECTURE 8. ORTHOGONAL PROJECTIONS

k 6= j we get:

0 = (v − w, vj )
= (v, vj ) − (w, vj )
Xn
= (v, vj ) − ck (vk , vj )
k=1
= (v, vj ) − cj (vj , vj )
= (v, vj ) − cj kvj k2 .

Solving for cj we get

(v, vj )
cj = .
kvj k2
Thus n
X (v, vj )
w= vj .
j=1
kvj k2

8.3 The main theorem

We collect the results of the above computations in the following (main)theorem:
Theorem. Let V be a vector space with inner product (·, ·). Let W ⊂ V be
a subspace and assume that {v1 , . . . , vn } is an orthogonal basis for W . For
(v,v )
v ∈ V let w = nj=1 kvj kj2 vj ∈ W . Then v − w ⊥ W (or equivalently, w is
P

the vector in W closest to v).

Proof. We have

(v − w, vj ) = (v, vj ) − (w, vj )
n
X (v, vk )
= (v, vj ) − (vk , vj )
k=1
kvk k2
(v, vj )
= (v, vj ) − kvj k2
kvj k2
= (v, vj ) − (v, vj )
= 0
Hence v − w ⊥ vj . But, as we saw before, this implies that v − w ⊥ W
because v1 , . . . , vn is a basis.
8.4. ORTHOGONAL PROJECTIONS 51

8.4 Orthogonal projections

Let us now look at what we just did from the point of view of linear maps.
What is given in the beginning is a vector space with an inner product and
a subspace W . Then for each v ∈ V we associated a unique vector w ∈ W .
Thus we got a map
P : V −→ W, v 7→ w
We even have an explicit formula for P (v): Let (if possible) v1 , . . . , vn be an
orthogonal basis for W , then
n
X (v, vk )
P (v) = vk
k=1
kvk k2

This shows that P is linear.

We showed earlier that if v ∈ W , then
n
X (v, vk )
v= vk
k=1
kvk k2

So P (v) = v for all v ∈ W . In particular, we get

Lemma. P 2 = P .

The map P is called the orthogonal projection onto W . The projection

part comes from P 2 = P and orthogonal from the fact that v − P (v) ⊥ W .

8.5 Summary
The result of this discussion is the following:
To find the vector w closest to v we have to:

1. Find (if possible) a basis u1 , . . . , un for W .

2. If this is not an orthogonal basis, then use Gram-Schmidt to construct

an orthogonal basis v1 , . . . , vn .
Pn (v,vk )
3. Then w = k=1 kvk k2 vk .
52 LECTURE 8. ORTHOGONAL PROJECTIONS

8.6 Examples
Example. Let W be the line W = R(1, 2). Then u = (1, 2) is a basis (orthog-
onal!) for W . It follows that the orthogonal projection is given by

x + 2y
P (x, y) = (1, 2).
5
Let (x, y) = (3, 1). Then
P (3, 1) = (1, 2).

Example. Let W be the line given by y = 3x. Then (1, 3) ∈ W and hence
W = R(1, 3).It follows that

x + 3y
P (x, y) = (1, 3).
10
Example. Let W be the plane generated by the vectors (1, 1, 1) and (1, 0, 1).
Find the orthogonal projection P : R3 −→ W .
Solution. We notice first that ((1, 1, 1), (1, 0, 1)) = 2 6= 0, so this is not an
orthogonal basis. Using Gram-Schmidt we get:
v1 = (1, 1, 1)
v2 = (1, 0, 1) − 23 (1, 1, 1) = ( 31 , − 23 ), 31 = 13 (1, −2, 1).
To avoid fractions, we can use (1, −2, 1) instead of 13 (1, −2, 1). Thus the
orthogonal projection is:
x+y+z x − 2y + z
P (x, y, z) = (1, 1, 1) + (1, −2, 1)
 3 6
2x + 2y + 2z x − 2y + z
= + ,
6 6
2x + 2y + 2z x − 2y + z
−2 ,
6 6 
2x + 2y + 2z x − 2y + z
+
6 6
 
x+z x+z
= , y, .
2 2

Example. Let W be the plane {(x, y, z) ∈ R3 |x + y + 2z = 0}. Find the

orthogonal projection P : R3 −→ W .
8.7. EXERCISES 53

Solution. We notice that our first step is to find an orthogonal basis for W .
The vectors (1, −1, 0) and (2, 0, −1) are in W , but are not orthogonal.We
have
(2, 0, −1) − 22 (1, −1, 0) = (1, 1, −1) ∈ W
and orthogonal to (1, −1, 0). So we get:
x−y x+y−z
P (x, y, z) = (1, −1, 0) + (1, 1, −1)
2 3 
5x − y − 2z −x + 5y − 2z −x − y + z
= , , .
6 6 3

8.7 Exercises
1. Let V ⊂ R2 be the line V = R(1, −1).
(a) Write a formula for the orthogonal projection P : R2 → V.
(b) What is: i) P (1, 1), ii) P (2, 1), iii) P (2, −2)?
2. Let W ⊂ R3 be the plane
W = {(x, y, z) ∈ R3 : x − 2y + z = 0}.
(a) Find the orthogonal projection P : R3 → W.
(b) What is: i) P (1, 1, 2), ii) P (1, −2, 1), iii) P (2, 1, 1)?
3. Let W ⊂ R3 be the plane generated by the vectors (1, 1, 1) and (1, −1, 1).
(a) Find the orthogonal projection P : R3 → W.
(b) What is: i) P (1, 1, 2), ii) P (2, 0, 1)?
4. Let W be the space of continuous functions on [0, 1] generated by the
constant function 1 and x. Thus W = {a0 + a1 x : a0 , a1 ∈ R}. Find
the orthogonal projection of the following functions onto W :
i) P (x2 ), ii) P (ex ), iii) P (1 + x2 ).
5. Let W be the space of piecewise continuous functions on [0, 1] gener-
ated by χ[0,1/2) and χ[1/2,1) . Find orthogonal projections of the following
functions onto W :
i) P (x), ii) P (x2 ), iii) P (χ[0,3/4] ).
54 LECTURE 8. ORTHOGONAL PROJECTIONS
Lecture 9

2-D Haar Wavelet Transform

9.1 Tensor Product of Functions

In 1-dimension we used step functions of the form:
n −1
2X
p= sj χ [ j j+1
, )
2n 2n
j=1

to approximate ”arbitrary” functions.

Thus we made the following steps:
(1) Decide up on the resolution, and divide the interval [0, 1) into subinter-
vals Ij = [ 2jn , j+1
2n
).
(2) Take the orthogonal projection
n −1
2X
f →p= sj χ [ j j+1
, )
2n 2n
j=1

(3) Replace p by the vector [s0 , ..., s2n −1 )

(4) Apply the fast wavelet transform to the sequence [s0 , ..., s2n −1 ).
In 2-D we have now two directions, x and y. We will therefore have to
divide the square:
I = [0, 1) × [0, 1) = {(x, y) ∈ R2 : 0 ≤ x < 1, 0 ≤ y < 1}
into smaller squares:
i i+1 j j+1
Iijn = {(x, y) ∈ R2 : ≤ x < , ≤ y < }
2n 2n 2n 2n

55
56 LECTURE 9. 2-D HAAR WAVELET TRANSFORM

Then we approximate functions of two variables by functions that are con-

stants on each of the squares Iij . That is,
X
f →p= sj χIij

We start by discussing special functions of two variables.

Definition. Let p, q : [0, 1) −→ R. Then, p⊗q is the function on [0, 1)×[0, 1)

given by:
p ⊗ q(x, y) = p(x)q(y).

Example. Let A, B ⊆ [0, 1). Then, χA ⊗ χB = χA×B .

PROOF: We have χA ⊗ χB (x, y) = χA (x)χB (y). Thus, χA ⊗ χB (x, y) = 1

if and only if x ∈ A and y ∈ B. But that is the same as saying that
(x, y) ∈ A × B.
Example. Let ϕij = χ[ j j+1
, ) and
2n 2n

ψij = χ[ j
, j+1 ) − χ[ j+2 , j+3 )
2i+1 2i+1 2i+1 2i+1

Recall that ϕij is used to express the averages, and ψij is used to express the
details.
We have:
ϕni ⊗ ϕnj = ϕIijn .
Can be used to express the ”average values” on Iijn and

ϕni ⊗ ψnj , ψni ⊗ ϕnj , ψni ⊗ ψnj .

can be used to express details going from one column to the next, one row
to the next, and going diagonally.
Let f be a step function in [0, 1) × [0, 1). Instead of associating to f an
array, we have to use a matrix. Thus if:
n −1
2X
f (x, y) = sij χIijn
i,j=1
9.2. THE TWO-DIM. HAAR WAVELET TRANSFORM 57

then, f corresponds to the matrix:

 
s00 s01 . . . s0,2n −1
 s10 s11 . . . s1,2n −1 
 
f ↔  . . . . . . 

 . . . . . . 
s2n −1,0 s2n −1,1 . . . s2n −1,2n −1

Notice that the first indices (the ones corresponding to the x-variable)
indicate the row, and the second indices correspond to the column.
Example. 2 × 2-matrix
 
1 1
χ[0, 1 ) ⊗ χ[0,1) ↔
2 0 0

Example. Find the 2 × 2-matrix corresponding to the function:

χ[0, 1 ) ⊗ (χ[0, 1 ) − χ[ 1 ,1) ) + 2χ[ 1 ,1) ⊗ χ[0,1)

2 2 2 2

The matrix is:  

1 −1
2 2

9.2 The two-dim. Haar Wavelet Transform

In two dimensions we apply the Haar wavelet transform on the columns and
on the rows.
Example. Suppose that the function f is represented by the 2 × 2 matrix:
 
4 2
0 6

Then each row represents a one-dimensional vector of length 2. Namely, [4, 2]

and [0, 6].
Applying the 1 − D wavelet transform to these vectors gives:

[4, 2] → [3, 1]

and
[0, 6] → [3, −3]
58 LECTURE 9. 2-D HAAR WAVELET TRANSFORM

Thus,    
4 2 3 1
−→
0 6 3 −3
We now apply the 1 − D wavelet transform to the rows:

[3, 3]T → [3, 0]T

and
[1, −3]T → [−1, 2]T
This results to the new matrix,
   
3 1 3 1
−→
3 −3 0 −3

Notice that the element 3 represents the average value of all values:
4+2+0+6 12
= = 3.
4 4
To understand the other matrix elements, we will do this for a general matrix.
So, we have:
   s00 +s01 s00 −s01 
s00 s01 1st −step
−→ 2
s10 +s01
2
s10 −s01
s10 s11 2 2

We apply the 1-D transform to each column and we get:

 1  s00 +s01 s10 +s11  1  s00 −s01 s10 −s11  
2st −step + 2  2 2 + 2 
−→ 2 2
1 s00 −s01
1 s00 +s01
2 2
− s10 +s
2
11
2 2
− s10 −s
2
11

Remarks: 1. The number 12 s00 +s s10 +s11

= s00 +s01 +s 10 +s11
 
2
01
+ 2 4
is the av-
erage value.
2. The number 12 s00 −s + s10 −s
 
2
01
2
11
is the average change moving from the
first to the second column.
3. The number 12 s00 +s − s10 +s

2
01
2
11
is the average change moving from the
first to the second row (or the details  for the average).
4. The number 21 s00 −s2
01
− s10 −s11
2
represents the average changes along the
diagonals.
9.3. THE INVERSE TRANSFORM 59

The method for bigger for matrices is the same:

   
5 7 −1 −5 6 −3 −1 2
 1 −1 −3 −3  1−D on rows  0 −3 1 0 
  −→  
 7 3 2 2   5 2 2 0 
2 1 0 0 1 0 2 0
 
3 −3 0 1
 3 1
1−D on columns 
−→
2 0 
 3 0 −1 1 
2 1 0 0


3 −3
Note Now notice that the 2 × 2 matrix located in the upper-
3 1
half corner, consists of average values. We apply the 2-D wavelet transform
again to this matrix and get:
     
3 −3 0 3 1 2
−→ −→
3 1 2 1 −1 1

9.3 The Inverse Transform

Given a 2×2 matrix (or a 4×4 matrix) knowing that this matrix is a result of
a 2 − D Haar wavelet transform, how can we reconstruct the original matrix?
Let us start with a 2 × 2 matrix. Recall that the 2-D Haar wavelet
transform consists of 2-steps:

1st -step: (⇒) 1-D Haar Wavelet Transform on the rows.

2nd -step: () 1-D Haar Wavelet Transform on the columns.

Therefore to invert this we need to:

1st -step: Apply the inverse 1-D Haar Wavelet Transform on the columns.
2st -step: Apply the inverse 1-D
 Haar Wavelet
 Transform on the rows.
2 1
Example. Given the matrix , find the initial matrix.
−1 0
     
2 1 1−D inverse on columns 2−1 1+0 1 1
−→ =
−1 0 2 − (−1) 1 − 0 3 1
60 LECTURE 9. 2-D HAAR WAVELET TRANSFORM

   
1−D inverse on rows 1+1 1−1 2 0
−→ =
3+1 3−1 4 2

9.3.1 Bigger Matrices

Now look at 4 × 4 matrices. Recall that the last step was to apply the 2-D
wavelet transform on the upper left corner (2×2 matrix). Here is an overview
of the steps:

1st -step:1-D on the rows.

2nd -step:1-D on the columns.
3rd -step:Take the 2 × 2 matrix in the left upper corner and apply step 1 and
2 to this matrix, keeping all other numbers the same.
For the inverse we need to invert each of the previous steps. Thus:

1st -step: Pick the 2 × 2 matrix in the left upper corner. Apply the inverse
1-D on the columns.
2nd -step: Apply the 1-D inverse transform on the rows on the 2 × 2 matrix
from step 1.
3rd -step: We have now a new 4 × 4 matrix. Apply the inverse 1-D to the
columns.
4th -step: Apply the inverse 1-D to the rows.
 
2 1 0 3
 1 0 2 −1 
Example. Given the matrix   1 2 −1 2 , find the initial matrix.


0 −1 1 3
 
2 1
We start by the 2 × 2 matrix and we apply the inverse transform to this
1 0
matrix:      
2 1 columns 3 1 rows 4 2
−→ −→ .
1 0 3 1 2 0
We have now the new matrix:
 
4 2 0 3
 2 0
 2 −1 

 1 2 −1 2 
0 −1 1 3
9.3. THE INVERSE TRANSFORM 61

Example. To this new matrix, we will apply the inverse row-column trans-
form:    
4 2 0 3 5 4 −1 5
 2 0
 2 −1   inverse −→
on columns 
 3 0 1 1 
 1 2 −1 2   2 −1 3 2 
0 −1 1 3 2 1 1 −4
 
4 6 9 −1
inverse on rows 
 4 2 1 −1 
−→  5 −1 1

−3 
3 1 −3 5
 
4 6 9 −1
 4 2 1 −1 
So the final answer is:   5 −1 1 −3 


3 1 −3 5
62 LECTURE 9. 2-D HAAR WAVELET TRANSFORM
Lecture 10

Complex Vector Spaces

10.1 Introduction
Up to now we have only considered vector spaces over the field of real num-
bers. For the Fourier transform we will need vector spaces where we can
multiply vectors by complex numbers. If x ∈ R, then we know that x2 ≥ 0
and x2 = 0 only if x = 0. Therefore there is no real number such that
x2 = −1, or: There is no real solution to the equation

x2 + 1 = 0.

More generally let us look at the equation

x2 + bx + c = 0

By completing the square we get

b 1√ 2
x=− ± b − 4c
2 2
There are now 3 possibilities:
1. b2 − 4c > 0. Then the quadratic equation above gives two solutions
b 1√ 2
x=− + b − 4c
2 2
and
b 1√ 2
x=− − b − 4c
2 2

63
64 LECTURE 10. COMPLEX VECTOR SPACES

2. b2 − 4c = 0. Then we have one solution

b
x=−
2

3. b2 − 4c < 0. Then there is no real solution to quadratic equation

x2 + bx + c = 0.

10.2 Complex numbers

√
We now introduce a new number i = −1 such that

i2 = −1

The complex numbers are all expressions of the form

z = x + iy , x, y ∈ R.

The set of complex numbers is denoted by C. We say that x = <z is the real
part of z and y = =z is the imaginary part of z.
Recall that the set of real numbers can be thought of as a line, the real
line. To picture the set of complex numbers we use the plane. A vector
~v = (x, y) corresponds to the complex number z = x + iy.

10.2.1 Addition and multiplication on C

The addition of two complex numbers z = x + iy and w = s + it then
corresponds to the addition of the corresponding vectors. Thus

(x + iy) + (s + it) = (x + s) + i(y + t).

To find out what the product of z and w is, we use the familiar rules
along with i2 = −1. Thus

(x + iy) · (s + it) = xs + xit + iys + iyit

= xs + i2 yt + i(xt + ys)
= (xs − yt) + i(xt + ys).
10.2. COMPLEX NUMBERS 65

10.2.2 Conjugate and absolute value of z

Before we find the inverse -or reciprocal- of z = x + iy, we need to introduce
the complex conjugate:
x + iy = x − iy
Thus, complex conjugate corresponds to a reflection around the x-axis.
Now multiply z by z:

z · z = (x + iy)(x − iy)
= x2 − (iy)2
= x2 + y 2 .

√
Thus zz := |z| is the length of the vector (x, y) or the absolute value of
the complex number z.

10.2.3 Reciprocal of z
Lemma. Let z = x + iy be a complex number with |z| =
6 0. Then

1 z x − iy
= 2 = 2 .
z |z| x + y2

(x+iy)(x−iy) x2 +y 2
Proof. We have z xx−iy
2 +y 2 = x2 +y 2
= x2 +y 2
=1

10.2.4 Examples

• (2 + 3i) + (5 − 2i) = 7 + i.

• (2 − 3i)(1 + i) = (2 + 3) + (−3 + 2)i = 5 − i.

1 2−i 2
• 2+i
= 5
= 5
− 15 i.

2+3i (2+3i)(1−5i) (2+15)+(3−10)i 17−7i

• 1+5i
= 26
= 26
= 26
.
66 LECTURE 10. COMPLEX VECTOR SPACES

10.3 The complex exponential function

The idea behind the Fourier transform is to represent a function (or a signal)
in the frequency domain using the complex exponential function.
Definition. Let {zn }n∈N be a sequence of complex numbers. We say that
zn converges to the complex number w if for all  > 0 there exists a N ∈ N
such that
|zn − w| < 
for all n ≥ N . We write zn → w or limn→∞ zn = w if zn converges to w.
Pn zk
Let z ∈ C and define zn = k=0 k! . Then it can be shown that the
sequence {zn } converges. We denote the limit by
∞ N
z
X zk X zk
e = = lim .
k=0
k! N →∞
k=0
k!

It can be shown that

ez+w = ez ew .
In particular
1
= e−z .
ez

Theorem. (The Euler formula)

Let z = x + iy ∈ C. Then
ez = ex (cos y + i sin y).

10.3.1 Examples

• eπi = cos π + i sin π = −1.

π
• ei 2 = cos π2 + i sin π2 = i.
π e2
• e2+i 4 = e2 (cos π4 + i sin π4 ) = √
2
(1 + i).

Lemma. Let z = x + iy. Then

ez = ez
10.4. COMPLEX-VALUED FUNCTIONS 67

Proof.
ez = ex (cos y + i sin y)
= ex (cos y − i sin y)
= ex−iy
= ez

10.4 Complex-valued functions

Let F : I −→ C, I ⊆ R an interval, be a function. Then we can write
F (t) = f (t) + ig(t)
where f, g : I −→ R.
The function F is continuous if and only if f and g are both continuous.
In that case we have
lim F (t) = lim f (t) + ( lim g(t))i
t→t0 t→t0 t→t0
= f (t0 ) + g(t0 )i
= F (t0 ).

10.4.1 Integration and differentiation

We integrate and differentiate F by integrating f (t) and g(t) (differentiating
f (t) and g(t)). Thus
Z b Z b Z b
F (t) dt = f (t) dt + g(t) dt i
a a a

dF df dg
(t) = (t) + (t) i
dt dt dt

Examples
1)
Z 1
2t + 3t2 i dt = t2 ]10 + t3 ]10 i
0
= 1 + i.
68 LECTURE 10. COMPLEX VECTOR SPACES

2) Z 2π Z 2π
(1+i)t
e dt = et cos t + et sin t dt
0 0

We have
Z 2π Z 2π
t
e cos t dt = cos tet ]2π
+ 0 et sin t dt
0 0
Z 2π
2π
= e −1− et cos t dt
0

Thus

Z 2π
1
et cos t dt = (e2π − 1).
0 2

Similarly we have

Z 2π
1
et sin t dt = − (e2π − 1).
0 2

Thus,
2π
e2π − 1
Z
e(1+i)t dt =
(1 − i).
0 2
But we could have also used the rule
Z
1
eat dt = eat + C
a
where a is any complex number, a 6= 0. Then

Z 2π
1 (1+i)t 2π
e(1+i)t dt = e ]0
0 1+i
1
= [e2π+i2π − e0 ]
1+i
1
= [e2π − 1]
1+i
10.5. COMPLEX VECTOR SPACES 69

Furthermore
1 1
= (1 − i)
1+i 2
Thus we have
2π
e2π − 1
Z
e(1+i)t dt = (1 − i)
0 2
R1
Example. Evaluate the integral 0 (t + 2it2 )(t2 − 3it) dt.
Solution: First we have to carry out the multiplication
(t + 2it2 )(t2 − 3it) = t3 + 6t3 − 2it4 − 3it2
= 7t3 − (2t4 + 3t2 )i
Thus
Z 1 Z 1 Z 1
2 2 3
(t + 2it )(t − 3it) dt = 7t dt − ( 2t4 + 3t2 )i
0 0 0
7 41 2
= t ]0 − [ t5 + t3 ]10 i
4 5
7 7
= − i.
4 5

10.5 Complex vector spaces

The axioms for complex vector spaces are the same as those for real vector
spaces except the scalars are now complex numbers.
Example. Let
Cn = {(z1 , . . . , zn ) : z1 , . . . , zn }.
The addition is given by
u = (z1 , . . . , zn ), v = (w1 , . . . , wn ).
u + v = (z1 + w1 , . . . , zn + wn )
The scalar multiplication is given by
λu = (λz1 , . . . , λzn ).
Example. Let I ⊂ R be an interval. Let V be the space of functions f : I −→
C. The addition and scalar multiplication are given by
(f + g)(t) = f (t) + g(t)
(λf )(t) = λf (t).
70 LECTURE 10. COMPLEX VECTOR SPACES

10.5.1 Complex Inner Product

Let V be a complex vector space. A map (·, ·) : V × V −→ C is called an
inner product if

1. (u, u) ≥ 0 for all u ∈ V .

2. (u, u) = 0 if and only if u = 0.

3. For fixed v ∈ V , the map u 7→ (u, v) is linear,

i.e., (λu + µw, v) = λ(u, v) + µ(w, v)
for all λ, µ ∈ C and all u, w ∈ V .

4. For all u, v ∈ V we have (u, v) = (v, u).

Lemma. Let (·, ·) be an inner product on the complex vector space V . Then

(v, λu + µw) = λ(u, v) + µ(v, w)

for all λ, µ ∈ C and all u, v, w ∈ V .

Proof. We have

(v, λu + µw) = (λu + µw, v)

= λ(u, v) + µ(w, v)
= λ (u, v) + µ (w, v)
= λ(v, u) + µ(v, w)

Examples
Example (1). Let V = Cn . Define for u = (z1 , . . . , zn ) and v = (w1 , . . . , wn )

(u, v) = z1 w1 + · · · + zn wn .

Then

(u, u) = z1 z1 + · · · + zn zn
= |z1 |2 + · · · + |zn |2 ≥ 0.
10.5. COMPLEX VECTOR SPACES 71

If (u, u) = 0, then we must have |z1 | = |z2 | = · · · |zn | = 0, so u = 0. Let

y = (t1 , . . . , tn ) ∈ V and λ ∈ C. Then

(u + y, v) = (z1 + t1 )w1 + · · · + (zn + tn )wn

= (z1 w1 + · · · + zn wn ) + (t1 + · · · + tn )wn
= (u, v) + (y, v).

Similarly,

(λu, v) = (λz1 )w1 + · · · + (λzn )wn

= λ(z1 w1 ) + · · · + λ(zn wn )
= λ(u, v).

Finally,

(u, v) = z1 w1 + · · · + zn wn
= z1 w1 + · · · + zn wn
= (v, u).

Example (2). Let V be the space of piecewise continuous functions f :

[0, 1] −→ C. Define Z 1
(f, g) = f (t)g(t) dt.
0

10.5.2 Norm in a complex vector space

Definition. Let V be a complex vector space with inner product (·, ·). Then
the norm (or length) of a vector u ∈ V is defined by
p
kuk := (u, u).

Remark. Notice that kuk = 0 if and only if u = 0 and that kλuk = |λ| kuk.

Examples
1. V = C2 and u = (1, i). Then

kuk2 = 1 + ii = 1 + i(−i) = 1 + 1 = 2.
72 LECTURE 10. COMPLEX VECTOR SPACES

2. V = C2 and u = (1 + i, 2 + 3i):

kuk2 = (1 + i)(1 − i) + (2 + 3i)(2 − 3i) = 1 + 1 + 4 + 9 = 15

√
or kuk = 15.

3. Let V be the space of piecewise continuous functions on [0, 1]. Let

a ∈ R and

f (t) = eait = cos (at) + i sin (at).

Then

f (t)f (t) = |f (t)|2 = (cos (at) + i sin (at))(cos (at) − i sin (at))
= (cos (at))2 + (sin (at))2
= 1.

Hence,
s s
Z 1 Z 1
kf k = |f (t)|2 dt = 1 dt = 1.
0 0

4. Let V be the space of piecewise continuous functions on [0, 1]. Let

f (t) = t + it2 .

Then

|f (t)|2 = (t + it2 )(t − it2 )

= t2 + t4 .

Hence,
s r r
Z 1
1 1 8
kf k = t2 + t4 dt = + = .
0 3 5 15
10.5. COMPLEX VECTOR SPACES 73

5. Let V be the space of piecewise continuous functions on [0, 1]. Let

f (t) = t2 + 1 + 2i(t − 3).

Then

|f (t)|2 = (1 + t2 )2 + 4(t − 3)2

= 1 + 2t2 + t4 + 4t2 − 24t + 36
= t4 + 6t2 − 24t + 37.

Hence,
Z 1
2
kf k = t4 + 6t2 − 24t + 37 dt
0
1
= + 2 − 12 + 37
5
136
= .
5
74 LECTURE 10. COMPLEX VECTOR SPACES
Lecture 11

Discrete and Fast Fourier

Transforms

11.1 Introduction
The goal of the chapter is to study the Discrete Fourier Transform (DFT)
and the Fast Fourier Transform (FFT). In the course of the chapter we will
see several similarities between Fourier series and wavelets, namely

• Orthonormal bases make it simple to calculate coefficients,

• Algebraic relations allow for fast transform, and

• Complete bases allow for arbitrarily precise approximations.

There is, however, a very important difference between Fourier series and
wavelets, namely
Wavelets have compact support, Fourier series do not.

11.2 The Discrete Fourier Transform (DFT)

11.2.1 Definition and Inversion
Let ~e0 , . . . , ~eN −1 denote the usual standard basis for Cn . A vector ~f =
(f0 , . . . , fN −1 ) ∈ CN may then be written as ~f = f0~e0 + · · · + fN −1~en−1 .

75
76 LECTURE 11. DISCRETE AND FAST FOURIER TRANSFORMS

Important example:Assume the array ~f is a sample from a function

f : R → C, that is, we use the sample points x0 := 0, . . . , x` := ` ·
(2π/N ), . . . , xN1 := (N −1)·(2π/N ) with values ~f = (f (x0 ), . . . , f (x` ), . . . , f (xN −1 )).
The Discrete Fourier Transform expresses such an array ~f with linear
combinations of arrays of the type

N −1
~ k := eikx` = 1, eik2π/N , . . . , eik`·2π/N , . . . , eik(N −1)·2π/N


w `=0

k`
wk )` = ei·2π/N
= (~ =: ωNk`
.

Definition For each positive integer N , we define an inner product on CN

by
N −1
1 X
h~z, w
~ iN = zm · wm .
N m=0

Lemma For each positive integer N , the set

{~
wk | k ∈ {0, . . . , N − 1}}
is orthonormal with respect to the inner product h·, ·iN .
In fact {w ~ N −1 } is an orthonormal basis for CN .
~ 0, . . . , w

Sketch of Proof For all k, ` ∈ Z so that ` = k + JN for some J, we have

N −1 N −1
1 X 1 X ikm·2π/N −i`m·2π/N
hw ~ ` iN =
~ k, w (w
~ k )m (w
~ ` )m = e e
N N
m=0 m=0
N −1 N −1
1 X
ikm·2π/N −i(k+JN )m·2π/N 1 X
= e e = 1 = 1.
N N
m=0 m=0

For the remaining k, ` ∈ Z we use the geometric series to see that

N −1 N −1
1 X 1 X i[k−`]m·2π/N
h~ ~ ` iN
wk , w = (~
wk )m (~
w` )m = e
N N
m=0 m=0
N −1
1 X
m 1 1 − (ei[k−`]m·2π/N )N
= ei[k−`]·2π/N =
N N 1 − ei[k−`]·2π/N
m=0
1 1 − (e2πi )[k−`] 1 1−1
= i[k−`]·2π/N
= = 0.
N 1−e N 1 − ei[k−`]·2π/N
11.3. DISCRETE FOURIER TRANSFORM 77

11.2.2 The Fourier matrix

N
Definition For each positive integer N , define the Fourier matrix FΩ by
N
F Ωk,` w` )k = eik`2π/N = ωN
= (~ k`
.
N
Example: If N = 1, then ωN = ω1 = 1, and FΩ = 1. If N = 2, then
ωN = ω2 = −1, and

(ω20 )0 (ω21 )0
   
2 1 1
FΩ = = (the Haar matrix)
(ω20 )1 (ω21 )1 1 −1

If N = 4, then ωN = ω4 = ei2π/4 = i, and

(i1 )0 (i2 )0 (i3 )0

   
1 1 1 1 1
4 1
 (i1 )1 (i2 )1 (i3 )1 
 = 1 i −1 −i  .
 
FΩ = 
1 (i1 )2 (i2 )2 3
(i ) 2  1 −1 1 −1
1 (i1 )3 (i2 )3 3
(i ) 3 1 −i −1 i

11.3 Discrete Fourier Transform

Definition For each positive integer N and each array f~ ∈ CN , the Discrete
Fourier Transform of f~ is the array f̂ defined by
N −1
ˆ ~ 1 X
fk = hf , w
~ k iN = fm · e−ikm·2π/N .
N m=0

It is very important to understand this definition. The left hand side is

simply the orthogonal projection of ~f onto the basis vector w
~ k.
Since the DFT consists of the coefficients of ~f expressed with respect to
N −1
the new basis (~
wk )k=0 , the DFT simply “rotates” the coordinates in CN .

11.3.1 Two Results

Proposition The DFT corresponds to a multiplication by the transposed
conjugate matrix N1 N T
FΩ .
78 LECTURE 11. DISCRETE AND FAST FOURIER TRANSFORMS

Proof: Simply note that

N −1
1 X
fˆN,k = h~f , w
~ k iN = f` e−ik`·2π/N
N
m=0
N −1
1 X 1
N ΩT N ΩT · ~f k .


= F · f` = · F
N N
m=0

Proposition 13 For each positive integer N and each array ~f ∈ CN , we

have the following inversion formulas:
N −1 N −1
X 1 X
~f = fˆk w
~k , fˆk = h~f , w
~ k iN = fm · e−ikm·2π/N .
N
m=0 m=0

−1
wk )N
One proof The easiest way to prove Proposition 13 is to note that (~ k=0
N
forms an orthonormal basis for C . Therefore, the formula
N
X −1 N
X −1
~f = fˆk w
~k = h~f , w
~ k iN w
~k
k=0 k=0

represents the orthogonal projection of ~f on CN . But the projection of ~f on

CN is ~f , since ~f already lies in the range of the projection (the range being
CN ).

Example With N = 4, the fourth root of unity is ωN = ω4 = i. The N

~ k thus become
arrays w
~ 0 = [i]0·0 , [i]0·1 , [i]0·2 , [i]0·3 = (1, 1, 1, 1),


w
~ 1 = [i]1·0 , [i]1·1 , [i]1·2 , [i]1·3 = (1, i, −1, −i),


w
~ 2 = [i]2·0 , [i]2·1 , [i]2·2 , [i]2·3 = (1, −1, 1, −1),


w
~ 3 = [i]3·0 , [i]3·1 , [i]3·2 , [i]3·3 = (1, −i, −1, i).


w

Assume ~f = (f0 , f1 , f2 , f3 ) = (9, 7, 5, 7). Then (remember the complex con-

jugation!):
fˆ0 = h~f , w
~ 0 iN = h(9, 7, 5, 7), (1, 1, 1, 1)i4 = 7,
ˆ ~
f1 = hf , w ~ 1 iN = h(9, 7, 5, 7), (1, i, −1, −i)i4 = 1,
fˆ2 = h~f , w
~ 2 iN = h(9, 7, 5, 7), (1, −1, 1, −1)i4 = 0,
fˆ3 = h~f , w
~ 3 i = h(9, 7, 5, 7), (1, −i, −1, i)i = 1.
N 4
11.4. UNITARY OPERATORS 79

Therefore
~f = (9, 7, 5, 7)
= 7 · (1, 1, 1, 1) + 1 · (1, i, −1, −i)
+ 0 · (1, −1, 1, −1) + 1 · (1, −i, −1, i)

is the Inverse Discrete Fourier Transform of the array ~f = (9, 7, 5, 7).

11.4 Unitary Operators

We have just seen that the Discrete Fourier Transform is a linear operator

~f 7→ b 1
DF T : CN → CN , f= N ΩT ~f .
NF

Its inverse is given by

DF T −1 : CN → CN , f 7→ ~f = N
b ~
F Ωf .

√
Using the multiplicative constant 1/ N instead of 1/N , the Discrete
Fourier Transform thus is multiplication by the matrix N √1 N T
F U := N F Ω .
T
−1
In the exercises you are asked to show that (N
F U) = (N
F U ) . Such an
operator has a special name:
Definition 17 For each linear space V with an inner product h·, ·i, a
linear operator L : V → V is unitary if

hLv, Lwi = hv, wi

for all v, w ∈ V .

11.5 The Fast Fourier Transform

11.5.1 Introduction
We have seen how to convert a sample ~f = (f0 , . . . , fN −1 ) to a frequency
~
sample f̂ = (fˆ0 , . . . , fˆN −1 ) and back again. But there is an aspect we haven’t
touched upon yet, namely
80 LECTURE 11. DISCRETE AND FAST FOURIER TRANSFORMS

How long does it take to perform these calculations?

Let us try to estimate the number of calculations that are required. So let
us start with N complex numbers (f0 , . . . , fN −1 ).

• Each number fj = Refj + iImfj is being multiplied by e−2πijn/N =

cos 2πijn
N
− i sin 2πijn
N
; this gives 4N operations.

• This has to be done for each of the numbers fj , totalling 4N 2 real

multiplications, or N 2 complex multiplications.

• The numbers should then be added, but that would merely amount to
2N operations. Our number of operations is thus 4N 2 .

So how big is this number?

An Example Assume we need one minute to compute the Fourier trans-

form for a sequence with 4 samples. It would therefore take us approximately
60 60
= ≈ 0.94sec per operation.
4 × 42 64
4 × 82 × 15
• With N = 8 we would need = 4 min,
16 × 60

4 × 162 × 15
• With N = 16 we would need = 16 min,
16 × 60

4 × 322 × 15
• With N = 32 we would need = 64 min, and
16 × 60

4 × 2562 × 15
• With N = 28 = 256 we would need = 4096 min, (almost
16 × 60
three days).

• A standard TV need roughly 10000000 pixel values every second to preserve

relevant information. This would take us around 118900256 years (!!!) to
analyze all this information.
11.5. THE FAST FOURIER TRANSFORM 81

11.5.2 The Forward FFT

The point about the Fast Fourier Transform (FFT) is that it gives the same co-
efficients as the Discrete Fourier Transform (DFT) but requires less calculations.
For each integer N of the form N = 2k for some k and for each array

~f = (f0 , . . . , fN −1 ) = (f (0), . . . , f (N − 1)) ∈ CN ,

we define two arrays

~ = (f (0), f (2), . . . , f (2j), . . . , f (N − 2)) and

even f
~
odd f = (f (1), f (3), . . . , f (2j + 1), . . . , f (N − 1)).

We want to know how the DFT of even~f and the DFT of ~ are related to
odd f
the DFT of ~f . The answer is given in the next result:

Central result
Lemma 19 For each k ∈ {0, 1, . . . , N2 −1} and all arrays ~f = (f (0), . . . , f (N −
1)) ∈ CN

1  
fˆk = · even fˆk + [e−i·2π/N ]k · [odd fˆk ] ,
2
1  
fˆ N = · even fˆk − [e−i·2π/N ]k · [odd fˆk ] .
k+
2 2

We will not give a proof but merely see what the Lemma says in the case
N = 2. Then N/2 = 1 and ~f = (f0 , f1 ). Thus even~f = (f0 ) and odd~f = (f1 ).
Using that e−i·2π/2 = −1, the Lemma simply says that

f0 + f1 f0 − f1
fˆ0 = and fˆ1 = .
2 2

Time required?
We will only count complex operations.

• We start with an array ~f = (f0 , . . . , fN −1 ) where N = 2k .

• The DFT needed approximately N 2 complex multiplications.

82 LECTURE 11. DISCRETE AND FAST FOURIER TRANSFORMS

• Using the FFT (where we decompose ~f into two smaller arrays, divide
each of these into two smaller arrays, and so on), we end up with k
arrays each of length 2.

We thus need around kN complex multiplications instead of N 2 .

Is this significant?
In particular, taking N = 210 ≈ 1.024 × 103 and calculate the time, the DFT
takes
4 × 220 × 15
≈ 1247 years
16 × 60 × 60 × 24 × 365
but the FFT only takes
4 × 10 × 210 × 15
≈ 10.67 min.
16 × 60 × 60
Obviously this is a very distinct reduction in the time required to perform
the calculations :-)

11.5.3 The Inverse FFT (IFFT)

For each integer N , the FFT admits an inverse map that we call the Inverse
Fast Fourier Transform (IFFT). Starting with an array f̂ ∈ CN we thus
contruct the array of data ~f ∈ CN such that f̂ is the DFT of ~f .
Using that ~f = fˆ0 w
~ 0 + · · · + fˆk w
~ k + · · · + fˆN −1 w
~ N −1 we thus get for the
coordinates with index ` that
N
X −1 N
X −1
f` = fˆk (~
wk )` = fˆk eik`·2π/N .
k=0 k=0

We may interpret this formula as yet another FFT:

N
X −1 N
X −1
f` = fˆk eik`·2π/N = fˆk eik`·2π/N
k=0 k=0
N −1
1 X ˆ
=N· fˇk e−ik`k·2π/N = N · fˆ`
N
k=0

So in order to calculate the data ~f from f̂ , it suffices to form the complex

conjugate of f̂ , take its FFT multiplied by N , and then take the complex
conjugate one more time.
11.5. THE FAST FOURIER TRANSFORM 83

11.5.4 Interpolation by the IFFT

Thus far we have used the sample points x` = ` · 2π
N
, but the IFFT provides
a way to interpolate the function f in other points. To see how it works, we
recall that (by definition of the DFT),
N
X −1
f (x` ) = fˆk eikx` .
k=0

One way to interpolate f at x is therefore to approximate f (x) by

N
X −1
f (x) ≈ fˆk eikx .
k=0

However, if we consider several values of x spaced by multiples of 2π

N
, the
Fast Fourier Transform is a more convenient tool. So let u = x − x` for any
`. Thus
N1
X N
X −1
f (x) ≈ fˆk eikx = fˆk eik[u+x` ]
k=0 k=0
N
X −1   N
X −1  
= fˆk eiku eikx` = fˆk eiku eik`·2π/N ,
k=0 k=0

where the sums thus obtained are just the IFFT applied to the coefficients
ˆ N −1
(fk eiku )k=0 . Therefore, to interpolate f at the points x = u + ` · 2π , ` ∈
N
ˆ iku iu k
{0, . . . , N − 1}, it suffices to multiply each coefficient fk by e = (e ) and
then calculate the IFFT of the array (fˆk eiku )N −1
k=0 .

11.5.5 Bit Reversal

One way to better facilitate a recursive calculation of the Fast Fourier Trans-
form is to rearrange the initial date in such a way that the Fast Fourier
Transform operates on adjacent pairs of arrays in each step of the recursion.
More precisely:
Definition 23 Bit reversal transforms a finite sequence (pk−1 , pk−2 , . . . , p1 , p0 )
of k (binary) integers pj ∈ {0, 1} into the bit-reversed sequence (p0 , p1 , . . . , pk−2 , pk−1 ).
84 LECTURE 11. DISCRETE AND FAST FOURIER TRANSFORMS

So bit reversal transforms a binary integer

p = pk−1 2k−1 + pk−2 2k−2 + · · · + p1 2 + p0

into the binary integer

q = B(p) := pk−1 + pk−2 2 + · · · + p1 2k−2 + p0 2k−1 .

Proposition 24 For each index N = 2n , the FFT amounts to the following

chain of operations:
(0) For each binary index p ∈ {0, . . . , N − 1}, calculate the bit-reversed
index q = B(p) and arrange the data ~z = (z0 , . . . , zN −1 ) in the order
~zB := (zB(0) , . . . , zB(n−1) );

(1) For each k ∈ {0, . . . , n − 1}, perform one step of the FFT on each of the
2(n−k)−1 pairs of adjacent sequences of 2k elements.

We can therefore perform the Fast Fourier Transform in the following way:

• For each p ∈ {0, . . . , N − 1}, calculate q = B(p) and arrange the data
~z = (z0 , . . . , zN −1 ) as

~zB = (zB(0) , . . . , zB(N −1) )

• Compute the N/2 Fast Fourier Transform steps from one to two points
of each of the N/2 pairs:

([ẑB(0) , ẑB(1) ], . . . , [ẑB(N −2) , ẑB(N −1) ]).

• Compute the N/4 Fast Fourier Transform steps from two to four points
of each of the N/4 sequences of four numbers:

([ẑB(0) , ẑB(1) , ẑB(2) , ẑB(3) ], . . . ,

[ẑB(N −3) , ẑB(N −2) , ẑB(N −1) , ẑB(N −1) ]).

• Compute the Fast Fourier Transform step from N/2 to N points of the
sequence of N numbers:

([ẑB(0) , . . . , ẑB(N/2)−1) ], [ẑB(N/2) , . . . , ẑB(N −1) ]).

11.5. THE FAST FOURIER TRANSFORM 85

11.5.6 Applications of the FFT

Noise Reduction Through FFT
The idea is that if we are given a periodic signal with a larger amplitude
added to noise, the Fast Fourier Transform can be used to decompose the the
superposition of noise and the signal itself into a linear combination if terms
with different frequencies. This will identify and preserve the contribution
of the signal due to the larger coefficients and rejects the smaller coefficients
from the noise. The original signal thus reemerges.
See p.165-167 in the book.

11.5.7 Multidimensional DFT and FFT