Principal Component Analysis (PCA)

1. PCA is a method used to reduce number of variables in dataset

by extracting important one from a large dataset.
2. It reduces the dimension of our data with the aim of retaining
as much information as possible.
3. In other words, this method combines highly correlated
variables together to form a smaller number of an artificial set
of variables which is called principal components (PC) that
account for most variance in the data.
4. A PC can be defined as a linear combination of optimally-
weighted observed variables.
5. The PC retains max variation that was present in the original
components.
6. The PCs are eigenvectors of a covariance matrix, & hence they are
orthogonal(Two vectors are said to be orthogonal to each other if and only their dot product is zero.).
7. The output of PCA are these PCs, the number of which is less than
or equal to the number of original variables.
8. The PCs possess some useful properties which are listed below :-
• It transforms a high-dimensional dataset into a lower-dimensional representation while
preserving the most important information. PCA achieves this by identifying the principal
components that are linear combinations of the original variables.
• The variation present in the PC decrease as we move from the 1st
PC to the last one.
 Steps in PCA:
a. Getting the dataset
b. Representing data into 2-D matrix, where row refer to data items and columns corresponds to
features.
c. Standardizing the data: If the features are on different scales, it's essential to standardize them
(mean=0, variance=1) to ensure that no feature dominates the analysis. Here the matrix will be
named as Z.
d. Calculating Covariance of Matrix(Z) : For this, take the matrix Z, transpose it and then multiply it
by Z. Output matrix will be covariance matrix of Z. This matrix shows how the features relate to
each other.
e. Eigenvalue Decomposition: PCA finds the eigenvalues and eigenvectors of the covariance matrix.
Each eigenvector represents a principal component, and the corresponding eigenvalue indicates
how much variance that component explains.
f. Sorting the Eigen vectors: Sort Eigen values in descending order and simultaneously sort the eigen
vectors accordingly.
g. Selection of Principal Components: The principal components are ranked by their corresponding
eigenvalues. The first few components that explain most of the variance are selected.
h. Removing less or unimportant features from new dataset: We will keep only the relevant or
important features in the new dataset and unimportant features will be removed.
 Applications of PCA
Field Application of PCA

Image Processing Image compression and feature reduction

Biomedical Data Analysis Gene expression analysis, biomarker discovery

Finance and Economics Risk assessment, portfolio optimization, time series

NLP Text classification, document clustering, sentiment analysis

Environmental Science Analyzing environmental data, pollution detection

Remote Sensing Analyzing satellite and remote sensing data

Speech Recognition Speech pattern recognition, improving accuracy

Signal Processing Audio and video compression, denoising

Biometric Authentication Reducing dimensionality in biometric data

These are just some of the many diverse applications of PCA in different fields.
PCA's ability to reduce dimensionality and uncover meaningful patterns in data makes it a valuable tool across a
wide range of domains.
 Compute the principal component using PCA Algorithm.

Given data = { 2, 3, 4, 5, 6, 7 ; 1, 5, 3, 6, 7, 8 }.

Consider the two dimensional patterns=➔>>

(2, 1), (3, 5), (4, 3), (5, 6), (6, 7), (7, 8).
Step-01: Step-03:

Get data. Subtract mean vector (µ) from the given feature vectors.
The given feature vectors are- x1 – µ = (2 – 4.5, 1 – 5) = (-2.5, -4)
•x1 = (2, 1) x2 – µ = (3 – 4.5, 5 – 5) = (-1.5, 0)
•x2 = (3, 5) x3 – µ = (4 – 4.5, 3 – 5) = (-0.5, -2)
•x3 = (4, 3) x4 – µ = (5 – 4.5, 6 – 5) = (0.5, 1)
•x4 = (5, 6) x5 – µ = (6 – 4.5, 7 – 5) = (1.5, 2)
•x5 = (6, 7) x6 – µ = (7 – 4.5, 8 – 5) = (2.5, 3)
•x6 = (7, 8)

Step-02:
Calculate the mean vector (µ).
Mean vector (µ)
= ((2 + 3 + 4 + 5 + 6 + 7) / 6,
Feature vectors (xi) after subtracting mean vector (µ) are-
(1 + 5 + 3 + 6 + 7 + 8) / 6) = (4.5, 5)

Thus,
 From step 3,we get:
Step-04:
Calculate the covariance matrix.

Covariance matrix

is given by-

Now,
Now, Covariance matrix = (m1 + m2 + m3 + m4 + m5 + m6) / 6

On adding the above matrices and dividing by 6, we get-

Step-05:

Calculate the eigen values and eigen vectors of the covariance matrix.
λ is an eigen value for a matrix M if it is a solution of the characteristic equation |M – λi| = 0..
So, we have-

From here,
(2.92 – λ)(5.67 – λ) – (3.67 x 3.67) = 0
16.56 – 2.92λ – 5.67λ + λ2 – 13.47 = 0
λ2 – 8.59λ + 3.09 = 0

Solving this quadratic equation, we get λ = 8.22, 0.38

Thus, two eigen values are λ1 = 8.22 and λ2 = 0.38.

Clearly, the second eigen value is very small compared to the first eigen value. So, the second eigen vector can be left out.
Eigen vector corresponding to the greatest eigen value is the PC for the given data set.

So, we find the eigen vector corresponding to eigen value λ1.

We use the following equation to find the eigen vector- MX = λX

where- M = Covariance Matrix X = Eigen vector λ = Eigen value

Substituting the values in the above equation, we get-

Solving these, we get-

2.92X1 + 3.67X2 = 8.22X1

3.67X1 + 5.67X2 = 8.22X2

On simplification, we get-

5.3X1 = 3.67X2 ………(1)

3.67X1 = 2.55X2 ………(2)

Equation (1): 5.3X1 = 3.67X2 Now, let's work with equation (2):
Equation (2): 3.67X1 = 2.55X2
Divide both sides of equation (2) by 3.67 to isolate X1:
Let's start with equation (1): (3.67X1) / 3.67 = (2.55X2) / 3.67

Divide both sides of equation (1) by 5.3 to isolate X1:

(5.3X1) / 5.3 = (3.67X2) / 5.3 Simplify:

Simplify: X1 = (2.55X2) / 3.67

X1 = (3.67X2) / 5.3

Now you have two expressions for X1: (i) X1 = (3.67X2) / 5.3 (ii) X1 = (2.55X2) / 3.67

So, you have found that X1 can be expressed in terms of X2 as follows: X1 = (3.67X2) / 5.3 = 0.69X2

Therefore, from equations (1) and (2), you can conclude that X1 is equal to 0.69 times X2.
From (1) and (2), X1 = 0.69X2

From (2), the eigen vector is-

Thus, principal component for the given data set is-

(Solution is in next slide)

 Solution (B)

To find the eigenvector corresponding to equation (2), which is: 3.67X1 = 2.55X2

Set up a system of equations and solve for X1 and X2. The equation can be rewritten as: 3.67X1 - 2.55X2 = 0

This represents a single equation with two variables, X1 and X2. To find the eigenvector, solve this equation by
expressing one variable in terms of the other.

Let's solve for X1 in terms of X2: 3.67X1 = 2.55X2

Now, divide both sides by 3.67 to isolate X1: X1 = (2.55X2) / 3.67

So, the relationship between X1 and X2 from equation (2) is: X1 = (2.55X2) / 3.67

Now, to represent this relationship as an eigenvector, you can write it in the following form:

| X1 | | 2.55 |
|----| = |------|
| X2 | | 3.67 |
So, the eigenvector corresponding to equation (2) is [2.55, 3.67].