EVALUATION

INTRODUCTION
Introduction
• Evaluation is the fifth stage in the AI project cycle.
• In modelling, we can make different types of models
• How do we check if one is better than the other?
• That’s where Evaluation comes into play.
• In the Evaluation stage, we will explore different
methods of evaluating an AI model.
• Model Evaluation is an integral part of the model
development process.
• It helps to find the best model that represents our
data and how well the chosen model will work in the
future
What is evaluation?
• Evaluation is the process of understanding
the reliability of any AI model, based on
outputs by feeding test dataset into the
model and comparing with actual
answers.
Evaluation
• There can be different Evaluation techniques,
depending of the type and purpose of the model.
• Remember that it’s not recommended to use the
data we used to build the model to evaluate it.
• This is because our model will simply remember
the whole training set, and will therefore always
predict the correct label for any point in the
training set. This is known as overfitting.
• Let us go through various terms which are very
important to the evaluation process
Model Evaluation Terminologies
• There are various new terminologies which come into
the picture when we work on evaluating our model. Let’s
explore them with an example of the Forest fire scenario.
• The Scenario
Imagine that you have come up with an AI based
prediction model which has been deployed in a forest
which is prone to forest fires. Now, the objective of the
model is to predict whether a forest fire has broken out
in the forest or not. Now, to understand the efficiency of
this model, we need to check if the predictions which it
makes are correct or not. Thus, there exist two
conditions which we need to ponder upon: Prediction
and Reality. The prediction is the output which is given
by the machine and the reality is the real scenario in the
forest when the prediction has been made. Now let us
look at various combinations that we can have with these
two conditions.
Case 1: Is there a forest fire?

• Here, we can see in the picture that a forest fire has

broken out in the forest. The model predicts a Yes which
means there is a forest fire. The Prediction matches with
the Reality. Hence, this condition is termed as True
Positive.
Case 2: Is there a forest fire?

• Here there is no fire in the forest hence the reality is

No. In this case, the machine too has predicted it
correctly as a No. Therefore, this condition is termed
as True Negative.
Case 3: Is there a forest fire?

• Here the reality is that there is no forest fire. But the

machine has incorrectly predicted that there is a
forest fire. This case is termed as False Positive.
Case 4: Is there a forest fire?

• Here, a forest fire has broken out in the forest because of

which the Reality is Yes but the machine has incorrectly
predicted it as a No which means the machine predicts
that there is no Forest Fire. Therefore, this case becomes
False Negative.
Confusion matrix
• The result of comparison between the prediction
and reality can be recorded in what we call the
confusion matrix.
• The confusion matrix allows us to understand
the prediction results.
• Note that confusion matrix is not an
evaluation metric but a record which can
help in evaluation.
Confusion matrix
Let us once again take a look at the four conditions
that we went through in the Forest Fire example:
Let us now take a look at the
confusion matrix:

Prediction and Reality can be easily mapped

together with the help of this confusion matrix.
Evaluation Methods
• Now as we have gone through all the possible
combinations of Prediction and Reality, let us see
how we can use these conditions to evaluate the
model.
• Accuracy
Accuracy is defined as the percentage of correct
predictions out of all the observations. A prediction
can be said to be correct if it matches the reality.
Here, we have two conditions in which the
Prediction matches with the Reality: True Positive
and True Negative.
Formula for Accuracy of a model

• Here, total observations cover all the possible cases

of prediction that can be True Positive (TP), True
Negative (TN), False Positive (FP) and False
Negative (FN).
Accuracy as an evaluation technique
• As we can see, Accuracy talks about how true the
predictions are by any model.
• Is high accuracy equivalent to good
performance?
• How much percentage of accuracy is reasonable
to show good performance?
Analysing Accuracy
• Let us go back to the Forest Fire example.
• Assume that the model always predicts that
there is no fire.
• But in reality, there is a 2% chance of forest fire
breaking out.
• In this case, for 98 cases, the model will be right
but for those 2 cases in which there was a forest
fire, then too the model predicted no fire.
• Here,
True Positives = 0
True Negatives = 98
Total cases = 100
Therefore, accuracy becomes:
(98 + 0) / 100 = 98%

This is a fairly high accuracy for an AI model. But

this parameter is useless for us as the actual cases
where the fire broke out are not taken into account.
Hence, there is a need to look at another parameter
which takes account of such cases as well.
Accuracy alone is not enough
Precision
• Precision is defined as the percentage of true
positive cases versus all the cases where the
prediction is true.
• That is, it takes into account the True Positives
and False Positives.
Precision as an evaluation technique
• Going back to the Forest Fire example, in this case, assume
that the model always predicts that there is a forest fire
irrespective of the reality.
• In this case, all the Positive conditions would be taken into
account that is, True Positive (Prediction = Yes and Reality =
Yes) and False Positive (Prediction = Yes and Reality = No).
In this case, the fire fighters will check for the fire all the time
to see if the alarm was True or False.
• You might recall the story of the boy who falsely cries out that
there are wolves every time and so when they actually arrive,
no one comes to his rescue.
• Similarly, here if the Precision is low (which means there are
more False alarms than the actual ones) then the fire fighters
would get complacent and might not go and check every time
considering it could be a false alarm.
Precision as an evaluation technique
• This makes Precision an important evaluation
criteria. If Precision is high, this means the True
Positive cases are more, giving lesser False
alarms.
• But again, is good Precision equivalent to a good
model performance? Why?
Drawback of precision
• Let us consider that a model
has 100% precision. Which
means that whenever the
machine says there’s a fire,
there is actually a fire (True
Positive).
• In the same model, there can
be a rare exceptional case
where there was actual fire but
the system could not detect it.
This is the case of a False
Negative condition.
• But the precision value would
not be affected by it because it
does not take FN into account.
Is precision then a good
parameter for model
performance? We will look at
a few more metrics and
decide.
Difference between Accuracy and
Precision
• Accuracy is how close a value is to its true value.
An example is how close an arrow gets to the
bull's-eye center. Precision is how repeatable a
measurement is. An example is how close a
second arrow is to the first one (regardless of
whether either is near the mark).

• In a set of measurements, accuracy is closeness

of the measurements to a specific value, while
precision is the closeness of the measurements
to each other.