LECTURE 17

Classification
Building models of classification in sklearn

Data Science, Fall 2023 @ Knowledge Stream

Sana Jabbar

1
 • Introduction to Classification
• Types of Classification
• Classification Algorithms
• Performance Metrics
• Overfitting and Underfitting
• Conclusion

Outline
Lecture 16

2
Classification
● Classification is defined as the process of recognition and grouping of objects
● Classification refers to a predictive modeling problem where a class label is
predicted for a given example of input data
● For Classification, the training dataset must be sufficiently representative of
the problem and have many examples of each class label.
1. Logistic Regression
2. k-Nearest Neighbors
3. Decision Trees
4. Support Vector Machine
5. Naive Bayes

3
Logistic Regression

4
Logistic Regression

5
K-Nearest Neighbor

• K nearest neighbors is a simple algorithm that stores all available

cases and classifies new cases based on a similarity measure.
• Algorithms for classification
1. Logistic Regression
2. k-Nearest Neighbor
3. Decision Trees
4. Support Vector Machine
5. Naive Bayes

6
Decision Trees (DTs)

• A hierarchical, tree structure, consists of a root node, branches,

internal nodes and leaf nodes
• Algorithms for binary classification
1. Logistic Regression
2. k-Nearest Neighbors
3. Decision Trees
4. Support Vector Machine
5. Naive Bayes

7
Support Vector Machine (SVM)

• SVM is used to classify data by finding the optimal decision

boundary that maximally separates different classes
• Algorithms for binary classification
1. Logistic Regression
2. k-Nearest Neighbors
3. Decision Trees
4. Support Vector Machine
5. Naive Bayes

8
Support Vector Machine (SVM)

● We have linear separable classes.

● We can have multiple hyperplanes separating these classed.

Q: Which one is the best decision boundary?

A: Maximum Margin Classifier (e.g., Support Vector Machine)
9
Support Vector Machine (SVM)

● Idea: Choose a fat separator

● The best boundary is the one that maximizes the margin or distance
between the boundary and the “difficult points” close to the
decision boundary.
● Margin pishes against the data points are called support vectors.
● Hard margin idea: Find a maximum margin
classifier with no errors on the training
data.
● Soft margin idea: Find the maximum
margin classifier while minimizing the
number of training errors.

10
Support Vector Machine (SVM)

● from sklearn.svm import SVC

svm_classifier = SVC(kernel='linear', C=1.0)
svm_classifier.fit(X_train, y_train)
y_pred = svm_classifier.predict(X_test)

You can choose different kernels (e.g., 'linear', 'poly', 'rbf') by

specifying the kernel parameter. The C parameter controls the trade-
off between maximizing the margin and minimizing the classification
error.

11
Naive Bayes

• The Naïve Bayes classifier is used for classification tasks, like text
classification.
• Algorithms for binary classification
1. Logistic Regression
2. k-Nearest Neighbors
3. Decision Trees
4. Support Vector Machine
5. Naive Bayes

12
Naive Bayes

● The Naive Bayes classifier is a family of probabilistic machine

learning models based on Bayes' theorem.
● The "naive" assumption is of feature independence.
● It's commonly used for text classification tasks, particularly for
document categorization and spam email filtering.
● Estimate from the data using the Bayes Theorem

13
Naive Bayes

Given Outlook, Temperature,

Humidity, and Wind
Information, we want to carry
out a prediction for Play: Yes or
No.

Mathematically

Predict for sunny outlook,

humidity high, cool temp, and
weak wind.

14
Naive Bayes

15
16
Naive Bayes

Play = No is more likely!

17
Performance Matrices

Confusion
Matrix

Classification

18
Performance Matrices

Confusion
Matrix

Recall

Classification
Classification
Model

19
Performance Matrices

Confusion
Matrix

Recall

Classification
Classification
Model

Precision

20
Performance Matrices

Confusion
Matrix

Recall

Classification
Classification
Model

Precision

F1-SCore
21
Performance Matrices

Confusion
Matrix

Recall

Classification
Classification
Model ROC curve is a graphical
representation of the
performance of the binary
Precision ROC-AUC classifier

F1-SCore
ROC Curve
● We get the data of 10 people about how high blood level is and whether
there is disease or not.

23
ROC Curve
● 5 individuals have the disease and 4 are classified correctly and 1 is
misclassified.
● The true positive rate (TPR) is 0.8

24
ROC Curve
● 5 individuals are healthy and 3 are classified correctly and 2 are
misclassified.
● The false positive rate (FPR) is 0.4

25
ROC Curve
● We calculate the value of TPR and FPR for different value of thresholds.

26
ROC Curve
● The true positive rate (TPR) is 1
● The false positive rate (FPR) is 1

27
ROC Curve
● The true positive rate (TPR) is 1
● The false positive rate (FPR) is 0.8

28
ROC Curve
● The true positive rate (TPR) is 0
● The false positive rate (FPR) is 0

29
ROC Curve

● For this point (0.2, 0.8)

● 80% of individuals are
correctly classified as
with diseases.
● 20% of indiviuals are
misclassified as with
diseases.

30
ROC Curve

● from sklearn.metrics import roc_curve

● from sklearn.metrics import roc_auc_score

● fpr, tpr, thresholds = roc_curve(y_true, y_scores)

● auc = roc_auc_score(y_true, y_scores)

31
Performance Matrices

Confusion
Matrix

Recall Accuracy

Classification
Classification
Model

Precision ROC-AUC

F1-SCore
32