Cross Validation in Machine Learning

Cross-validation is a technique used to check how well a machine learning model performs on unseen data while preventing overfitting. It works by:

• Splitting the dataset into several parts.
• Training the model on some parts and testing it on the remaining part.
• Repeating this resampling process multiple times by choosing different parts of the dataset.
• Averaging the results from each validation step to get the final performance.

Types of Cross-Validation

There are several types of cross-validation techniques which are as follows:

1. Holdout Validation

In Holdout Validation method typically 50% data is used for training and 50% for testing. Making it simple and quick to apply. The major drawback of this method is that only 50% data is used for training, the model may miss important patterns in the other half which leads to high bias.

2. LOOCV (Leave One Out Cross Validation)

In this method the model is trained on the entire dataset except for one data point which is used for testing. This process is repeated for each data point in the dataset.

• All data points are used for training, resulting in low bias.
• Testing on a single data point can cause high variance, especially if the point is an outlier.
• It can be very time-consuming for large datasets as it requires one iteration per data point.

3. Stratified Cross-Validation

It is a technique that ensures each fold of the cross-validation process has the same class distribution as the full dataset. This is useful for imbalanced datasets where some classes are underrepresented.

• The dataset is divided into k folds, keeping class proportions consistent in each fold.
• In each iteration, one fold is used for testing and the remaining folds for training.
• This process is repeated k times so that each fold is used once as the test set.
• It helps classification models generalize better by maintaining balanced class representation.

4. K-Fold Cross Validation

K-Fold Cross Validation splits the dataset into k equal-sized folds. The model is trained on k-1 folds and tested on the remaining fold. This process is repeated k times each time using a different fold for testing.

Note: It is always suggested that the value of k should be 10 as the lower value of k takes towards validation and higher value of k leads to LOOCV method.

Example of K Fold Cross Validation

The diagram below shows an example of the training subsets and evaluation subsets generated in k-fold cross-validation. Here we have total 25 instances.

• Here we will take k as 5.
• 1st iteration: The first 20% of data [1–5] is used for testing and the remaining 80% [6–25] is used for training.
• 2nd iteration: The second 20% [6–10] is used for testing and the remaining data [1–5] and [11–25] is used for training.
• This process continues until each fold has been used once as the test set.

Each iteration uses different subsets for testing and training, ensuring that all data points are used for both training and testing.

Comparison between K-Fold Cross-Validation and Hold Out Method

K-Fold Cross-Validation and Hold Out Method are widely used technique and sometimes they are confusing so here is the quick comparison between them:

Python implementation for k fold cross-validation

Step 1: Importing necessary libraries

We will import scikit learn.

from sklearn.model_selection import cross_val_score, KFold
from sklearn.svm import SVC
from sklearn.datasets import load_iris

Step 2: Loading the dataset

let's use the iris dataset which is a multi-class classification in-built dataset.

iris = load_iris()
X, y = iris.data, iris.target

Step 3: Creating SVM classifier

SVC is a Support Vector Classification model from scikit-learn.

svm_classifier = SVC(kernel='linear')

Step 4: Defining the number of folds for cross-validation

Here we will be using 5 folds.

num_folds = 5
kf = KFold(n_splits=num_folds, shuffle=True, random_state=42)

Step 5: Performing k-fold cross-validation

cross_val_results = cross_val_score(svm_classifier, X, y, cv=kf)

Step 6: Evaluation metrics

print("Cross-Validation Results (Accuracy):")
for i, result in enumerate(cross_val_results, 1):
    print(f"  Fold {i}: {result * 100:.2f}%")
    
print(f'Mean Accuracy: {cross_val_results.mean()* 100:.2f}%')

Output:

The output shows the accuracy scores from each of the 5 folds in the K-fold cross-validation process. The mean accuracy is the average of these individual scores which is approximately 97.33% indicating the model's overall performance across all the folds.

Advantages

1. Better performance estimate: Provides a more reliable evaluation than a single train-test split.
2. Reduces overfitting: Helps ensure the model generalizes well to unseen data.
3. Efficient use of data: All data points are used for both training and testing at different iterations.
4. Flexible: Works with different types of datasets and models.

Disadvantages

1. Computationally Expensive: It can be computationally expensive especially when the number of folds is large.
2. Time-consuming: Methods like LOOCV can take a long time for datasets with many data instances.
3. Bias-Variance Tradeoff: Few folds may result in high bias while too many folds may result in high variance.