UNIT 4

LEARNING

Forms of Learning in Machine Learning

1 Supervised Learning

Definition: The model learns from labeled data (input-output pairs). The goal is to predict the output for new, unseen
data based on the patterns learned from the training set.
OR
Definition: A machine learning model is trained using labeled data, where the input features and corresponding
outputs are known.

Examples:

Classification: Predicting discrete labels (e.g., email spam vs. not spam).

Regression: Predicting continuous values (e.g., housing prices).

Evaluation Metrics:

Accuracy: The percentage of correct predictions.

Precision & Recall: Measures of the relevance of the predictions, especially important in imbalanced datasets.

F1 Score: Harmonic mean of precision and recall.

2 Unsupervised Learning

Definition: The model learns from unlabeled data. The goal is to uncover hidden patterns or structures without
predefined outputs.

Examples:

Clustering: Grouping data into clusters (e.g., customer segmentation).

Dimensionality Reduction: Reducing the number of features while maintaining essential information (e.g., PCA).

3 Semi-supervised Learning

Definition: Uses both labeled and unlabeled data. It is particularly useful when labeling data is costly or time-
consuming.

Example: Labeling a small portion of a dataset and using a large portion of unlabeled data to train a model.

4 Reinforcement Learning

Definition: Learning through trial and error. The model (agent) interacts with the environment and receives feedback
in the form of rewards or penalties, which helps improve future decisions.

Examples: Game-playing AI (e.g., AlphaGo), robotic control systems.

Regression and Classification with Linear Models

Linear Regression

Definition: Predicts a continuous target variable based on one or more features.

Equation:

y = w1*x1 + w2*x2 + ... + wn*xn + b

Goal: Find weights w that minimize the error (typically using MSE loss).

Logistic Regression

Definition: Used for binary classification tasks. Outputs probabilities between 0 and 1 using the sigmoid function.

Equation:

p = 1 / (1 + exp(-z)) where z = wX + b.

Key Differences:

Linear Regression: Used for continuous outputs.

Logistic Regression: Used for binary classification.

Applications:

Linear Regression: Predicting house prices, sales forecasting.

Logistic Regression: Email classification, disease detection.

Clustering and Dimensionality Reduction

Clustering:
• Definition: Unsupervised learning technique that groups similar data points together.
• Types:
o K-means: Groups data into k clusters by minimizing the distance between points and cluster centers.
o Hierarchical Clustering: Builds a tree of clusters.
Dimensionality Reduction:
• Goal: Reduce the number of features while preserving data variance.
• Methods:
o Principal Component Analysis (PCA): Transforms the data into a set of orthogonal components
that explain the most variance.
o t-SNE: A technique for reducing dimensions and visualizing high-dimensional data.
Applications:
• Market segmentation, customer analysis, and pattern recognition.

Decision Trees
Overview:
A Decision Tree is a supervised learning algorithm that is used for both classification and regression tasks. It models
decisions and their possible consequences, including chance event outcomes, resource costs, and utility.
Structure:
• Nodes: Each internal node represents a test on an attribute (feature).
• Edges: Each branch represents the outcome of the test.
• Leaves: Terminal nodes represent class labels or continuous output values.
How it Works:
1. Splitting: The data is split into subsets based on the feature values. The goal is to create subsets that are as
pure as possible, meaning the data in each subset is as homogenous as possible.
2. Choosing the Best Split: Common methods for choosing the best feature to split on are:
o Gini Index: Measures the "impurity" of a node. The goal is to minimize Gini.
o Information Gain: Measures the effectiveness of a feature in classifying data. The feature that
results in the highest information gain is chosen for splitting.
3. Tree Depth: A deeper tree may lead to better fitting of data but could also lead to overfitting.

Support Vector Machine (SVM) (6 Pages)

Overview:
A Support Vector Machine (SVM) is a supervised learning model that can be used for both classification
and regression tasks. It is particularly powerful for high-dimensional data and has a solid theoretical
foundation. SVMs are designed to find the hyperplane that best separates data into two classes.
Key Concepts:
• Hyperplane: A decision boundary that separates data points from different classes. In 2D, this is a line; in
3D, it is a plane; in higher dimensions, it is a hyperplane.
• Support Vectors: The data points that are closest to the hyperplane and influence its position. These support
vectors are critical for defining the optimal decision boundary.
• Margin: The distance between the hyperplane and the support vectors. SVM aims to maximize this margin,
as it helps the model generalize better on unseen data.

Types of Support Vector Machines

1. Linear SVM:
o Definition: Used when the data is linearly separable (i.e., a straight line or hyperplane can separate
the data into classes).
o Working: The SVM algorithm finds the hyperplane that best divides the classes by maximizing the
margin between them.
o Applications: Email spam filtering, sentiment analysis.
2. Non-Linear SVM:
o Definition: Used when the data is not linearly separable. The Kernel Trick is applied to map data
into a higher-dimensional space where it becomes linearly separable.
3. Common kernels include:
o Linear Kernel: For linearly separable data.
o Polynomial Kernel: For more complex decision boundaries.
o Radial Basis Function (RBF) Kernel: For highly complex and non-linear boundaries.
 Advantages:
• Effective in High Dimensions: SVM works well in spaces with many features (e.g., text classification).
• Robust to Overfitting: Especially in high-dimensional spaces.
• Flexible: Can be adapted for both classification and regression (SVR - Support Vector Regression).
Disadvantages:
• Computationally Expensive: Especially for large datasets, training an SVM can be time-consuming.
• Memory Intensive: Requires large memory for storing support vectors.
• Sensitivity to Parameters: Performance is sensitive to the choice of the kernel, regularization parameter,
and other hyperparameters.
Applications:
• Text classification (e.g., spam filtering).
• Image classification (e.g., face detection).
• Bioinformatics (e.g., protein structure classification).

Artificial Neural Networks (ANN)

Overview:
An Artificial Neural Network (ANN) is a computational model inspired by the way biological neural
networks in the human brain process information. ANNs are widely used for tasks like pattern recognition,
classification, regression, and more. They are the foundation of deep learning.
Structure:
• Neurons (Nodes): Basic units that receive inputs, process them, and output a value.
• Layers:
o Input Layer: Receives the features.
o Hidden Layers: Perform transformations on the data. There can be multiple hidden layers in deep
networks.
o Output Layer: Produces the final output.
Types of Neural Networks
1. Feedforward Neural Networks (FNN):
o The simplest type of neural network where the data flows in one direction—from the input layer to
the output layer. No cycles or loops in the network.
o Applications: Basic classification and regression tasks.
2. Convolutional Neural Networks (CNN):
o Definition: Specialized type of neural network primarily used for image processing. CNNs use
convolutional layers to extract features from images.
o Working: Convolution operations apply filters (kernels) to the input image to extract spatial features.
o Applications: Image classification, object detection, facial recognition.
3. Recurrent Neural Networks (RNN):
o Definition: A type of neural network designed for sequence data, where outputs from previous steps
are fed back as inputs.
 o Types:
▪ Vanilla RNN: Basic RNN with feedback connections.
▪ Long Short-Term Memory (LSTM): A type of RNN that handles long-term dependencies
by using specialized memory cells.
▪ Gated Recurrent Units (GRU): A simpler version of LSTM.
o Applications: Time series forecasting, speech recognition, machine translation.

1. Training: The network is trained using a dataset and a loss function (e.g., Mean Squared Error for
regression, Cross-Entropy for classification). The goal is to minimize the error between predicted outputs
and true labels.
Advantages:
• Adaptability: Can be used for a variety of tasks, from classification to regression to complex tasks like
natural language processing.
• Non-linearity: Neural networks can model complex, non-linear relationships.
• Scalability: Can scale to very large datasets, particularly with deep networks.
Disadvantages:

• Require Large Datasets: Neural networks require a lot of data to perform well.
• Computationally Intensive: Training large networks requires significant computational resources (e.g.,
GPUs).
• Interpretability: Neural networks are often described as "black-box" models because understanding why a
network makes a particular decision can be difficult.
Applications:
• Image Recognition: Convolutional Neural Networks (CNNs) are a class of neural networks used for image
recognition tasks.
• Speech Recognition: Used in virtual assistants like Siri and Alexa.
• Natural Language Processing (NLP): Used in tasks like translation, sentiment analysis, and text
generation.