Scribd
Upload
6 views

FNET EXPERIMENTATION

This document presents a comparative study of Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT) within FNet architectures, highlighting the advantages of FNet over traditional transformers in terms of memory usage and training speed. The study utilizes the Jigsaw Toxic Comment Classification dataset to evaluate model performance based on various metrics. Key methodologies include the implementation of FNet with different DCT variants and preprocessing steps for data preparation.

Uploaded by

Sachin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
6 views

FNET EXPERIMENTATION

This document presents a comparative study of Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT) within FNet architectures, highlighting the advantages of FNet over traditional transformers in terms of memory usage and training speed. The study utilizes the Jigsaw Toxic Comment Classification dataset to evaluate model performance based on various metrics. Key methodologies include the implementation of FNet with different DCT variants and preprocessing steps for data preparation.

Uploaded by

Sachin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 6

Comparative Study of DFT and

DCT in FNet Architectures


Submitted by Param Rastogi and
Sachin
Dayalbagh Educational Institute
November 2024
Introduction to Transformers
• What are Transformers?
• - Revolutionized NLP with self-attention
mechanisms (Vaswani et al., 2017).
• - Applications: Machine Translation, Text
Summarization, Sentiment Analysis.

• Limitations:
• - High memory usage due to quadratic
complexity of self-attention.
FNet Model Overview
• What is FNet?
• - Introduced by Lee-Thorp et al. in 2021.
• - Replaces self-attention with Fourier
Transform (FT).

• Key Advantages:
• - Faster training (80% on GPU, 70% on TPU).
• - Reduced memory requirements.
Methodology

• Model Architecture: FNet with DFT, 1D-DCT,


and 2D-DCT variants.
• Evaluation Metrics: Accuracy, Precision, Recall,
F1-score, Training time, and Memory usage.
• Experimental Design: Preprocessing steps
include tokenization, normalization, and
padding.
Dataset Description
• Dataset: Jigsaw Toxic Comment Classification
• - Purpose: Classify toxicity in online
comments.
• - Size: 230,000 labeled comments.
• - Labels: Toxic, Severe Toxic, Obscene, Threat,
Insult, Identity Hate.

• Preprocessing Steps:
• - Tokenization using TweetTokenizer.
FNet Model Architecture
• Refer to Figure 1 in the report for the
architecture diagram of FNet variants.

You might also like