UNIT - I

Introduction to Data Science

Data Science
T. Y. BTECH

SCHOOL OF COMPUTER ENGINEERING AND TECHNOLOGY

Prepared By Shilpa Sonawani
 5 Vs of Big Data

• Raw Data: Volume

• Change over time: Velocity
• Data types: Variety
• Data Quality: Veracity
• Information for Decision Making: Value
 Data Science – A Definition

Data Science is the science which uses computer

science, statistics and machine learning,
visualization and human-computer interactions
to collect, clean, integrate, analyze, visualize,
interact with data to create data products.

3
 Ben Fry’s Model
Visualizing Data Process
1. Acquire
2. Parse
3. Filter
4. Mine
5. Represent
6. Refine
7. Interact

4
 Jeff Hammerbacher’s Model
1. Identify problem

2. Instrument data sources

3. Collect data

4. Prepare data (integrate, transform, clean, filter, aggregate)

5. Build model

6. Evaluate model

7. Communicate results
5
 Data Analysts vs Data Scientists

DATA ANALYST SKILLS DATA SCIENTIST SKILLS

Data Mining Data Mining
Data Warehousing Data Warehousing
Math, Statistics Math, Statistics, Computer Science
Tableau and Data Visualization Tableau and Data
Visualization/Storytelling
SQL Python, R, JAVA, Scala, SQL, Matlab,
Pig
Business Intelligence Economics
SAS Big Data/Hadoop
Advanced Excel skills Machine Learning
Key Concept: Structured Data
A data model is a collection of concepts for
describing data.

A schema is a description of a particular

collection of data, using a given data model.
 NumPy/Python
• NumPy is a Python library used for working with arrays.
• It also has functions for working in domain of linear algebra, fourier
transform, and matrices.
• NumPy stands for Numerical Python.
• In Python we have lists that serve the purpose of arrays, but they are slow
to process.
• NumPy aims to provide an array object that is up to 50x faster than
traditional Python lists.
• The array object in NumPy is called ndarray, it provides a lot of supporting
functions that make working with ndarray very easy.
• Arrays are very frequently used in data science, where speed and resources
are very important
• NumPy arrays are stored at one continuous place in memory unlike lists, so
processes can access and manipulate them very efficiently
 Pandas
• Pandas officially stands for ‘Python Data Analysis
Library’, THE most important Python tool used by Data
Scientists today.
• Pandas is an open source Python library that allows users to
explore, manipulate and visualise data in an extremely
efficient manner. It is literally Microsoft Excel in Python.
• It is easy to read and learn
• It is extremely fast and powerful
• It integrates well with other visualisation libraries
• Pandas can take in a huge variety of data, the most common
ones are csv, excel, sql or even a webpage.
 Pandas/Python
• Series: a named, ordered dictionary
– The keys of the dictionary are the indexes
– Built on NumPy’s ndarray
– Values can be any Numpy data type object

• DataFrame: a table with named columns

– Represented as a Dict (col_name -> series)
– Each Series object represents a column

11
 Operations
• map() functions
• filter (apply predicate to rows)
• sort/group by
• aggregate: sum, count, average, max, min
• Pivot or reshape
• Relational:
– union, intersection, difference, cartesian product (CROSS
JOIN)
– select/filter, project
– join: natural join (INNER JOIN), theta join, semi-join, etc.
– rename
12
 Applications
Amazing real-time Data Science Applications:
Recommendation- Most of the apps and websites like Amazon, YouTube, Flipkart, etc. give
recommendation over as per the viewer’s interest. Online music applications like Spotify give
recommendations as per your taste in music. So these are good examples of data science
recommendation applications.
Search Results- Machine Learning algorithms used to find the most relevant search for Google
search engines. Such an algorithm used for the most visited sites on google chrome.
Intelligent Assistant- Google assistant, Siri are examples of intelligent assistants. The advanced
machine learning algorithm converts voice input into text output. These smart assistants
recognize the voice and provide the required information in both voice and text outputs.
Autonomous driving vehicles- Automobile companies like Waymo and Tesla looking for the next
generation of autonomous vehicles. 3D images were taken by the cameras and the information
provided to the algorithms for further processing.
Piracy Detection- YouTube is an example of piracy detection using machine learning algorithms.
Due to the big database, copied contents cannot be detected manually. So it helps to detect and
remove the copied content to reduce human efforts.
Image Recognition- Facebook is the application that uses image recognition by data science and
machine learning for the friend suggestion. Even Google lens uses an image recognition algorithm
to provide the related information to you.
Data Cleaning-Dirty Data
• The Statistics View:
• There is a process that produces data
• We want to model ideal samples of that process, but
in practice we have non-ideal samples:
• Distortion – some samples are corrupted by a process
• Selection Bias - likelihood of a sample depends on its
value
• Left and right censorship - users come and go from our
scrutiny
• Dependence – samples are supposed to be independent,
but are not (e.g. social networks)
 Dirty Data
• The Database View:
• Some of the values are missing, corrupted, wrong,
duplicated
• Results are absolute (relational model)
• You get a better answer by improving the quality
of the values in your dataset
 Dirty Data
• The Domain Expert’s View:
• This Data Doesn’t look right
• This Answer Doesn’t look right
• What happened?

• Domain experts have an implicit model of the data

that they can test against…
 Dirty Data

• The Data Scientist’s View:

• Some Combination of all of the above

Solution:
Data Quality Problems
• (Source) Data is dirty on its own.
• Transformations corrupt the data (complexity of software
pipelines).
• Data sets are clean but integration (i.e., combining them)
screws them up.
• “Rare” errors can become frequent after transformation or
integration.
• Data sets are clean but suffer “bit rot”
• Old data loses its value/accuracy over time

• Any combination of the above

 Dirty Data Problems
1) Parsing text into fields (separator issues)
2) Naming conventions: ER: NYC vs New York
3) Missing required field (e.g. key field)
4) Different representations (2 vs Two)
5) Fields too long (get truncated)
6) Primary key violation (from un- to structured or during
integration
7) Redundant Records (exact match or other)
8) Formatting issues – especially dates
9) Licensing issues/Privacy/ keep you from using the data
as you would like?
Conventional Definition of Data Quality

• Accuracy
– The data was recorded correctly.
• Completeness
– All relevant data was recorded.
• Uniqueness
– Entities are recorded once.
• Timeliness
– The data is kept up to date.
• Special problems in federated data: time consistency.
• Consistency
– The data agrees with itself.
How we can deal with the noisy data

• Data Binning : In this approach sorting of data is performed

concerning the values of the neighborhood. This method is also
known as local smoothing.
• Preprocessing in Clustering : In the approach, the outliers may
be detected by grouping the similar data in the same group,
i.e., in the same cluster.
• Machine Learning : A Machine Learning algorithm can be
executed for smoothing of data. For example, Regression
Algorithm can be used for smoothing of data using a specified
linear function.
• Removing manually: The noisy data can be deleted manually
by the human being, but it is a time-consuming process, so
mostly this method is not given priority.
Missing, Noisy and inconsistent Data
What is Data Preprocessing?
Data Preprocessing

Data Preprocessing is a technique that is used to

convert the raw data into a clean data set.
Data is gathered from different sources it is collected in
raw format which is not feasible for the analysis.

The set of steps is known as Data Preprocessing. It

includes –
• Data Cleaning
• Data Integration
• Data Transformation
• Data Reduction
Why is Data Preparation Important?

• Data Preprocessing is necessary because of the presence of

unformatted real-world data. Mostly real-world data is
composed of –
• Inaccurate data (missing data) – There are many reasons for
missing data such as data is not continuously collected, a
mistake in data entry, technical problems with biometrics and
much more.
• The presence of noisy data (erroneous data and outliers)
– The reasons for the existence of noisy data could be a
technological problem of gadget that gathers data, a human
mistake during data entry and much more.
• Inconsistent data – The presence of inconsistencies are due to
the reasons such that existence of duplication within data,
human data entry, containing mistakes in codes or names, i.e.,
violation of data constraints and much more.
How is Data Preprocessing performed?
How missing data can be handled. Three different steps can be
executed which are given below –
• Ignoring the missing record – It is the simplest and efficient method
for handling the missing data. But, this method should not be
performed at the time when the number of missing values are
immense or when the pattern of data is related to the unrecognized
primary root of the cause of statement problem.
• Filling the missing values manually – This is one of the best-chosen
methods. But there is one limitation that when there are large data
set, and missing values are significant then, this approach is not
efficient as it becomes a time-consuming task.
• Filling using computed values – The missing values can also be
occupied by computing mean, mode or median of the observed
given values. Another method could be the predictive values that
are computed by using any Machine Learning or Deep Learning
algorithm.
Tasks of Data Preparation

• Data Cleaning :This is the first step which is implemented in Data

Preprocessing. In this step, the primary focus is on handling missing data, noisy
data, detection, and removal of outliers, minimizing duplication and computed
biases within the data.
• Data Integration :This process is used when data is gathered from various data
sources and data are combined to form consistent data. This consistent data
after performing data cleaning is used for analysis.
• Data Transformation :This step is used to convert the raw data into a specified
format according to the need of the model. The options used for
transformation of data are given below –
• Normalization – In this method, numerical data is converted into the specified
range, i.e., between 0 and 1 so that scaling of data can be performed.
• Aggregation – This method is used to combine the features into one. For
example, combining two categories can be used to form a new group.
• Generalization – In this case, lower level attributes are converted to a higher
standard.
• Data Reduction :After the transformation and scaling of data duplication, i.e.,
redundancy within the data is removed and efficiently organize the data.
What Is Data Wrangling?

• It is used to convert the raw data into the format that is convenient
for the consumption of data. Data Wrangling is a technique that is
executed at the time of making an interactive model.
• Steps:
– extracting the data from different data sources
– sorting of data using certain algorithm is performed
– decompose the data into a different structured format
– finally store the data into another database.

Data is converted to the proper feasible format before applying

any model to it.
By performing filtering, grouping and selecting appropriate data
accuracy and performance of the model could be increased.
 Data Wrangling in Python

Da Data wrangling in python deals with the below functionalities:

1.Data exploration: In this process, the data is studied, analyzed and understood by
visualizing representations of data.
2.Dealing with missing values: Most of the datasets having a vast amount of data
contain missing values of NaN, they are needed to be taken care of by replacing them
with mean, mode, the most frequent value of the column or simply by dropping the
row having a NaN value.
3.Reshaping data: In this process, data is manipulated according to the
requirements, where new data can be added or pre-existing data can be modified.
4.Filtering data: Some times datasets are comprised of unwanted rows or columns
which are required to be removed or filtered
5.Other: After dealing with the raw dataset with the above functionalities we get an
efficient dataset as per our requirements and then it can be used for a required
purpose like data analyzing, machine learning, data visualization, model training etc.
in Python
 Why is Data Wrangling Important?

• Data Wrangling is used to handle the issue

of Data Leakage while implementing Machine
Learning and Deep Learning.
• Data Leakage is responsible for the cause of
invalid Machine Learning/Deep Learning
model due to the over optimization of the
applied model.
How is Data Wrangling performed?

• If one considers the complete data set for

normalization and standardization, then the cross-
validation is performed for the estimation of the
performance of the model leads to the beginning of
data leakage.
• The effect of Data Leakage could be minimized by
recalculating for the required Data Preparation during
the cross-validation process that includes feature
selection, outliers detection, and removal, projection
methods, scaling of selected features and much more.
• Another solution is that dividing the complete dataset
into training data set that is used to train the model
and validation dataset which is used to evaluate the
performance and accuracy of the applied model.
Tasks of Data Wrangling
• Discovering: Firstly, data should be understood thoroughly and examine
which approach will best suit. For example: if have a weather data when we
analyze the data it is observed that data is from one area and so primary
focus is on determining patterns.
• Structuring :As the data is gathered from different sources, the data will be
present in various shapes and sizes. Therefore, there is a need for
structuring the data in proper format.
• Cleaning :Cleaning or removing of data should be performed that can
degrade the performance of analysis.
• Enrichment :Extract new features or data from the given data set to
optimize the performance of the applied model.
• Validating: This approach is used for improving the quality of data and
consistency rules so that transformations that are applied to the data could
be verified.
• Publishing :After completing the steps of Data Wrangling, the steps can be
documented so that similar steps can be performed for the same kind of
data to save time.
Data Leakage
Data Leakage can be demonstrated in many ways that are
given below –

• The Leakage of data from test dataset to training data set.

• Leakage of computed correct prediction to the training
dataset.
• Leakage of future data into the past data.
• Usage of data outside the scope of the applied algorithm
• In general, the leakage of data is observed from two
primary sources of Machine Learning/Deep Learning
algorithms such as feature attributes (variables) and
training data set.
• Checking the presence of Data Leakage within the
applied model
Minimizing Data Leakage