Scribd
Upload
167 views

SQLSat374 - ETL On Cloud - Azure Data Factory - Reza Rad

This document discusses using Azure Data Factory for ETL on the cloud. It provides an overview of Azure Data Factory, including what it is, the tools required to use it, its main components, and monitoring capabilities. Azure Data Factory allows composing data storage, processing, and movement services into scalable and reliable data pipelines in the cloud. It can access various data sources and load data to destinations while performing transformations. The document demonstrates Azure Data Factory concepts like linked services, datasets, pipelines, and activities through a log processing example.

Uploaded by

Sudha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
167 views

SQLSat374 - ETL On Cloud - Azure Data Factory - Reza Rad

This document discusses using Azure Data Factory for ETL on the cloud. It provides an overview of Azure Data Factory, including what it is, the tools required to use it, its main components, and monitoring capabilities. Azure Data Factory allows composing data storage, processing, and movement services into scalable and reliable data pipelines in the cloud. It can access various data sources and load data to destinations while performing transformations. The document demonstrates Azure Data Factory concepts like linked services, datasets, pipelines, and activities through a log processing example.

Uploaded by

Sudha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 46

ETL on Cloud?

Azure Data Factory

Reza Rad
RADACAD
Our Sponsors
About Me: Reza Rad

DW/BI Architect, Consultant


SQL Server MVP
Author of SQL Server and BI books
Author of MSBI webcast series on RADACAD
Microsoft Certified Trainer
Microsoft Certified Professional

@Rad_Reza [email protected]

www.rad.pasfu.com www.linkedin.com/in/rezarad
Agenda

Azure 101
What is Azure Data Factory?
Required Tools for Azure Data Factory
Components of Azure Data Factory
Monitoring
Microsoft Azure

 Microsoft Cloud Computing Platform


 Hosted by Microsoft
 Lower Maintenance Efforts
 Pay as you go
 Easy Scale Out
Microsoft Azure Portal
Microsoft Azure Portal - Preview
Few of Services That You Might Use

Azure VM Azure Website SQL Azure Azure Storage

Azure Machine Visual Studio


HDInsight
Learning Online
Azure Data Factory

 Fully Managed Service for Composing Data


Storages, Processing, and Movement Services
into Streamlined, Scalable, and Reliable Data
Production Pipelines.
What You can do with Azure Data Factory
 Access to data sources
 such as SQL Server On premises, SQL Azure, and Azure Blob storage
 Data transformation
 through Hive, Pig, and C#.
 Monitoring
 the pipeline of data, validation and execution of scheduled jobs
 Load it into desired Destinations
 such as SQL Server On premises, SQL Azure, and Azure Blob storage
 And last but not least; This is Cloud based service.
Evolving Approaches to Analytics

Extrac Transfor Load


t m

ETL Tool Transforme EDW BI Tools


Original Data (SSIS, etc) d Data (SQL Svr, Teradata, etc)

Data Marts

Data Lake(s)

Dashboards

Apps
Evolving Approaches to Analytics

Extrac Transfor Load


t m

ETL Tool Transforme EDW BI Tools


Original Data (SSIS, etc) d Data (SQL Svr, Teradata, etc)

Data Marts

Data Lake(s)

Dashboards

Ingest Apps
(EL)
Original Data
Evolving Approaches to Analytics

Extrac Transfor Load


t m

ETL Tool Transforme EDW BI Tools


Original Data (SSIS, etc) d Data (SQL Svr, Teradata, etc)

Data Marts

Data Lake(s)

Dashboards
Scale-out
Ingest Apps
(EL) Storage &
Original Data
Compute
(HDFS, Blob
Storage, etc)

Streaming data

Transform & Load


Azure Data Factory Architecture
Administration Panel
Demo

Azure Portal
Azure Subscription
Data Management Gateway

 To Connect to On Premises SQL Server


Azure Power Shell
SSMS or SQL Azure Console
Azure Storage Explorer (Optional)
What Else? JSON!

 JSON used to define metadata in ADF


Demo

Azure Data Factory


Linked Services

 Data Store
 Data Management Gateway
Data Store

 Azure Storage
 SQL Azure
 SQL Server On-Premises (using Data
Management Gateway)
 HDInsight Compute
Datasets

 Input and output tables for ADF


 Structure can be defined with
JSON
 JSON files can be loaded to ADF
with Azure Power Shell
Dataset

 Structure
 Location
 Availability
Pipeline

 Flow and pipeline of the data


 Needs data sets to work with
 Structured with JSON
 JSON file will be loaded into ADF with Azure
Power Shell
Pipeline

 Activities
 Policy
Activity

 Copy Activity
 Hive or Pig Activity
 Custom C# Activity
Activity Properties
Source and Sink Types
Diagram View
Monitoring Executions
Data Slice
Demo

Log Processing Azure Data Factory


Summary

Azure 101
What is Azure Data Factory?
Required Tools for Azure Data Factory
Components of Azure Data Factory
Monitoring
Reference To Study

 Azure Data Factory Documentation


http://azure.microsoft.com/en-
us/documentation/services/data-factory/
Questions?

@Rad_Reza [email protected]

www.rad.pasfu.com www.linkedin.com/in/rezarad
Thank You!

You might also like