Scribd
Upload
304 views

Hadoop Ecosystem PDF

The document discusses the Hadoop ecosystem. It describes the core Hadoop components of HDFS for storage and MapReduce for processing. It also discusses other common Hadoop tools like HBase, Hive, Pig, Impala, Sqoop, Flume, Hue, and Zookeeper. HDFS stores large files across clusters in a fault-tolerant way. MapReduce allows distributed processing of large datasets in parallel. HBase is a distributed database that provides big table capabilities. Hive, Pig, and Impala allow SQL-like querying of Hadoop data. Sqoop moves data between Hadoop and relational databases. Flume collects streaming data, and Zookeeper provides coordination services.

Uploaded by

Rishabh Gupta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
304 views

Hadoop Ecosystem PDF

The document discusses the Hadoop ecosystem. It describes the core Hadoop components of HDFS for storage and MapReduce for processing. It also discusses other common Hadoop tools like HBase, Hive, Pig, Impala, Sqoop, Flume, Hue, and Zookeeper. HDFS stores large files across clusters in a fault-tolerant way. MapReduce allows distributed processing of large datasets in parallel. HBase is a distributed database that provides big table capabilities. Hive, Pig, and Impala allow SQL-like querying of Hadoop data. Sqoop moves data between Hadoop and relational databases. Flume collects streaming data, and Zookeeper provides coordination services.

Uploaded by

Rishabh Gupta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 55

The Hadoop Ecosystem

Overview
Big Data Challenges
Distributed system and challenges
Hadoop Introduction
History
Who uses Hadoop
The Hadoop Ecosystem
 Hadoop core components
 HDFS
 Map Reduce
 Other Hadoop ecosystem components
 Hbase
 Hive
 Pig
 Impala
 Sqoop
 Flume
 Hue
 Zookeeper
Demo
Big Data Challenges
Solution: Distributed system
Distributed System Challenges

Programming Complexity
Finite bandwidth
Partial failure
The data bottleneck
New Approach to distributed computing

Hadoop:
A scalable fault-tolerant distributed system for data storage and
processing 
Distribute data when the data is stored
Process data where the data is
Data is replicated
Hadoop Introduction

Apache Hadoop is an open-source software framework for storage and large-scale


processing of data-sets on clusters of commodity hardware.
Some of the characteristics:
 Open source
 Distributed processing
 Distributed storage
 Scalable
 Reliable
 Fault-tolerant
 Economical
 Flexible
History

Originally built as a Infrastructure for the “Nutch” project.


Based on Google’s map reduce and google File System.
Created by Doug Cutting in 2005 at Yahoo
Named after his son’s toy yellow elephant.
Who uses Hadoop

http://wiki.apache.org/hadoop/PoweredBy
http://wiki.apache.org/hadoop/
Distributions%20and%20Commercial%20Support
The Hadoop Ecosystem

http://hadoopecosystemtable.github.io/
Hadoop Core Components

HDFS – Hadoop Distributed File System (Storage)


Map Reduce (Processing)
Hadoop Core Components
A multi-node Hadoop cluster
Nodes
NameNode:
Master of the system
Maintains and manages the blocks which are present on the DataNodes

DataNodes:
Slaves which are deployed on each machine and provide the actual storage
Responsible for serving read and write requests for the clients

Jobtracker:
takes care of all the job scheduling and assign tasks to Task Trackers.

TaskTracker:
a node in the cluster that accepts tasks - Map, Reduce and Shuffle operations - from a
jobtracker
HDFS

Hadoop Distributed File System (HDFS) is designed to reliably store very large
files across machines in a large cluster. It is inspired by the GoogleFileSystem.
Distribute large data file into blocks
Blocks are managed by different nodes in the cluster
Each block is replicated on multiple nodes
Name node stored metadata information about files and blocks
Map Reduce
The Mapper:
 Each block is processed in isolation by a map task called mapper
 Map task runs on the node where the block is stored

The Reducer:
 Consolidate result from different mappers
 Produce final output
What makes Hadoop unique

Moving computation to data, instead of moving data to computation.


Simplified programming model: allows user to quickly write and test
Automatic distribution of data and work across machines
Other Hadoop components in Ecosystem
HBase Hadoop database for random read/write access

Hive SQL-like queries and tables on large datasets

Pig Data flow language and compiler


Oozie Workflow for interdependent Hadoop jobs

Sqoop Integration of databases and data warehouses with Hadoop

Flume Configurable streaming data collection

ZooKeeper Coordination service for distributed applications


Hbase

HBase is an open source, non-relational, distributed database modeled after


Google's BigTable.
It runs on top of Hadoop and HDFS, providing BigTable-like capabilities for
Hadoop.
Features of Hbase

Type of NoSql database


Strongly consistent read and write
Automatic sharding
Automatic RegionServer failover
Hadoop/HDFS Integration
HBase supports massively parallelized processing via MapReduce for using
HBase as both source and sink.
 HBase supports an easy to use Java API for programmatic access.
HBase also supports Thrift and REST for non-Java front-ends.
Hbase in CAP theorem

Eric Brewer’s CAP theorem, HBase is a CP type system.


When to use Hbase

When there is real big data: millions or billions of rows, in other way data can not
store in a single node.
When random read/write access to big data
When require to do thousands of operations on big data
When there is no need of extra features of RDMS like typed columns, secondary
indexes, transactions, advanced query languages, etc.
When there is enough hardware.
Difference between Hbase and HDFS

HDFS Hbase
Good for storing large file Built on top of HDFS. Good for hosting
very large tables like billions of rows X
millions of column
Write once. Append to files in some of Read/write many
recent versions but not commonly used
No random read/write Random read/write
No individual record lookup rather read Fast records lookup(update)
all data
Hive

An sql like interface to Hadoop.


Data warehouse infrastructure built on top of Hadoop
Provide data summarization, query and analysis
Query execution via MapReduce
Hive interpreter convert the query to Map reduce format.
Open source project.
Developed by Facebook
Also used by Netflix, Cnet, Digg, eHarmony etc.
Hive

HiveQL example:

SELECT customerId, max(total_cost) from hive_purchases GROUP BY


customerId HAVING count(*) > 3;
Pig

A scripting platform for processing and analyzing large data sets


Apache Pig allows to write complex MapReduce programs using a simple
scripting language.
High level language: Pig Latin
Pig Latin is data flow language.
Pig translate Pig Latin script into MapReduce to execute within Hadoop.
Open source project
Developed by Yahoo
Pig

Pig Latin example:

A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:


float);
X = FOREACH A GENERATE name,$2;
DUMP X;
Pig and Hive

Both requires compiler to generate Map reduce jobs


Hence high latency queries when used for real time responses to ad-hoc queries
Both are good for batch processing and ETL jobs
Fault tolerant
Impala

Cloudera Impala is a query engine that runs on Apache Hadoop. 


Similar to HiveQL.
Does not use Map reduce
Optimized for low latency queries
Open source apache project
Developed by Cloudera
Much faster than Hive or pig
Comparing Pig, Hive and Impala
Description of Feature Pig Hive Impala

SQL based query No yes yes


language
Schema optional required required

Process data with yes yes no


external scripts

Extensible file format yes yes no


support

Query speed slow slow fast

Accessible via ODBC/ no yes yes


JDBC
Sqoop

Command-line interface for transforming data between relational database and


Hadoop
Support incremental imports
Imports use to populate tables in Hadoop
Exports use to put data from Hadoop into relational database such as SQL
server

Hadoop sqoop RDBMS


How Sqoop works

The dataset being transferred is broken into small blocks.


Map only job is launched.
Individual mapper is responsible for transferring a block of the dataset.
How Sqoop works
Flume

Apache Flume is a distributed, reliable, and available service for


efficiently collecting, aggregating, and moving large amounts of
streaming data into the Hadoop Distributed File System (HDFS). 
How flume works
How flume works

Data flows like:


Agent tier -> Collector tier -> Storage tier
Agent nodes are typically installed on the machines that generate the logs and
are data’s initial point of contact with Flume. They forward data to the next tier
collector nodes , which aggregate the separate data flows and forward them
of 
storage tier .
to the final 
Hue

Graphical front end to the cluster.


Open source web interface.
Makes Hadoop platform (HDFS, Map reduce, oozie, Hive, etc.) easy to use
Hue
Zookeeper

Because coordinating distributed systems is a Zoo.


ZooKeeper is a centralized service for maintaining configuration information,
naming, providing distributed synchronization, and providing group services.
DEMO
Hadoop Installation (CDH ) for windows
Download and install VM player
https://my.vmware.com/web/vmware/free#desktop_end_user_computing/
vmware_player/6_0
Hadoop Installation (CDH ) for windows

Make sure you have enabled


virtualization in bios
Hadoop Installation (CDH ) for windows
Download “Quick start VM with CDH” : Download for VMWare
http://www.cloudera.com/content/cloudera/en/downloads/quickstart_vms/
cdh-4-7-x.html
Hadoop Installation (CDH ) for windows
Unzip “cloudera-quickstart-vm-4.7.0-0-vmware”
Open CDH using VMPlayer:
 Open VM Player
 Click open a virtual machine
 Select the file “cloudera-quickstart-vm-4.7.0-0-vmware” in the extracted directory of
“cloudera-quickstart-vm-4.7.0-0-vmware”. Virtual machine will be added to your VM
player.
 Select this virtual machine and click play virtual machine.
References

http://training.cloudera.com/essentials.pdf
http://en.wikipedia.org/wiki/Apache_Hadoop
http://practicalanalytics.wordpress.com/2011/11/06/explaining-hadoop-to-
management-whats-the-big-data-deal/
https://developer.yahoo.com/hadoop/tutorial/module1.html
http://hadoop.apache.org/
http://wiki.apache.org/hadoop/FrontPage
Questions?
Thanks

You might also like