Scribd
Upload
19 views

Bda 11

Uploaded by

madhkarshingadel4545
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
19 views

Bda 11

Uploaded by

madhkarshingadel4545
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 10

DEPARTMENTOFARTIFICIALINTELLIGENCE

&
DATASCIENCE

Subject: Big Data Analytics Course Code: CSL702


Semester: 7 Course: AI&DS
Laboratory no.:407 Name of subject teacher: Prof. Gitanjali Korgaonkar
Name of student: Sakshi Gholap Roll no: VU2F2122021

Experiment No. 11

Title: Use of Sqoop tool to transfer data between Hadoop and relational database servers.
a) Sqoop – Installation

b) To execute basic commands of Hadoop eco system component Sqoop.

Objectives:
To gain hands-on experience in various aspects of Sqoop tool ,installation and commands.

Outcomes: Students are able to perform Sqoop tool.

Theory:
 What is Sqoop?
Apache Sqoop is a tool designed for efficiently transferring bulk data between
Apache Hadoop and external datastores such as relational databases, enterprise data
warehouses.

Sqoop is used to import data from external datastores into Hadoop Distributed File System or
related Hadoop eco-systems like Hive and HBase. Similarly, Sqoop can also be used to
extract data from Hadoop or its eco-systems and export it to external datastores such as
relational databases, enterprise data warehouses.

Sqoop works with relational databases such as Teradata, Netezza, Oracle, MySQL, Postgres
etc.
Why is Sqoop used?
For Hadoop developers, the interesting work starts after data is loaded into HDFS.

Developers play around the data in order to find the magical insights concealed in that Big
Data. For this, the data residing in the relational database management systems need to be
transferred to HDFS, play around the data and might need to transfer back to relational
database management systems.

Sqoop automates most of the process, depends on the database to describe the schema of the
data to be imported.
Sqoop uses MapReduce framework to import and export the data, which provides parallel
mechanism as well as fault tolerance. Sqoop makes developers life easy by providing
command line interface. Developers just need to provide basic information like source,
destination and database authentication details in the sqoop command. Sqoop takes care of
remaining part.

The scoop architecture is as follows :


Implementation:
Step 1- Sqoop Installation
However, to extract the Sqoop tarball and move it to ―/usr/lib/sqoopǁ directory we use the
following command.
$tar -xvf sqoop-1.4.4.bin
$ su password:
hadoop-2.0.4-alpha.tar.gz #
mv sqoop-1.4.4.bin #exit
hadoop-2.0.4-alpha /usr/lib/sqoop

Step 2- Configuring bashrc


Also, by appending the following lines to ~/.bashrc file we have to set up the Sqoop
environment #Sqoop
export SQOOP_HOME=/usr/lib/sqoop export PATH=$PATH:$SQOOP_HOME/bin
Now, to execute ~/.bashrc
$ source ~/.bashrc

Step 3: Configuring Sqoop


$ cd $SQOOP_HOME/conf
$ mv sqoop-env-template.sh sqoop-env.sh
HADOOP_COMMON_HOME=/usr/local/hadoop
HADOOP_MAPRED_HOME=/usr/local/hadoop

Step 4: Download and Configure MySQL-connector-java


From the following link, we can download MySQL-connector-java-5.1.30.tar.gz file.
In adddition, to extract MySQL-connector-java tarball and move MySQL-connector-java-
5.1.30- bin.jar to /usr/lib/sqoop/lib directory we use the following command.
$ tar -zxf mysql-connector-java-5.1.30.tar.gz
$ su password: # cd mysql-connector-java-5.1.30
# mv mysql-connector-java-5.1.30-bin.jar /usr/lib/sqoop/lib

Step 5: Verifying Sqoop


$ cd $SQOOP_HOME/bin
$ sqoop-version
Expected output
14/12/17 14:52:32 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
Sqoop 1.4.5 git commit id
5b34accaca7de251fc91161733f906af2eddbe83

Hence, in this way Sqoop installation is complete.

Basic Commands:
Sqoop import command imports a table from an RDBMS to HDFS. Each record from a table is
considered as a separate record in HDFS. Records can be stored as text files, or in
binary representation as Avro or SequenceFiles.
Generic Syntax:
$ sqoop import (generic args) (import args)

2. Importing a Table into HDFS


Syntax: $ sqoop import --connect --table --username --password --target-dir
--connect Takes JDBC url and connects to database
--table Source table name to be imported
--username Username to connect to database
--password Password of the connecting user
--target-dir Imports data to the specified directory

3. Importing Selected Data from Table


Syntax:
$ sqoop import --connect --table --username --password --columns --where
--columns Selects subset of columns
--where Retrieves the data which satisfies the condition

4. Importing Data from Query


Syntax:
$ sqoop import --connect --table --username --password --query
--query Executes the SQL query provided and imports the results

5. Incremental Exports
Syntax:
$ sqoop import --connect --table --username --password --incremental --check-column --
last- value Sqoop import supports two types of incremental imports:
1. Append
2. Lastmodified.

6. Sqoop-Eval
Sqoop-eval command allows users to quickly run simple SQL queries against a database
and the results are printed on to the console. Generic Syntax:
$ sqoop eval (generic args) (eval args)

7. Sqoop-List-Database
Used to list all the database available on RDBMS server. Generic Syntax:
$ sqoop list-databases (generic args) (list databases args)
$ sqoop-list-databases (generic args) (list databases args)
Syntax:
$ sqoop list-databases –connect

8. Sqoop-List-Tables
Used to list all the tables in a specified database. Generic Syntax:
$ sqoop list-tables (generic args) (list tables args)
$ sqoop-list-tables (generic args) (list tables args)
Syntax:
$ sqoop list-tables –connect
Conclusion:Thus, Data Visualization using R/HIVE was performed successfully.
R1 R2 R3
DOP DOS Conduction File Record Viva -Voce Total Signature
5 Marks 5 Marks 5 Marks 15 Marks

You might also like