INTRODUCTION HADOOP

 Introduction to Big Data

Big Data Characteristics:-

V - volume

V - velocity ( frequency or volume of data in real time or in particular time)

V - variety (types of data from structured, semi-structured to unstructured)

V- veracity ( reliability, credibility, data accuracy and data integrity) 

V - value  (value to given by the data, purity etc....)


data which satisfies the above characteristics called big data


Introduction to Hadoop:-

    Hadoop is one for the popular open source framework to handle and process the big data using Batch processing.


Major Hadoop Components:-

1. HDFS (mandatory)

2. Map reduce (Mandatory)

3, Hive

4. Pige

5. Sqoop

6. flume

7. Oziee

8. Mahout

9. H-Base

Comments

Post a Comment

Popular posts from this blog

HDFS QUOTAS

Image
 There  are two quotas in hdfs,  Space Quota - will set maximum space for directory normal Quota - will no of file can stored in directory Hadoop HDFS Space Quota cmd: to set sapce quota : hadoop dfsadmin -setSpaceQuota <quota-size> /<foldername> or hdfs dfsadmin -setSpaceQuota <quota-size> <directory-path> to clear space quota : hadoop dfs -clrSpaceQuota /<foldername> practical example :- In below example, i'm trying to move LICENSE.txt to /mr_output folder after space quota set. mr_output already contains 2 files use : hadoop fs -count -q -h -v /<foldername> to check quota information for that directory. In below,  i set space quota to 500B(byte) and tried to move license file to folder. It throwed quota error. Then i cleared space quota  Hadoop HDFS Quota cmds: to set quota : hadoop dfsadmin -setQuota <file-count> /<foldername> or hdfs dfsadmin -setQuota <file-count> <directory-path> to clear : ha...
Read more

Hadoop Single Node installation Guide Ubuntu

Hadoop-2 Single Node Cluster Creation Installation ----------------------------------------------------------------- 1. Download Hadoop and Java tar -zxvf hadoop-2.9.1.tar.gz (Extract the tar file) tar -zxvf jdk-8u45-linux-x64.tar (Extract the tar file) sudo apt-get install vim (Install USER Friendly Editer) vi .bashrc (Set the java Path in your Home Path) export JAVA_HOME=/home/username/jdk1.8.0_45 export PATH=HOME/bin:JAVA_HOME/bin:PATH source .bashrc (Execute the bashrc file) echo JAVA_HOME (Check the java path) ========================================================================================================================================= 2. Modify Hadoop Configuration Files NAMENODE ----> core-site.xml RESOURCE MANGER ----> mapperd-site.xml SECONDARYNAMENODE ----> DATANODE ----> slaves NODEMANGER ----> slaves & yarn-site.xml vi etc/hadoop/core-site.xml <property> <name>fs.default.name</name> <value>h...
Read more