THE HADOOP ECOSYSTEM

 There  are two quotas in hdfs,  Space Quota - will set maximum space for directory normal Quota - will no of file can stored in directory Hadoop HDFS Space Quota cmd: to set sapce quota : hadoop dfsadmin -setSpaceQuota <quota-size> /<foldername> or hdfs dfsadmin -setSpaceQuota <quota-size> <directory-path> to clear space quota : hadoop dfs -clrSpaceQuota /<foldername> practical example :- In below example, i'm trying to move LICENSE.txt to /mr_output folder after space quota set. mr_output already contains 2 files use : hadoop fs -count -q -h -v /<foldername> to check quota information for that directory. In below,  i set space quota to 500B(byte) and tried to move license file to folder. It throwed quota error. Then i cleared space quota  Hadoop HDFS Quota cmds: to set quota : hadoop dfsadmin -setQuota <file-count> /<foldername> or hdfs dfsadmin -setQuota <file-count> <directory-path> to clear : ha...

Hadoop-2 Single Node Cluster Creation Installation ----------------------------------------------------------------- 1. Download Hadoop and Java tar -zxvf hadoop-2.9.1.tar.gz (Extract the tar file) tar -zxvf jdk-8u45-linux-x64.tar (Extract the tar file) sudo apt-get install vim (Install USER Friendly Editer) vi .bashrc (Set the java Path in your Home Path) export JAVA_HOME=/home/username/jdk1.8.0_45 export PATH=HOME/bin:JAVA_HOME/bin:PATH source .bashrc (Execute the bashrc file) echo JAVA_HOME (Check the java path) ========================================================================================================================================= 2. Modify Hadoop Configuration Files NAMENODE ----> core-site.xml RESOURCE MANGER ----> mapperd-site.xml SECONDARYNAMENODE ----> DATANODE ----> slaves NODEMANGER ----> slaves & yarn-site.xml vi etc/hadoop/core-site.xml <property> <name>fs.default.name</name> <value>h...

 Introduction to Big Data Big Data Characteristics:- V - volume V - velocity ( frequency or volume of data in real time or in particular time) V - variety (types of data from structured, semi-structured to unstructured) V- veracity ( reliability, credibility, data accuracy and data integrity)  V - value  (value to given by the data, purity etc....) data which satisfies the above characteristics called big data Introduction to Hadoop:-     Hadoop is one for the popular open source framework to handle and process the big data using Batch processing. Major Hadoop Components:- 1. HDFS (mandatory) 2. Map reduce (Mandatory) 3, Hive 4. Pige 5. Sqoop 6. flume 7. Oziee 8. Mahout 9. H-Base