HDFS QUOTAS

 There  are two quotas in hdfs,

  1.  Space Quota - will set maximum space for directory

  2. normal Quota - will no of file can stored in directory

Hadoop HDFS Space Quota cmd:

to set sapce quota : hadoop dfsadmin -setSpaceQuota <quota-size> /<foldername> or hdfs dfsadmin -setSpaceQuota <quota-size> <directory-path>

to clear space quota : hadoop dfs -clrSpaceQuota /<foldername>

practical example :-
In below example, i'm trying to move LICENSE.txt to /mr_output folder after space quota set. mr_output already contains 2 files


use : hadoop fs -count -q -h -v /<foldername>
to check quota information for that directory.

In below, 
i set space quota to 500B(byte) and tried to move license file to folder.
It throwed quota error.
Then i cleared space quota 




Hadoop HDFS Quota cmds:

to set quota : hadoop dfsadmin -setQuota <file-count> /<foldername> or hdfs dfsadmin -setQuota <file-count> <directory-path>
to clear : hadoop dfs -clrSpaceQuota /<foldername>

example:


in above, initial file count of mr_output was 2 + 1 for mr_output . so, total 3. i set quota to 4 and moved one file to mr_output. quota full. again tried then got quota error.


cmd to check quota
hadoop fs -count -q -h -v /<foldername>

fs cmds
move file : hadoop fs -put <frompathwithfilename> <topathwithfilename> 
list files and folders  : hadoop fs -ls / or /<foldername>


when do update on Hive table,
will create new file with updated data and update filename in meta mapping. but old file still there in hdfs. to clear that  use hive compaction. we should not use frequently but in specific interval to remove old files.

Comments

Post a Comment

Popular posts from this blog

INTRODUCTION HADOOP

 Introduction to Big Data Big Data Characteristics:- V - volume V - velocity ( frequency or volume of data in real time or in particular time) V - variety (types of data from structured, semi-structured to unstructured) V- veracity ( reliability, credibility, data accuracy and data integrity)  V - value  (value to given by the data, purity etc....) data which satisfies the above characteristics called big data Introduction to Hadoop:-     Hadoop is one for the popular open source framework to handle and process the big data using Batch processing. Major Hadoop Components:- 1. HDFS (mandatory) 2. Map reduce (Mandatory) 3, Hive 4. Pige 5. Sqoop 6. flume 7. Oziee 8. Mahout 9. H-Base
Read more

Hadoop Single Node installation Guide Ubuntu

Hadoop-2 Single Node Cluster Creation Installation ----------------------------------------------------------------- 1. Download Hadoop and Java tar -zxvf hadoop-2.9.1.tar.gz (Extract the tar file) tar -zxvf jdk-8u45-linux-x64.tar (Extract the tar file) sudo apt-get install vim (Install USER Friendly Editer) vi .bashrc (Set the java Path in your Home Path) export JAVA_HOME=/home/username/jdk1.8.0_45 export PATH=HOME/bin:JAVA_HOME/bin:PATH source .bashrc (Execute the bashrc file) echo JAVA_HOME (Check the java path) ========================================================================================================================================= 2. Modify Hadoop Configuration Files NAMENODE ----> core-site.xml RESOURCE MANGER ----> mapperd-site.xml SECONDARYNAMENODE ----> DATANODE ----> slaves NODEMANGER ----> slaves & yarn-site.xml vi etc/hadoop/core-site.xml <property> <name>fs.default.name</name> <value>h...
Read more