HDPV2 - Services Review
A review of all Services RunningThis blog discusses all the services / java process running / configured till now as part of Hadoop V2 Configuration.First let me list out all...
Wednesday, April 25, 2018
Hadoop V2 - Fair Scheduler Configuration
In this blog I discuss on how to configure Fair Scheduler. Fair Scheduler is also one of the scheduler used in production environments.In my words it is more fairer than...
Monday, April 23, 2018
Hadoop V2 - Capacity Scheduler Configuration
In this blog I discuss how to do fair scheduler configuration for Hadoop 2 I will design Queues and Capacity as per below diagram. (All the detailed configuration is present...
Hadoop V2 - Sqoop Import from Oracle Database 12c
In this blog I will demonstrate how to import data using sqoop from Oracle to HDFSIf you have followed my last blog, you have your sqoop installation ready. Step 1...
Friday, April 20, 2018
Hadoop V2 - Sqoop - NoSuchMethodError
Symptoms - Sqoop job fails when importing with avro type format. Container logs show org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V Solution - Run the job by adding arguements to...
Thursday, April 19, 2018
Hadoop V2 - Exception from container-launch
In this blog I will discuss how to find out issues with container launch when you are running your jobs / applications. Looking at below error trace from a MR...
Hadoop V2 - Sqoop Install
In this blog I discuss Sqoop deployment, Sqoop stands for SQL to Hadoop. SQL is a tool which can import / export data from RDBMSSqoop - Comes bundled with special...
Wednesday, April 18, 2018
Hadoop V2 - QJM (Automatic)
In this blog I discuss my configuration of Automatic Failover using QJM.This is in continuation with my previous QJM blog for manual configuration. Automatic failover is configured using ZKFC -...
Hadoop V2 - HAAdmin
In this blog I discuss usage of haadmin command. I have already setup Manual HA configuration using QJM in last blog. haadmin command is supported for failing over , switching...
Hadoop V2 - QJM (Manual Configuration)
In this blog I discuss setting up Namenode High availability using QJM (Quorum Journal Manager)Functioning of QJM1. 3 (or 5 or odd number) nodes running QJM2. NN writes Edit logs...
Hadoop V2 - GetConf
In this short blog I will discuss on getconf class of hdfs. This class lists the configuration as listed in the configuration fileIt can be used to get details on...
Hadoop V2 - Safe Mode
In this blog I will discuss about Safe Mode in NamenodeSafe Mode is a special mode of Hadoop which is read only mode (No client connections still) and no changes...
Hadoop V2 - Namenode Parameters
This blog covers details of Namenode and Namenode related parametersCheckpoint Frequency Checkpointing can be configured by setting - dfs.namenode.checkpoint.period. This parameter controls the time between 2 checkpoints. - dfs.namenode.checkpoint.txns. This...
Tuesday, April 17, 2018
Hadoop V2 - HttpFS setup
In this blog I discuss setup of HttpFS in HadoopIN hadoop HttpFS1. Acts as a proxy server for catering to REST requests2. Acts as single point of contact for all...
Monday, April 16, 2018
Hadoop V2 - WebHdfs
In this blog I discuss how to setup WebHDFS.In your hdfs-site.xml Setup below property on namenode in hdfs-site.xml<property> <name>dfs.webhdfs.enabled</name> <value>true</value></property>Distribute this on all nodesThis will need bounce of all Datanodes...
Hadoop V2 - FSCK
FSCK is one of the key utilized and monitoring command for monitoring hdfs FSCK1. Similar to Linux fsck, finds out block corruptions and issue with the File System 2. Does...
Hadoop V2 - Snapshot
In this blog I discuss HDFS snapshot feature HDFS Snapshot - 1. Feature to take snapshots of directory to prevent errors2. It is used to query old versions of data3....
Hadoop V2 - Trash
In this blog I discuss about HDFS Trash Feature.Trash is a feature provided by HDFS similar to recycle bin of Windows. However there are differences / few changes 1. It...
