HDPV2 - Decomissioning a Node

In this blog I am going to discuss how to decommission a node in HDFS
Decommissioning is a graceful process in which blocks get replicated to other nodes.


Step 1
Following Properties must be set on your HDFS
cd /etc/hadoop/conf
hdfs-site.xml
<property>
        <name>dfs.hosts.exclude</name>
        <value>/etc/hadoop/conf/dfs.hosts.exclude</value>
</property>

<property>
        <name>dfs.hosts</name>
        <value>/etc/hadoop/conf/dfs.hosts.include</value>
</property>

yarn-site.xml
<property>
        <name>yarn.resourcemanager.node.include-path</name>
        <value>/etc/hadoop/conf/dfs.hosts.include</value>
</property>
<property>
        <name>yarn.resourcemanager.node.exclude-path</name>
        <value>/etc/hadoop/conf/dfs.hosts.exclude</value>
</property>


Step 2
[As root on nn]

Remove entry of new node from slaves and include file and add it to the exclude file
(I am removing d3n)

cd /etc/hadoop/conf
cat slaves
d1n
d2n
d4n

[As root on rm]
cd /etc/spark/conf
cat slaves
d1n
d2n
d3n

Step 3
[As root on nn]
Distribute slaves, dfs.hosts.include and dfs.hosts.exclude to nn, snn and rm

Step 4
[As hdfs on nn]
hdfs dfsadmin –refreshNodes

Step 5
[As yarn on rm]
yarn rmadmin -refreshNodes


Step 6
[As hdfs on nn]
Verify using
hdfs dfsadmin -report

Step 7
Stop all daemons.

Decomission the node once the output of hdfs dfsadmin -report no longer shows the name of the node.

[As hdfs on d3n]
hadoop-daemon.sh stop datanode

[As yarn on d3n]
yarn-daemon.sh stop nodemanager

[As spark on d3]
stop-slave.sh