In this blog we are going to have Pre-req done for Hadoop V1 Cluster
The Pre-req are pretty much same for Hadoop V1 and Hadoop V2 we will see what changes in the later blogs.
Below is my /etc/hosts for HDP V1 Cluster. The names are pretty apparent talking about their roles and functionality. nn,snn are namenode and secondary namenode and d1n-d4n are datanodes.
Our Controller Node is namenode or nn or namenode.cluster.com
We start by putting below in the /etc/hosts file of namenode
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.51 namenode.cluster.com namenode nn
192.168.10.53 snamenode.cluster.com snamenode snn
192.168.10.54 d1node.cluster.com d1node d1n
192.168.10.55 d2node.cluster.com d2node d2n
192.168.10.58 d3node.cluster.com d3node d3n
192.168.10.57 d4node.cluster.com d4node d4n
Step 1 - Download Hadoop V1 (1.2.1)
wget http://www-us.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
--2018-02-20 06:22:53-- http://www-us.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
Connecting to 172.20.24.50:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 63851630 (61M) [application/x-gzip]
Saving to: ‘hadoop-1.2.1.tar.gz’
100%[==========================================================================>] 63,851,630 369KB/s in 3m 6s
2018-02-20 06:25:59 (335 KB/s) - ‘hadoop-1.2.1.tar.gz’ saved [63851630/63851630]
Step 2
Setup ssh equivalency between root user of all nodes (You can delete this later if you want)
This will help us to set-up everything as root user.
[As root]
./sshUserSetup.sh -user root -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced
Give the passwords once prompted - End of the script you should have passwordless connectivity between root users on all nodes.
The Script sshUserSetup is something you will be knowing if you are an OracleDBA
If not you can do this manually or provide your email below, i'll send it over.
All what script does is create ssh equivalency between all the nodes given.
Step 3
[As root - Managed Hosts file creation]
Create hosts file , each host in the file is in a line in each line
cd /tmp
chmod 777 hosts
cat hosts
nn
snn
d1n
d2n
d3n
d4n
Step 4
[As root - Hosts File Updation]
##for i in $(cat hosts) ; do scp /etc/hosts ${i}:/etc/hosts; done
Step 5
[As root - Group Creation]
## for i in $(cat hosts) ; do echo "ssh ${i} groupadd -g 1000 hadoop" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd
Step 6
[As root - User Creation]
## for i in $(cat hosts) ; do echo "ssh ${i} useradd -u 1002 -g hadoop hduser" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd
for i in $(cat hosts) ; do echo "ssh ${i} useradd -u 1003 -g hadoop mapred" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd
Step 7
[As root - hduser and mapred user password change ]
Change hduser to mapred for second execution.
Create Script as below and run it
#!/bin/bash
for server in `cat hosts`; do
echo $server;
ssh ${server} 'passwd hduser <<EOF
hadoop
hadoop
EOF';
done
Step 8
[Hadoop User Equivalency] [As hduser and mapred - on namenode]
This will create ssh equivalency for hduser and mapred
#./sshUserSetup.sh -user hduser -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced
#./sshUserSetup.sh -user mapred -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced
Step 9
[As root - Java Installation]
#for i in $(cat hosts) ; do echo "scp jdk-8u152-linux-x64.rpm ${i}:/tmp &" >> /tmp/sendjdk.bash ; done
Paste and run contents of the file
Step 10
[As root - Install Java]
# for i in $(cat hosts) ; do ssh ${i} rpm -Uvh /tmp/jdk-8u152-linux-x64.rpm ; done;
Step 11
[AS root - Verify Changes done till now]
Java
# for i in $(cat hosts) ; do ssh ${i} java -version ; done;
[Other Automated Helper Scripts]
[Partition Creation]
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdc' ; done;
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "n\np\n1\n\n\nt\n8e\nw" | fdisk /dev/sdd' ; done;
[For deleting the partitions if required]
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "d\nw" | fdisk /dev/sdd' ; done;
[File System formatting]
# for i in $(cat hosts) ; do ssh ${i} mkfs.ext4 /dev/sdc1 ; done;
[Directory Creation]
# for i in $(cat hosts) ; do ssh ${i} mkdir /opt/HDV1; done;
[Ownership Change]
# for i in $(cat hosts) ; do ssh ${i} chmod 777 /opt ; done;
# for i in $(cat hosts) ; do ssh ${i} chown hduser:hadoop /opt/hadoop ; done;
[FSTAB Entry Addition]
# for i in $(cat hosts) ; do ssh ${i} echo "/dev/sdc1 /opt/HDPV1 ext4 defaults 1 2 >> /etc/fstab" ; done
[Mounting the file system]
#for i in $(cat hosts) ; do ssh ${i} mount /opt/HDPV1 ; done
[VG and FS Extension]
for i in $(cat hosts) ; do ssh ${i} pvcreate /dev/sdd1 ; done;
for i in $(cat hosts) ; do ssh ${i} vgextend rootvg /dev/sdd1 ; done;
for i in $(cat hosts) ; do ssh ${i} lvextend /dev/mapper/rootvg-root_lv -L 15G ; done;
for i in $(cat hosts) ; do ssh ${i} resize2fs /dev/mapper/rootvg-root_lv ; done;
for i in $(cat hosts) ; do ssh ${i} lvextend /dev/mapper/rootvg-tmp_lv -L 8G ; done;
for i in $(cat hosts) ; do ssh ${i} resize2fs /dev/mapper/rootvg-tmp_lv ; done;
The Pre-req are pretty much same for Hadoop V1 and Hadoop V2 we will see what changes in the later blogs.
Below is my /etc/hosts for HDP V1 Cluster. The names are pretty apparent talking about their roles and functionality. nn,snn are namenode and secondary namenode and d1n-d4n are datanodes.
Our Controller Node is namenode or nn or namenode.cluster.com
We start by putting below in the /etc/hosts file of namenode
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.51 namenode.cluster.com namenode nn
192.168.10.53 snamenode.cluster.com snamenode snn
192.168.10.54 d1node.cluster.com d1node d1n
192.168.10.55 d2node.cluster.com d2node d2n
192.168.10.58 d3node.cluster.com d3node d3n
192.168.10.57 d4node.cluster.com d4node d4n
Step 1 - Download Hadoop V1 (1.2.1)
wget http://www-us.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
--2018-02-20 06:22:53-- http://www-us.apache.org/dist/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz
Connecting to 172.20.24.50:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 63851630 (61M) [application/x-gzip]
Saving to: ‘hadoop-1.2.1.tar.gz’
100%[==========================================================================>] 63,851,630 369KB/s in 3m 6s
2018-02-20 06:25:59 (335 KB/s) - ‘hadoop-1.2.1.tar.gz’ saved [63851630/63851630]
Step 2
Setup ssh equivalency between root user of all nodes (You can delete this later if you want)
This will help us to set-up everything as root user.
[As root]
./sshUserSetup.sh -user root -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced
Give the passwords once prompted - End of the script you should have passwordless connectivity between root users on all nodes.
The Script sshUserSetup is something you will be knowing if you are an OracleDBA
If not you can do this manually or provide your email below, i'll send it over.
All what script does is create ssh equivalency between all the nodes given.
Step 3
[As root - Managed Hosts file creation]
Create hosts file , each host in the file is in a line in each line
cd /tmp
chmod 777 hosts
cat hosts
nn
snn
d1n
d2n
d3n
d4n
Step 4
[As root - Hosts File Updation]
##for i in $(cat hosts) ; do scp /etc/hosts ${i}:/etc/hosts; done
Step 5
[As root - Group Creation]
## for i in $(cat hosts) ; do echo "ssh ${i} groupadd -g 1000 hadoop" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd
Step 6
[As root - User Creation]
## for i in $(cat hosts) ; do echo "ssh ${i} useradd -u 1002 -g hadoop hduser" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd
for i in $(cat hosts) ; do echo "ssh ${i} useradd -u 1003 -g hadoop mapred" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd
Step 7
[As root - hduser and mapred user password change ]
Change hduser to mapred for second execution.
Create Script as below and run it
#!/bin/bash
for server in `cat hosts`; do
echo $server;
ssh ${server} 'passwd hduser <<EOF
hadoop
hadoop
EOF';
done
Step 8
[Hadoop User Equivalency] [As hduser and mapred - on namenode]
This will create ssh equivalency for hduser and mapred
#./sshUserSetup.sh -user hduser -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced
#./sshUserSetup.sh -user mapred -hosts "nn snn d1n d2n d3n d4n" -noPromptPassphrase -confirm -advanced
Step 9
[As root - Java Installation]
#for i in $(cat hosts) ; do echo "scp jdk-8u152-linux-x64.rpm ${i}:/tmp &" >> /tmp/sendjdk.bash ; done
Paste and run contents of the file
Step 10
[As root - Install Java]
# for i in $(cat hosts) ; do ssh ${i} rpm -Uvh /tmp/jdk-8u152-linux-x64.rpm ; done;
Step 11
[AS root - Verify Changes done till now]
Java
# for i in $(cat hosts) ; do ssh ${i} java -version ; done;
[Other Automated Helper Scripts]
[Partition Creation]
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "n\np\n1\n\n\nw" | fdisk /dev/sdc' ; done;
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "n\np\n1\n\n\nt\n8e\nw" | fdisk /dev/sdd' ; done;
[For deleting the partitions if required]
# for i in $(cat hosts) ; do ssh ${i} 'echo -e "d\nw" | fdisk /dev/sdd' ; done;
[File System formatting]
# for i in $(cat hosts) ; do ssh ${i} mkfs.ext4 /dev/sdc1 ; done;
[Directory Creation]
# for i in $(cat hosts) ; do ssh ${i} mkdir /opt/HDV1; done;
[Ownership Change]
# for i in $(cat hosts) ; do ssh ${i} chmod 777 /opt ; done;
# for i in $(cat hosts) ; do ssh ${i} chown hduser:hadoop /opt/hadoop ; done;
[FSTAB Entry Addition]
# for i in $(cat hosts) ; do ssh ${i} echo "/dev/sdc1 /opt/HDPV1 ext4 defaults 1 2 >> /etc/fstab" ; done
[Mounting the file system]
#for i in $(cat hosts) ; do ssh ${i} mount /opt/HDPV1 ; done
[VG and FS Extension]
for i in $(cat hosts) ; do ssh ${i} pvcreate /dev/sdd1 ; done;
for i in $(cat hosts) ; do ssh ${i} vgextend rootvg /dev/sdd1 ; done;
for i in $(cat hosts) ; do ssh ${i} lvextend /dev/mapper/rootvg-root_lv -L 15G ; done;
for i in $(cat hosts) ; do ssh ${i} resize2fs /dev/mapper/rootvg-root_lv ; done;
for i in $(cat hosts) ; do ssh ${i} lvextend /dev/mapper/rootvg-tmp_lv -L 8G ; done;
for i in $(cat hosts) ; do ssh ${i} resize2fs /dev/mapper/rootvg-tmp_lv ; done;
Too good article,thank you for sharing this awesome blog with us.
ReplyDeletekeep sharing...
big data hadoop training
hadoop admin online training