OEM 13cR2 - Backup OEM Configuration

OEM 13cR2 - Backup OEM Configuration

OEM Vineet Sachdeva   March 31, 2018

OEM Configuration backup is essential in part of your OEM setup as it is requried during backup and recovery of your OEM. 

It is essential one single command, however
1. You should take backup regularly
2. Take backup after every change / patch you apply to system

The procedure to backup is 

<OMS_HOME>/bin/emctl exportconfig oms [-sysman_pwd <sysman password>]
[-dir <backup dir>] Specify directory to store backup file
[-keep_host] Specify this parameter if the OMS was installed using a virtual hostname (using
ORACLE_HOSTNAME=<virtual_hostname>)

/u01/app/oracle/em13cr2/middleware/bin/emctl exportconfig oms -sysman_pwd Oracle123 -dir /u01/app/oracle/em_backup/

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.

ExportConfig started...

Backup directory is /u01/app/oracle/em_backup

Machine is Admin Server host. Performing Admin Server backup...

Exporting emoms properties...

Exporting secure properties...

Export has determined that the OMS is not fronted

by an SLB. The local hostname was NOT exported.

The exported data can be imported on any host but

resecure of all agents will be required. Please

see the EM Advanced Configuration Guide for more

details.

Exporting configuration for pluggable modules...

Preparing archive file...

Backup has been written to file: /u01/app/oracle/em_backup/opf_ADMIN_20183113_050349.bka

The export file contains sensitive data.

 You must keep it secure.

ExportConfig completed successfully!

Hadoop V2 - Software Setup and Environment Configuration

Hadoop V2 - Software Setup and Environment Configuration

Hadoop V2 Vineet Sachdeva   March 31, 2018

This in continuation with my previous blog on Hadoop V2 Pre-req Setup

Step 1
[As root -  Namenode - Download  Hadoop Binaries]
Download Hadoop Binaries
wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.7.5/hadoop-2.7.5.tar.gz

Step 2
[As root -  Namenode - Send Hadoop Binaries]
# for i in $(cat /tmp/all_hosts) ; do echo "scp /tmp/hadoop-2.7.5.tar.gz ${i}:/tmp &" >> /tmp/sendhdpv1.bash ; done
bash /tmp/sendhdpv1.bash


Step 3
[As root - Extract Hadoop]
#for i in $(cat /tmp/all_hosts) ;do ssh ${i} tar -xzf /tmp/hadoop-2.7.5.tar.gz -C /usr/local        --overwrite ; done

Step 4
[As root - Setup sudoers configuration]
#for i in $(cat /tmp/all_hosts) ; do ssh ${i}  echo '"hdfs        ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers'; done


Step 5
[As root - Create Conf Directory]
#for i in $(cat /tmp/all_hosts) ;do ssh ${i} mkdir /etc/hadoop; done

[As root - Move the conf directory]
# for i in $(cat /tmp/all_hosts) ;do ssh ${i} mv /usr/local/hadoop-2.7.5/etc/hadoop /etc/hadoop/conf; done

[As root - Give Permissions]
# for i in $(cat hosts) ;do ssh ${i} chmod -R 755 /etc/hadoop ; done;


# for i in $(cat /tmp/all_hosts) ; do ssh ${i} 'mkdir -p /opt/HDPV2/logs  ; echo' ;  done;

# for i in $(cat /tmp/all_hosts) ; do ssh ${i} 'chmod 775  /opt/HDPV2/logs; echo' ;  done;

# for i in  $(cat /tmp/all_hosts) ; do ssh ${i} 'chown hdfs:hadoop  /opt/HDPV2/logs; echo' ;  done;

# for i in $(cat /tmp/all_hosts) ; do ssh ${i} 'mkdir -p /opt/HDPV2/pids ; echo' ;  done;

# for i in $(cat /tmp/all_hosts) ; do ssh ${i} 'chmod 775  /opt/HDPV2/pids; echo' ;  done;

# for i in  $(cat /tmp/all_hosts) ; do ssh ${i} 'chown hdfs:hadoop  /opt/HDPV2/pids; echo' ;  done;

# for i in $(cat /tmp/all_hosts) ; do ssh ${i} 'mkdir -p  /opt/HDPV2/1 /opt/HDPV2/2 /opt/HDPV2/tmp  ; echo' ;  done;

# for i in $(cat /tmp/all_hosts) ; do ssh ${i} 'chmod 775  /opt/HDPV2/1 /opt/HDPV2/2 /opt/HDPV2/tmp ; echo' ;  done;

# for i in  $(cat /tmp/all_hosts) ; do ssh ${i} 'chown hdfs:hadoop  /opt/HDPV2/1 /opt/HDPV2/2 /opt/HDPV2/tmp ; echo' ;  done;



[As root - Create Soft Link to Hadoop]
# for i in $(cat /tmp/all_hosts) ;do ssh ${i} ln -s /usr/local/hadoop-2.7.5 /usr/local/hadoop  ; done


[As root - Create Soft Link to Conf]
# for i in $(cat /tmp/all_hosts) ;do ssh ${i} ln -s /etc/hadoop/conf /usr/local/hadoop-2.7.5/etc/hadoop ; done

Hadoop V2 - Pre-Req Setup

Hadoop V2 - Pre-Req Setup

Hadoop V2 Vineet Sachdeva   March 31, 2018

In this blog I discuss on pre-req setup for Hadoop 2 installation, I have made several improvements in the pre-req setup which you will find as compared to my Hadoop 1 Blog.

Our Controller Node is namenode or nn or namenode.cluster.com
All the below steps are on namenode
My Cluster consists of 7 nodes -
Below is my /etc/hosts which I created on namenode.
All other nodes are just installed with same password on all the nodes. The roles of the nodes are pretty much self intuitive as per the names

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.10.51 namenode.cluster.com      namenode nn
192.168.10.52 rmanager.cluster.com      rmanager rm
192.168.10.53 snamenode.cluster.com     snamenode snn
192.168.10.54 d1node.cluster.com        d1node    d1n
192.168.10.55 d2node.cluster.com        d2node    d2n
192.168.10.58 d3node.cluster.com        d3node    d3n
192.168.10.57 d4node.cluster.com        d4node    d4n

Version on which I am installing
[Linux 7 Red Hat Enterprise Linux Server release 7.3 (Maipo) 3.10.0-514.el7.x86_64]

Step 1
[As root]
Install pdsh and expect
wget http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/pdsh-2.31-1.el7.x86_64.rpm
wget http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/pdsh-rcmd-rsh-2.31-1.el7.x86_64.rpm

yum install pdsh-2.31-1.el7.x86_64.rpm pdsh-rcmd-rsh-2.31-1.el7.x86_64.rpm -y
yum install expect -y



Step 2  - Create ssh key
[As root]
[root@namenode ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
88:1c:af:e6:23:72:de:b2:76:8d:c6:86:d5:5a:96:1c root@namenode.cluster.com
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|    .            |
|   . +E.         |
|    ooooS        |
|    ..*          |
|   +o*           |
|. *+O .          |
| =oBo.           |
+-----------------+


Step 2.1
[As root - Automation File Creation]
Create Below File (This is to setup passwordless ssh and file-copy)
(Source - http://www.techpaste.com/2013/04/shell-script-automate-ssh-key-transfer-hosts-linux/)
File Name - /tmp/keysetup.exp
#!/usr/bin/env expect
set host [lrange $argv 0 0]

set username [lrange $argv 1 1]
set password [lrange $argv 2 2]

set lusername [lrange $argv 3 3]
set timeout 10
spawn /usr/bin/ssh-keygen -R $host
spawn /usr/bin/ssh-copy-id -i /$lusername/.ssh/id_rsa.pub $username@$host
match_max 100000
expect "*?(yes/no)\?"
send -- "yes\r"
# Look for password prompt
expect {
"*?assword:*" { send -- "$password\r"; send -- "\r"; exp_continue }
eof { exit 1 }
"Now try*\r" { exit 0 }
timeout { exit 1 }
}
exit 0

Provide Execute Permissions
chmod u+x /tmp/keysetup.exp

Create all_hosts file with a hostname present in newline for each host
[root@namenode tmp]# cat /tmp/all_hosts
nn
snn
rm
d1n
d2n
d3n
d4n
chmod 777 /tmp/all_hosts

Step 3
[As root - Passworless Ssh Setup]

Run below to setup passwordless as user root.
# for i in $(cat /tmp/all_hosts) ; do /tmp/keysetup.exp ${i} root remote_password root ; done
Now namenode can login to all hosts without password.

Step 4
[As root - sysctl file creation and secure liimits file creation]

Create Sysctl file for hadoop
[root@namenode tmp]# cat /tmp/98-sysctl-hadoop.conf
fs.file-max=6815744
fs.aio-max-nr=1048576
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 262144 16777216
net.ipv4.tcp_wmem=4096 262144 16777216
vm.swappiness=10

Create secur_conf.conf file for security limits
[root@namenode tmp]# cat secur_conf.conf
@hadoop soft nofile 32768
@hadoop hard nofile 32768
@hadoop soft nproc 32768
@hadoop soft nproc 32768

Step 5
[As root - Distribute sysctl configuration file and Apply ]
# for i in $(cat /tmp/all_hosts) ; do scp /tmp/98-sysctl-hadoop.conf ${i}:/etc/sysctl.d/ ; done
# for i in $(cat /tmp/all_hosts) ; do ssh ${i} mv /etc/sysctl.d/99-g03r02c00.conf /etc/sysctl.d/95-g03r02c00.conf ; done
[As root - Apply Settings ]
# export WCOLL=/tmp/all_hosts
# pdsh -R exec  /usr/sbin/sysctl --system


Step 6
[As root - Distribute limits file and Deploy]
# for i in $(cat /tmp/all_hosts) ; do scp /tmp/secur_conf.conf ${i}:/tmp/ ; done
#  for i in $(cat /tmp/all_hosts) ; do ssh ${i} 'cat /tmp/secur_conf.conf >> /etc/security/limits.conf'  ; done

Step 7
[As root - Disable Transparent Huge Pages Compaction]
# Create File
cat /tmp/thp.disable
echo 'never'; defrag_file_pathname

# for i in $(cat /tmp/all_hosts) ; do scp /tmp/thp.disable ${i}:/tmp/ ; done
# for i in $(cat /tmp/all_hosts) ; do ssh ${i} 'cat /tmp/thp.disable >> /etc/rc.local'  ; done

Step 8
[As root - Disable SE Linux ]
# export WCOLL=/tmp/all_hosts
# pdsh -R exec setenforce 0

Step 9
[As root -  Reboot all machines (and wait)]
#for i in  $(cat /tmp/all_hosts) ; do ssh ${i} reboot ; done

Step 10
[As root - Hosts File Updation]
#for i in $(cat /tmp/all_hosts) ; do scp /etc/hosts ${i}:/etc/hosts; done

Step 11
[As root - Group Creation]
## for i in $(cat /tmp/all_hosts) ; do echo "ssh ${i} groupadd -g 1000 hadoop" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd

[As root - User Creation]

#for i in $(cat /tmp/all_hosts) ; do echo "ssh ${i} useradd -u 1003  -g hadoop mapred" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd

#for i in $(cat /tmp/all_hosts) ; do echo "ssh ${i} useradd -u 1004  -g hadoop yarn" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd

## for i in $(cat /tmp/all_hosts) ; do echo "ssh ${i} useradd -u 1005  -g hadoop hdfs" >> /tmp/useradd; done ; bash /tmp/useradd ; rm -f /tmp/useradd

Step 12
[As root - hdfs, mapred an yarn user password change ]
Change hduser to mapred and then yarn for executions.

Create Script as below and run it
#!/bin/bash
for server in `cat /tmp/all_hosts`; do
echo $server;
ssh ${server} 'passwd hduser <<EOF
hadoop
hadoop
EOF';
done

Step 13
[As hdfs, mapred, yarn - Setup ssh equivalency on namenode, rmanager, snamenode]

File Name - /tmp/keysetup.exp
#!/usr/bin/env expect
set host [lrange $argv 0 0]

set username [lrange $argv 1 1]
set password [lrange $argv 2 2]

set lusername [lrange $argv 3 3]
set timeout 10
spawn /usr/bin/ssh-keygen -R $host
spawn /usr/bin/ssh-copy-id -i /home/$lusername/.ssh/id_rsa.pub $username@$host
match_max 100000
expect "*?(yes/no)\?"
send -- "yes\r"
# Look for password prompt
expect {
"*?assword:*" { send -- "$password\r"; send -- "\r"; exp_continue }
eof { exit 1 }
"Now try*\r" { exit 0 }
timeout { exit 1 }
}
exit 0

Provide Execute Permissions
chmod 777 /tmp/keysetup.exp

[As hdfs]
# ssh-keygen
#for i in $(cat /tmp/all_hosts) ; do /tmp/keysetup.exp ${i} hdfs remote_password hdfs ; done
[As mapred]
# ssh-keygen
# for i in $(cat /tmp/all_hosts) ; do /tmp/keysetup.exp ${i} mapred remote_password mapred ; done
[As yarn]
# ssh-keygen
#for i in $(cat /tmp/all_hosts) ; do /tmp/keysetup.exp ${i} yarn remote_password yarn ; done


Step 14
[As root - Java Installation]
#for i in $(cat /tmp/all_hosts) ; do echo "scp jdk-8u152-linux-x64.rpm ${i}:/tmp &" >> /tmp/sendjdk.bash ; done
Paste and run contents of the file
# for i in $(cat /tmp/all_hosts) ; do  ssh ${i}  rpm -Uvh /tmp/jdk-8u152-linux-x64.rpm  ; done;

Step 15
[As root - Set Env Variables]
Create file profile.sh
cat /tmp/profile.sh
export JAVA_HOME=/usr/java/latest
export HADOOP_PREFIX=/usr/local/hadoop
export LOG=/opt/HDPV2/logs
export CONF=/etc/hadoop/conf
export PATH=$JAVA_HOME/bin:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$PATH

#for i in $(cat /tmp/all_hosts) ; do scp /tmp/profile.sh ${i}:/etc/profile.d/ ; done
#for i in $(cat /tmp/all_hosts) ; do ssh ${i} chmod 755 /etc/profile.d/profile.sh ; done
#for i in $(cat /tmp/all_hosts) ; do ssh ${i} source /etc/profile.d/profile.sh ; done

Step 16
[As root - Permissions Set]
My Mount is going to /opt/HDPV2 for my hadoop based data.
# for i in $(cat /tmp/all_hosts) ; do ssh ${i} chown root:hadoop /opt /opt/HDPV2 ; done

Hadoop - ENOENT: No such file or directory (Mapred Task Start)

Hadoop - ENOENT: No such file or directory (Mapred Task Start)

Hadoop V2 Vineet Sachdeva   March 04, 2018

You get below Error in your Mapreduce job and there are directories which need permission fix. 
The directories are 

1. <HADDOP_LOG>/userlogs to have right permission for mapred
2. mr/userlogs - These are all the directories which are mentioned as mapreduce local directories in the configuration files.

Once you fix the permission such that user who is starting mapreduce job - tasktrackers. It will get fixed automatically on next start of job.

2018-03-03 15:02:40,011 WARN org.apache.hadoop.mapred.TaskTracker: Exception while localization ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
        at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
        at java.lang.Thread.run(Thread.java:748)

2018-03-03 15:02:40,012 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred cause:ENOENT: No such file or directory
2018-03-03 15:02:40,013 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201803031350_0001_m_000019_3:
ENOENT: No such file or directory

Hadoop V1/V2 - Administrative Commands Starter

Hadoop V1/V2 - Administrative Commands Starter

Hadoop V2 Vineet Sachdeva   March 04, 2018

In this blog we are going to see some basic administrative commands in Hadoop - 


1. Status Report of DFS (Distributed File Systems)
 hadoop dfsadmin -report
Configured Capacity: 133660540928 (124.48 GB)
Present Capacity: 133660540928 (124.48 GB)
DFS Remaining: 133660270592 (124.48 GB)
DFS Used: 270336 (264 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 4 (4 total, 0 dead)

Name: 192.168.10.54:50010
Decommission Status : Normal
Configured Capacity: 33415135232 (31.12 GB)
DFS Used: 69632 (68 KB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 33415065600(31.12 GB)
DFS Used%: 0%
DFS Remaining%: 100%
Last contact: Fri Mar 02 12:59:27 CET 2018


Name: 192.168.10.57:50010
Decommission Status : Normal
Configured Capacity: 33415135232 (31.12 GB)
DFS Used: 61440 (60 KB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 33415073792(31.12 GB)
DFS Used%: 0%
DFS Remaining%: 100%
Last contact: Fri Mar 02 12:59:27 CET 2018


Name: 192.168.10.55:50010
Decommission Status : Normal
Configured Capacity: 33415135232 (31.12 GB)
DFS Used: 69632 (68 KB)
on DFS Used: 0 (0 KB)
DFS Remaining: 33415065600(31.12 GB)
DFS Used%: 0%
DFS Remaining%: 100%
Last contact: Fri Mar 02 12:59:27 CET 2018


Name: 192.168.10.58:50010
Decommission Status : Normal
Configured Capacity: 33415135232 (31.12 GB)
DFS Used: 69632 (68 KB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 33415065600(31.12 GB)
DFS Used%: 0%
DFS Remaining%: 100%
Last contact: Fri Mar 02 12:59:27 CET 2018


2. Check Contents of DFS
[hduser@namenode ~]$ hadoop dfs -ls /
Found 1 items
drwxr-xr-x   - hduser supergroup          0 2018-02-22 04:14 /tmp

3. Create Directory
[hduser@namenode ~]$ hadoop dfs -mkdir /user/vin

4. Copy  and Delete File / Directory in HDFS
[hduser@namenode hadoop]$ ls -l conf/
total 84
-rwxr-xr-x 1 root root 7457 Jul 23  2013 capacity-scheduler.xml
-rwxr-xr-x 1 root root 1095 Jul 23  2013 configuration.xsl
-rwxr-xr-x 1 root root  415 Feb 22 03:48 core-site.xml
-rwxr-xr-x 1 root root   76 Feb 22 03:48 dfs.hosts.include
-rwxr-xr-x 1 root root  327 Jul 23  2013 fair-scheduler.xml
-rwxr-xr-x 1 root root 2497 Feb 22 03:48 hadoop-env.sh
-rwxr-xr-x 1 root root 2052 Jul 23  2013 hadoop-metrics2.properties
-rwxr-xr-x 1 root root 4644 Jul 23  2013 hadoop-policy.xml
-rwxr-xr-x 1 root root 1357 Feb 22 03:48 hdfs-site.xml
-rwxr-xr-x 1 root root 5018 Jul 23  2013 log4j.properties
-rwxr-xr-x 1 root root 2033 Jul 23  2013 mapred-queue-acls.xml
-rwxr-xr-x 1 root root 1359 Feb 25 13:49 mapred-site.xml
-rwxr-xr-x 1 root root    4 Feb 22 03:48 masters
-rwxr-xr-x 1 root root   16 Feb 22 03:48 slaves
-rwxr-xr-x 1 root root 2042 Jul 23  2013 ssl-client.xml.example
-rwxr-xr-x 1 root root 1994 Jul 23  2013 ssl-server.xml.example
-rwxr-xr-x 1 root root  382 Jul 23  2013 taskcontroller.cfg
-rwxr-xr-x 1 root root 3890 Jul 23  2013 task-log4j.properties

Copy file to particular Directory hdfs
[hduser@namenode etc]$ hadoop dfs -put group /user/vin/


Copy file to Directory with target file name
[hduser@namenode etc]$ hadoop dfs -put group /user/vin/mygroupfile

Wild Card Listing of File
[hduser@namenode etc]$ hadoop dfs -ls /user/vin/*group*
-rw-r--r--   3 hduser supergroup        682 2018-03-02 13:19 /user/vin/group
-rw-r--r--   3 hduser supergroup        682 2018-03-02 13:19 /user/vin/mygroupfile


[hduser@namenode hadoop]$ hadoop dfs -put conf/ /user/vin/

[hduser@namenode hadoop]$ hadoop dfs -ls /user/vin/conf
Found 18 items
-rw-r--r--   3 hduser supergroup       7457 2018-03-02 13:08 /user/vin/conf/capacity-scheduler.xml
-rw-r--r--   3 hduser supergroup       1095 2018-03-02 13:08 /user/vin/conf/configuration.xsl
-rw-r--r--   3 hduser supergroup        415 2018-03-02 13:08 /user/vin/conf/core-site.xml
-rw-r--r--   3 hduser supergroup         76 2018-03-02 13:08 /user/vin/conf/dfs.hosts.include
-rw-r--r--   3 hduser supergroup        327 2018-03-02 13:08 /user/vin/conf/fair-scheduler.xml
-rw-r--r--   3 hduser supergroup       2497 2018-03-02 13:08 /user/vin/conf/hadoop-env.sh
-rw-r--r--   3 hduser supergroup       2052 2018-03-02 13:08 /user/vin/conf/hadoop-metrics2.properties
-rw-r--r--   3 hduser supergroup       4644 2018-03-02 13:08 /user/vin/conf/hadoop-policy.xml
-rw-r--r--   3 hduser supergroup       1357 2018-03-02 13:08 /user/vin/conf/hdfs-site.xml
-rw-r--r--   3 hduser supergroup       5018 2018-03-02 13:08 /user/vin/conf/log4j.properties
-rw-r--r--   3 hduser supergroup       2033 2018-03-02 13:08 /user/vin/conf/mapred-queue-acls.xml
-rw-r--r--   3 hduser supergroup       1359 2018-03-02 13:08 /user/vin/conf/mapred-site.xml
-rw-r--r--   3 hduser supergroup          4 2018-03-02 13:08 /user/vin/conf/masters
-rw-r--r--   3 hduser supergroup         16 2018-03-02 13:08 /user/vin/conf/slaves
-rw-r--r--   3 hduser supergroup       2042 2018-03-02 13:08 /user/vin/conf/ssl-client.xml.example
-rw-r--r--   3 hduser supergroup       1994 2018-03-02 13:08 /user/vin/conf/ssl-server.xml.example
-rw-r--r--   3 hduser supergroup       3890 2018-03-02 13:08 /user/vin/conf/task-log4j.properties
-rw-r--r--   3 hduser supergroup        382 2018-03-02 13:08 /user/vin/conf/taskcontroller.cfg

Delete Directory
[hduser@namenode ~]$ hadoop fs -rmr /tmp/o1
Moved to trash: hdfs://nn:8020/tmp/o1
Delete File(s)
[hduser@namenode ~]$ hadoop fs -rm /user/vin/conf/mapred-site.xml
Moved to trash: hdfs://nn:8020/user/vin/conf/mapred-site.xml

[hduser@namenode ~]$ hadoop fs -rm /user/vin/conf/hadoop*
Moved to trash: hdfs://nn:8020/user/vin/conf/hadoop-env.sh
Moved to trash: hdfs://nn:8020/user/vin/conf/hadoop-metrics2.properties
Moved to trash: hdfs://nn:8020/user/vin/conf/hadoop-policy.xml


5. Set Quota on Directory
[hduser@namenode hadoop]$ hadoop dfs -mkdir /user/nish
[hduser@namenode hadoop]$ hadoop dfsadmin -setSpaceQuota 1098304000 /user/nish #1000M Quota

6. Check Quota
[hduser@namenode hadoop]$ hadoop dfs -count -q /user/nish
        none             inf        10983040        10983040            1            0                  0 hdfs://nn:8020/user/nish

7. Put File After Quota
 [hduser@namenode conf]$ ls -l mapred-site.xml
-rwxr-xr-x 1 root root 1359 Feb 25 13:49 mapred-site.xml

[hduser@namenode conf]$ hadoop dfs -put mapred-site.xml /user/nish/
18/03/02 13:16:50 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/nish is exceeded: quota=10983040 diskspace consumed=384.0m

*** Quota Should be atleast Block Size * replication Factor

8. Add Quota to number of files
# First Increase space Quota to 1G
[hduser@namenode etc]$ hadoop dfsadmin -setSpaceQuota 1073741824 /user/nish
Set # of Files to 10
[hduser@namenode etc]$ hadoop dfsadmin -setQuota 10 /user/nish

Check Quota Output
[hduser@namenode etc]$ hadoop dfs -count -q /user/nish
          10               8      1073741824      1073741824            1            1                  0 hdfs://nn:8020/user/nish

9. Put 10 Files After Quota
[hduser@namenode hadoop]$ hadoop dfs -put conf/* /user/nish/
put: org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota (directories and files) of directory /user/nish is exceeded: quota=10 file count=11

Check How many Files are placed?
 hadoop dfs -ls /user/nish
Found 9 items
-rw-r--r--   3 hduser supergroup       7457 2018-03-02 13:26 /user/nish/capacity-scheduler.xml
-rw-r--r--   3 hduser supergroup       1095 2018-03-02 13:26 /user/nish/configuration.xsl
-rw-r--r--   3 hduser supergroup        415 2018-03-02 13:26 /user/nish/core-site.xml
-rw-r--r--   3 hduser supergroup         76 2018-03-02 13:26 /user/nish/dfs.hosts.include
-rw-r--r--   3 hduser supergroup        327 2018-03-02 13:26 /user/nish/fair-scheduler.xml
-rw-r--r--   3 hduser supergroup       2497 2018-03-02 13:26 /user/nish/hadoop-env.sh
-rw-r--r--   3 hduser supergroup       2052 2018-03-02 13:26 /user/nish/hadoop-metrics2.properties
-rw-r--r--   3 hduser supergroup       4644 2018-03-02 13:26 /user/nish/hadoop-policy.xml
-rw-r--r--   3 hduser supergroup          0 2018-03-02 13:16 /user/nish/mapred-site.xml

10. Clear Quotas
[hduser@namenode hadoop]$ hadoop dfsadmin -clrSpaceQuota /user/nish
[hduser@namenode hadoop]$ hadoop dfsadmin -clrQuota /user/nish

Verify Quota
[hduser@namenode hadoop]$ hadoop dfs -count -q /user/nish
        none             inf            none             inf            1            9              18563 hdfs://nn:8020/user/nish

11.  FSCK
[hduser@namenode ~]$ hadoop fsck /
FSCK started by hduser from /192.168.10.51 for path / at Sat Mar 03 14:30:30 CET 2018
................................Status: HEALTHY
 Total size:    57952 B
 Total dirs:    13
 Total files:   32
 Total blocks (validated):      31 (avg. block size 1869 B)
 Minimally replicated blocks:   31 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          4
 Number of racks:               1
FSCK ended at Sat Mar 03 14:30:30 CET 2018 in 25 milliseconds


The filesystem under path '/' is HEALTHY

[hduser@namenode ~]$ hadoop fsck /user/vin/conf/capacity-scheduler.xml -files -blocks -locations
FSCK started by hduser from /192.168.10.51 for path /user/vin/conf/capacity-scheduler.xml at Sat Mar 03 14:31:36 CET 2018
/user/vin/conf/capacity-scheduler.xml 7457 bytes, 1 block(s):  OK
0. blk_4751168492478894034_1008 len=7457 repl=3 [192.168.10.55:50010, 192.168.10.57:50010, 192.168.10.54:50010]

Status: HEALTHY
 Total size:    7457 B
 Total dirs:    0
 Total files:   1
 Total blocks (validated):      1 (avg. block size 7457 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          4
 Number of racks:               1
FSCK ended at Sat Mar 03 14:31:36 CET 2018 in 1 milliseconds


The filesystem under path '/user/vin/conf/capacity-scheduler.xml' is HEALTHY


12. Killing a Job in Hadoop
Find out the job Id using super user and kill it  -

[hduser@namenode ~]$ hadoop job -list
hadoop 1 jobs currently running
JobId   State   StartTime       UserName        Priority        SchedulingInfo
job_201803041408_0003   1       1520169058581   mapred  NORMAL  NA

[hduser@namenode ~]$ hadoop job  -kill job_201803041408_0003
Killed job job_201803041408_0003

13. Killing Specific Task Attempt in Hadoop
[hduser@namenode ~]$ hadoop job -list
1 jobs currently running
JobId   State   StartTime       UserName        Priority        SchedulingInfo
job_201803041408_0006   4       1520169519215   mapred  NORMAL  NA
[hduser@namenode ~]$ hadoop job -list-attempt-ids job_201803041408_0006 reduce running
attempt_201803041408_0006_r_000000_0
attempt_201803041408_0006_r_000001_0
attempt_201803041408_0006_r_000002_0
attempt_201803041408_0006_r_000003_0
attempt_201803041408_0006_r_000004_0
attempt_201803041408_0006_r_000005_0
attempt_201803041408_0006_r_000006_0
attempt_201803041408_0006_r_000007_0
[hduser@namenode ~]$ hadoop job -kill-task attempt_201803041408_0006_r_000006_0
Killed task attempt_201803041408_0006_r_000006_0

Hadoop V1 - Fair Scheduler Configuration

Hadoop V1 - Fair Scheduler Configuration

Hadoop V1 Vineet Sachdeva   March 04, 2018

This is in continuation with my previous blog on Mapreduce Config
In this blog I am going to  Discuss Scheduler Configuration Fair-Scheduler -

Below is my mapred-site.xml for Fair Scheduler Configuration - 

<property>
    <name></name>
    <value></value>
</property>
<property>
    <name>mapred.jobtracker.taskScheduler</name>
    <value>org.apache.hadoop.mapred.FairScheduler</value>
</property>

<property>
    <name>mapred.fairscheduler.allocation.file</name>
    <value>/etc/hadoop/conf/fair-scheduler.xml</value>
</property>

<property>
    <name>mapred.fairscheduler.poolnameproperty</name>
    <value>user.name</value>
</property>

<property>
    <name>mapred.fairscheduler.preemption</name>
    <value>true</value>
</property>

<property>
    <name>mapred.fairscheduler.sizebasedweight</name>
    <value>true</value>
</property>

<property>
    <name>mapred.fairscheduler.assignmultiple.maps</name>
    <value>5</value>
</property>

<property>
    <name>mapred.fairscheduler.assignmultiple.reduces</name>
    <value>3</value>
</property>

Fair-Scheduler configuration File
vi /etc/hadoop/conf/fair-scheduler.xml


<allocations>
    <defaultMinSharePreemptionTimeout>600</defaultMinSharePreemptionTimeout>
    <pool name="prod-analytics">
        <minMaps>30<minMaps>
        <minReduces>10</minReduces>
    </pool>
    <pool name="dev-users">
        <weight>2</weight>
    </pool>
    <pool name="qa-users">
        <weight>1</weight>
    </pool>
    <user name="james">
        <maxRunningJobs>3</maxRunningJobs>
    </user>   

</allocations>


[As root]
# for i in $(cat /tmp/hosts) ;do scp mapred-site.xml fair-scheduler.xml ${i}:/etc/hadoop/conf/ ; done

[As root - Give Permissions[
# for i in $(cat /tmp/hosts) ;do ssh ${i} chmod -R 755 /etc/hadoop ; done;

[Stop and Restart mapred-Services]
stop-mapred.sh
start-mapred.sh

From the log file

[mapred@namenode logs]$ less hadoop-mapred-jobtracker-namenode.cluster.com.log | grep -i Fair
2018-03-03 13:43:03,169 INFO org.apache.hadoop.mapred.FairScheduler: Successfully configured FairScheduler


Verify Java Processes on all nodes

[mapred@namenode ~]$ for i in $(cat /tmp/hosts) ; do ssh ${i} 'hostname; jps | grep -vi jps; echo' ;  done;
namenode.cluster.com
4722 JobTracker


d1node.cluster.com
30419 TaskTracker

d2node.cluster.com
1600 TaskTracker

d3node.cluster.com
26777 TaskTracker

d4node.cluster.com
10144 TaskTracker