This is in continuation with my previous blog on Mapreduce Config
In this blog I am going to Discuss Scheduler Configuration Fair-Scheduler -
Below is my mapred-site.xml for Fair Scheduler Configuration -
<property>
<name></name>
<value></value>
</property>
<property>
<name>mapred.jobtracker.taskScheduler</name>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<value>/etc/hadoop/conf/fair-scheduler.xml</value>
</property>
<property>
<name>mapred.fairscheduler.poolnameproperty</name>
<value>user.name</value>
</property>
<property>
<name>mapred.fairscheduler.preemption</name>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.sizebasedweight</name>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.assignmultiple.maps</name>
<value>5</value>
</property>
<property>
<name>mapred.fairscheduler.assignmultiple.reduces</name>
<value>3</value>
</property>
Fair-Scheduler configuration File
vi /etc/hadoop/conf/fair-scheduler.xml
<allocations>
<defaultMinSharePreemptionTimeout>600</defaultMinSharePreemptionTimeout>
<pool name="prod-analytics">
<minMaps>30<minMaps>
<minReduces>10</minReduces>
</pool>
<pool name="dev-users">
<weight>2</weight>
</pool>
<pool name="qa-users">
<weight>1</weight>
</pool>
<user name="james">
<maxRunningJobs>3</maxRunningJobs>
</user>
</allocations>
[As root]
# for i in $(cat /tmp/hosts) ;do scp mapred-site.xml fair-scheduler.xml ${i}:/etc/hadoop/conf/ ; done
[As root - Give Permissions[
# for i in $(cat /tmp/hosts) ;do ssh ${i} chmod -R 755 /etc/hadoop ; done;
[Stop and Restart mapred-Services]
stop-mapred.sh
start-mapred.sh
From the log file
[mapred@namenode logs]$ less hadoop-mapred-jobtracker-namenode.cluster.com.log | grep -i Fair
2018-03-03 13:43:03,169 INFO org.apache.hadoop.mapred.FairScheduler: Successfully configured FairScheduler
Verify Java Processes on all nodes
[mapred@namenode ~]$ for i in $(cat /tmp/hosts) ; do ssh ${i} 'hostname; jps | grep -vi jps; echo' ; done;
namenode.cluster.com
4722 JobTracker
d1node.cluster.com
30419 TaskTracker
d2node.cluster.com
1600 TaskTracker
d3node.cluster.com
26777 TaskTracker
d4node.cluster.com
10144 TaskTracker
In this blog I am going to Discuss Scheduler Configuration Fair-Scheduler -
Below is my mapred-site.xml for Fair Scheduler Configuration -
<property>
<name></name>
<value></value>
</property>
<property>
<name>mapred.jobtracker.taskScheduler</name>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<value>/etc/hadoop/conf/fair-scheduler.xml</value>
</property>
<property>
<name>mapred.fairscheduler.poolnameproperty</name>
<value>user.name</value>
</property>
<property>
<name>mapred.fairscheduler.preemption</name>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.sizebasedweight</name>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.assignmultiple.maps</name>
<value>5</value>
</property>
<property>
<name>mapred.fairscheduler.assignmultiple.reduces</name>
<value>3</value>
</property>
Fair-Scheduler configuration File
vi /etc/hadoop/conf/fair-scheduler.xml
<allocations>
<defaultMinSharePreemptionTimeout>600</defaultMinSharePreemptionTimeout>
<pool name="prod-analytics">
<minMaps>30<minMaps>
<minReduces>10</minReduces>
</pool>
<pool name="dev-users">
<weight>2</weight>
</pool>
<pool name="qa-users">
<weight>1</weight>
</pool>
<user name="james">
<maxRunningJobs>3</maxRunningJobs>
</user>
</allocations>
[As root]
# for i in $(cat /tmp/hosts) ;do scp mapred-site.xml fair-scheduler.xml ${i}:/etc/hadoop/conf/ ; done
[As root - Give Permissions[
# for i in $(cat /tmp/hosts) ;do ssh ${i} chmod -R 755 /etc/hadoop ; done;
[Stop and Restart mapred-Services]
stop-mapred.sh
start-mapred.sh
From the log file
[mapred@namenode logs]$ less hadoop-mapred-jobtracker-namenode.cluster.com.log | grep -i Fair
2018-03-03 13:43:03,169 INFO org.apache.hadoop.mapred.FairScheduler: Successfully configured FairScheduler
Verify Java Processes on all nodes
[mapred@namenode ~]$ for i in $(cat /tmp/hosts) ; do ssh ${i} 'hostname; jps | grep -vi jps; echo' ; done;
namenode.cluster.com
4722 JobTracker
d1node.cluster.com
30419 TaskTracker
d2node.cluster.com
1600 TaskTracker
d3node.cluster.com
26777 TaskTracker
d4node.cluster.com
10144 TaskTracker
No comments:
Write comments