Advertisement

Sunday, March 4, 2018

Hadoop - ENOENT: No such file or directory (Mapred Task Start)

You get below Error in your Mapreduce job and there are directories which need permission fix. 
The directories are 

1. <HADDOP_LOG>/userlogs to have right permission for mapred
2. mr/userlogs - These are all the directories which are mentioned as mapreduce local directories in the configuration files.

Once you fix the permission such that user who is starting mapreduce job - tasktrackers. It will get fixed automatically on next start of job.

2018-03-03 15:02:40,011 WARN org.apache.hadoop.mapred.TaskTracker: Exception while localization ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
        at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
        at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205)
        at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226)
        at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603)
        at java.lang.Thread.run(Thread.java:748)

2018-03-03 15:02:40,012 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred cause:ENOENT: No such file or directory
2018-03-03 15:02:40,013 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201803031350_0001_m_000019_3:
ENOENT: No such file or directory

No comments:
Write comments