Advertisement

Friday, April 20, 2018

Hadoop V2 - Sqoop - NoSuchMethodError


Symptoms - 
Sqoop job fails when importing with avro type format. 

Container logs show 

org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
 

Solution - Run the job by adding arguements to the comand line -
-Dmapreduce.job.user.classpath.first=true

sqoop  import-all-tables -Dmapreduce.job.user.classpath.first=true --connect jdbc:oracle:thin:@192.168.1.71:6633:EMPRD \
--username SCOTT --password TIGER -m 1 -as-avrodatafile



This will make sure the right classes are accesible to sqop job during import and MR job.

No comments:
Write comments