hadoop打成jar包放到服务器上运行问题集

2024年6月7日 161次阅读 来源: 波波哥1986

有时候需要在某台服务器上运行本地的mapreduce任务,可以这样设置:

conf.set("fs.default.name", "local")
conf.set("mapred.job.tracker", "local")

###########################################################################################

打成jar包在服务器上执行时遇到

Exception in thread "main" java.io.IOException: No FileSystem for scheme: file
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2385)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
    at com.yunsuan.gaoyuan.recom.CreateTfidfJob.getFileIndex(CreateTfidfJob.java:143)
    at com.yunsuan.gaoyuan.recom.CreateTfidfJob.run(CreateTfidfJob.java:74)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.yunsuan.gaoyuan.recom.CreateTfidfJob.main(CreateTfidfJob.java:42)

检查检查生成的jar中META-INF->services->org.apache.hadoop.fs.FileSystem文件

缺少:org.apache.hadoop.fs.LocalFileSystem #处理local file scheme的类

原因:maven打jar包的时候,覆盖了一个..

处理方法:

 

config.set("fs.hdfs.impl",org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
config.set("fs.file.impl",org.apache.hadoop.fs.LocalFileSystem.class.getName());

参考:
http://blog.newitfarmer.com/big_data/big-data-platform/hadoop/13953/repost-no-filesystem-for-scheme-hdfsno-filesystem-for-scheme-file

########################################################################################################################

jar包在服务器上运行时

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.elapsedMillis()J
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:278)
    at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:217)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
    at com.yunsuan.gaoyuan.recom.MySequenceFilesFromDirectoryJob.runMapReduce(MySequenceFilesFromDirectoryJob.java:245)
    at com.yunsuan.gaoyuan.recom.MySequenceFilesFromDirectoryJob.run(MySequenceFilesFromDirectoryJob.java:115)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.yunsuan.gaoyuan.recom.CreateTfidfJob.run(CreateTfidfJob.java:85)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.yunsuan.gaoyuan.recom.CreateTfidfJob.main(CreateTfidfJob.java:38)

原因:依赖的guava版本不对

解决:改成guava-14.0.jar

########################################################################################################################

jar包在服务器上运行时

java.sql.SQLException: Access denied for user 'root'@'host08' (using password: YES)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:926)
    at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4732)
    at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1340)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2539)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2321)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    ......

mysql在本地使用-h,再用用户名密码登陆的话会出错…

解决:创建一个需要使用密码访问本地host的mysql的用户
grant all on *.* to ‘root’@’host08’ identified by ‘123456’

########################################################################################################################

结果通过mapred存入mysql时:

java.io.IOException: wrong key class: class com.yunsuan.gaoyuan.comm.TblsWritable is not class org.apache.hadoop.io.IntWritable
    at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:196)
    at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1307)
    at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1624)
    at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
    at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
    at com.yunsuan.gaoyuan.recom.ComputeSimilarityJob$ChangeToSimilarityMatrixReducer.reduce(ComputeSimilarityJob.java:302)
    at com.yunsuan.gaoyuan.recom.ComputeSimilarityJob$ChangeToSimilarityMatrixReducer.reduce(ComputeSimilarityJob.java:1)
    .....

解决:去掉setCombinerClass

ChangeToSimilarityMatrixJob.setCombinerClass(ChangeToSimilarityMatrixReducer.class)

    原文作者：波波哥1986
    原文地址: https://blog.csdn.net/u014617685/article/details/40588633
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞