我正在关注这个视频教程,尝试在我的机器上设置hadoop.
> How to Install Hadoop on Windows 10
我成功地设置了它:从sbin目录执行start-all.xml时没有错误.
但是当我尝试执行我的WordCount.jar文件时,会出现错误:
2/23 11:42:59 INFO localizer.ResourceLocalizationService: Created localizer for container_1550911199370_0001_02_000001
19/02/23 11:42:59 INFO localizer.ResourceLocalizationService: Localizer failed
org.apache.hadoop.util.DiskChecker$DiskErrorException: No space available in any of the local directories.
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:399)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116)
at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:545)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1142)
19/02/23 11:42:59 ERROR nodemanager.DeletionService: Exception during execution of task in DeletionService
java.lang.NullPointerException: path cannot be null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:281)
at org.apache.hadoop.fs.FileContext.delete(FileContext.java:769)
at org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:273)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/02/23 11:42:59 INFO container.ContainerImpl: Container container_1550911199370_0001_02_000001 transitioned from LOCAL
我确信我有足够的空间来处理工作.我的系统全新安装:
配置信息:
这是我的配置文件:
核心的site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
HDFS-site.xml中
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///C:/hadoop-2.8.0/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///C:/hadoop-2.8.0/data/datanode</value>
</property>
</configuration>
mapred-site.xml中
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
纱的site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.disk-health-checker.enable</name>
<value>false</value>
</property>
</configuration>
这是我执行jar的方式(准备输入/输出dirs):
hadoop fs -mkdir /top
hadoop fs -mkdir /top/input
hadoop fs -mkdir /top/output
hadoop -put C:/hadoop-2.8.0/wordcount2.txt /top/input
hadoop jar C:/hadoop-2.8.0/WordCount.jar /top/input/wordcount2.txt /top/output/output.txt
最佳答案 主要错误是:
org.apache.hadoop.util.DiskChecker$DiskErrorException: No space available in any of the local directories.
要解决此问题,您可以尝试:
(1)更改Hdfs-site.xml中的目录格式
在hdfs-site.xml文件中,尝试替换以下值:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///C:/hadoop-2.8.0/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///C:/hadoop-2.8.0/data/datanode</value>
</property>
</configuration>
同
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>C:\hadoop-2.8.0\data\namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>C:\hadoop-2.8.0\data\datanode</value>
</property>
</configuration>
(2)目录阅读&写权限
检查当前用户是否具有读取和写入hadoop目录的权限.
(3)节点管理器目录
尝试将以下属性添加到yarn-site.xml文件中:
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>C:/hadoop-2.8.0/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>C:/hadoop-2.8.0/yarn/logs</value>
</property>
更改目录后,尝试格式化namenode.
如果它仍然不起作用,您可以参考以下分步指南在Windows上安装Hadoop,它对我来说很好用:
> Step by step Hadoop 2.8.0 installation on Window 10
> How to Run Hadoop wordcount MapReduce Example on Windows 10