测试用windows 下无守护进程的 hadoop和虚拟机linux下的hadoop配合使用

1. 环境变量
   HADOOP_INCLUDE : D:\soft\hadoop-2.7.3
   PATH : %HADOOP_INCLUDE%\bin;%HADOOP_INCLUDE%\sbin;

2. 错误解决

下载 hadoop-common-2.2.0-bin解压，拷贝到%HADOOP_INCLUDE%\bin，注意不替换已有文件：
注意hadoop-common和hadoop版本号要一致

C:\Users\admin>hadoop fs -ls
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
16/12/15 17:55:06 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable D:\soft\hadoop-2.7.3\bin\winutils.exe in the Hadoop binar
ies.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:387)
        at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:440)

        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:486)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
16/12/15 17:55:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... us
ing builtin-java classes where applicable
Found 62 items
-ls: Fatal internal error
java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: (nu
ll) entry in command string: null ls -F C:\Users\admin\.AndroidStudio2.1
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:769)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
        at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLoca
lFileSystem.java:659)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getOwner(RawLocalFileSyste
m.java:642)
        at org.apache.hadoop.fs.shell.Ls.adjustColumnWidths(Ls.java:131)
        at org.apache.hadoop.fs.shell.Ls.processPaths(Ls.java:102)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:90)
        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLoca
lFileSystem.java:697)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getOwner(RawLocalFileSyste
m.java:642)
        at org.apache.hadoop.fs.shell.Ls.adjustColumnWidths(Ls.java:131)
        at org.apache.hadoop.fs.shell.Ls.processPaths(Ls.java:102)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:90)
        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
        at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

C:\Users\admin>hadoop fs -ls

几个配置文件

// mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
       <name>mapred.job.tracker</name>
       <value>hdfs://localhost:9001</value>
    </property>
</configuration>

//yarn-site.xml

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>localhost</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

//core-site.xml
hadoop.tmp.dir配置windows下的tmp位置

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>

    <property>  
        <name>hadoop.tmp.dir</name>  
        <value>/f:/tmp</value>  
        <description>A base for other temporary directories.</description>  
    </property>
</configuration>

//hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

格式化节点,启动hadoop

注意start-all.cmd回启动四个进程（windows command），stop-all.cmd会停止启动的进程，如果启动的进程中有出现错误的，则调用stop-all.cmd时不会出现错误的windows command

hadoop namenode -format
start-all.cmd
stop-all.cmd