Oozie WorkFlow中sqoop的命令file案例

2019年6月7日 214次阅读 来源: 志辉聊码

复制样例

[beifeng@hadoop-senior oozie-4.0.0-cdh5.3.6]$ cp -r examples/apps/sqoop oozie-apps/

样例重新命令

mv sqoop sqoop-import

复制mysql依赖包

[beifeng@hadoop-senior oozie-apps]$ mkdir sqoop-import/lib
[beifeng@hadoop-senior oozie-apps]$ cp hive-select/lib/mysql-connector-java-5.1.27-bin.jar sqoop-import/lib/

修改job.properties

nameNode=hdfs://hadoop-senior.beifeng.com:8020
jobTracker=hadoop-senior.beifeng.com:8032
queueName=default
examplesRoot=examples
oozieAppsRoot=user/beifeng/oozie-apps
oozieDataRoot=user/beifeng/oozie/datas

oozie.use.system.libpath=true

oozie.wf.application.path=${nameNode}/${oozieAppsRoot}/sqoop-import/workflow.xml
outputDir=sqoop-import-user/output

修改流程文件

<workflow-app xmlns="uri:oozie:workflow:0.5" name="sqoop-wf">
    <start to="sqoop-node"/>

    <action name="sqoop-node">
        <sqoop xmlns="uri:oozie:sqoop-action:0.3">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/${oozieDataRoot}/${outputDir}"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <command>import --options-file /opt/cdh5.3.6/oozie-4.0.0-cdh5.3.6/oozie-apps/sqoop-import/import-user.sql</command>
        </sqoop>
        <ok to="end"/>
        <error to="fail"/>
    </action>

    <kill name="fail">
        <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

注意：import –options-file path的路径必须是本地路径，而不是hdfs路径 浪费我2个小时。 <command>中的命令不是以 sqoop开头，而是以sqoop的命令开始

创建hdfs的输出目录

bin/hdfs dfs -mkdir -p /user/beifeng/oozie/datas/sqoop-import-user

上传sqoop流程到hdfs系统

bin/hdfs dfs -put ../oozie-4.0.0-cdh5.3.6/oozie-apps/sqoop-import  /user/beifeng/oozie-apps

import-user.sql

--connect 
jdbc:mysql://hadoop-senior.beifeng.com:3306/test 
--username 
root 
--password 
123456 
--table 
my_user_0321 
--num-mappers 
1  
--delete-target-dir  
--fields-terminated-by 
"\t" 
--target-dir 
/user/beifeng/oozie/datas/sqoop-import/output

运行sqoop流程job

export OOZIE_URL=http://hadoop-senior.beifeng.com:11000/oozie
bin/oozie job -config oozie-apps/sqoop-import2/job.properties  -run

查看运行结果

bin/oozie job -info JobId

JobId：0000016-180315133250705-oozie-beif-W 来自上一步运行结果

《Oozie WorkFlow中sqoop的命令file案例》

    原文作者：志辉聊码
    原文地址: https://www.jianshu.com/p/c757f60e9f82
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞