黑猴子的家：Hadoop 日志聚集

2023年8月2日 263次阅读

Job运行完成以后，将日志信息上传到HDFS系统上

1、配置yarn-site.xml

<!-- 开启日志聚集功能 -->
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

<!-- 日志保留时间设置7天 -->
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>

<!-- 日志检查时间设置为1天 -->
<property>
    <name>yarn.log-aggregation.retain-check-interval-seconds</name>         
    <value>86400</value>
</property>

<!-- 默认目录为/tmp/logs，根据需求修改 -->
<property>
    <name>yarn.nodemanager.remote-app-log-dir</name>         
    <value>/bigdata/logs/hadoop/Yarnlogs</value>
</property>

<property>
    <name>yarn.log.server.url</name>         
    <value>http://hadoop102:19888/jobhistory/logs </value>
</property>

<property>
    <name>mapred.child.java.opts</name>         
    <value>-Xmx1024m</value>
</property>

2、关闭nodemanager 、resourcemanager和historyserver

[victor@node1 hadoop-2.8.2]$ sbin/mr-jobhistory-daemon.sh stop historyserver
[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh stop nodemanager
[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh stop resourcemanager

3、、启动nodemanager 、resourcemanager和historyserver

[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh start resourcemanager
[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh start nodemanager
[victor@node1 hadoop-2.8.2]$ sbin/mr-jobhistory-daemon.sh start historyserver

4、删除hdfs上已经存在的hdfs文件

[victor@node1 hadoop-2.8.2]$ bin/hdfs dfs -rm -R /user/victor/wordcount/output

5、执行wordcount程序

[victor@node1 hadoop-2.8.2]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.2.jar wordcount /user/victor/wordcount/input /user/victor/wordcount/output

6、查看日志

$ http://node1:19888/jobhistory
点赞