centos7 安装ELK做日志收集(elasticsearch,logstash,kibana)

2024年2月16日 139次阅读 来源: crawler

关于elk不用说，大家多多少少都听过，最近我搭建了一套用作收集日志，供大家参考：

一.安装elasticsearch,logstash,kibana

强烈建议安装去es的官网安装：
今天是2019.4.9，目前最新的版本是6.7.1，三个都装6.7.1版本(版本最好一致，否则会有各种疑难杂症)
elasticsearch: https://www.elastic.co/downlo…
logstash: https://www.elastic.co/cn/dow…
kibana:https://www.elastic.co/cn/dow…

其中es的安装教程我在这篇文章里面已经写了：https://segmentfault.com/a/11…，下面介绍logstash和kibana的安装过程

1.logstash:
下载后

##进入安装包所在目录，解压
tar -xf logstash-6.7.1.tar.gz
##切换到bin目录
cd /logstash-6.7.1/bin
##编辑或者创建一个启动的配置文件，到时候用这个配置文件启动
vim input_flter_output.conf

input {
   file{
       path=> "/crawler/jenkins/*.log"    ##生成日志的目录
       type=> "cml"          ##索引的类型
       start_position=> "beginning"     ##一开始就输入原来的日志信息
  }
  stdin{}
}
filter{
 
}
output{
       elasticsearch{
       action=> "index"
       hosts=> "www.iamcrawler.cn:9500"      ##输出到elasticsearch上面
       index=> "log-%{+yyyy.MM.dd}"     ##生成一个log-时间的索引
       }
}
   #最后保存退出
   ##在bin目录下再编写一个启动bat
   vim run.bat
   sh logstash -f input_flter_output.conf  &
    #最后保存退出
   然后运行run.bat即可
   [root@iamcrawler bin]# sh run.bat
   ##正常会出现以下情况：
    [root@iamcrawler bin]# Sending Logstash logs to /crawler/logstash/logstash-6.7.1/logs which is now configured via log4j2.properties
    [2019-04-09T13:11:40,120][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [2019-04-09T13:11:40,138][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.7.1"}
    [2019-04-09T13:11:50,041][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
    [2019-04-09T13:11:50,697][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://www.iamcrawler.cn:9500/]}}
    [2019-04-09T13:11:51,065][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://www.iamcrawler.cn:9500/"}
    [2019-04-09T13:11:51,191][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
    [2019-04-09T13:11:51,196][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
    [2019-04-09T13:11:51,232][INFO ][logstash.outputs.elasticsearch] Using default mapping template
    [2019-04-09T13:11:51,253][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//www.iamcrawler.cn:9500"]}
    [2019-04-09T13:11:51,287][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
    [2019-04-09T13:11:51,399][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
    [2019-04-09T13:11:51,783][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/crawler/logstash/logstash-6.7.1/data/plugins/inputs/file/.sincedb_6677650ec826fa62a735f6625357dead", :path=>["/crawler/jenkins/*.log"]}
    [2019-04-09T13:11:51,896][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x729fdee9 run>"}
    [2019-04-09T13:11:52,010][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
    [2019-04-09T13:11:52,033][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    [2019-04-09T13:11:52,723][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

2.kibana的安装
kibana安装比较简单，下载后，进入安装目录的config目录，如：
cd /crawler/kibana/kibana-6.7.1-linux-x86_64/config

vim kibana.yml
##添加如下命令
server.host: 0.0.0.0
elasticsearch.url: "http://localhost:9500" #这里是es的http地址
##进入kibana目录，执行以下命令，后台运行kibana
./bin/kibana &

二.kibana的使用可以参照网上的很多教程，这里就不过多的描述了

    原文作者：crawler
    原文地址: https://segmentfault.com/a/1190000018806846
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞