Logstash简介

Logstash是一个接收,处理,转发日志的工具。支持系统日志,webserver日志,错误日志,应用日志,总之包括所有可以抛出来的日志类型。

依赖环境

Logstash运行仅仅依赖java运行环境(jre),在安装前需先部署好java环境,可用java -version命令检查本地环境。

​​‌‌​​​‌‌​‌​​‌‌‍​‌​‌‌‌​​‌‌‌‌​‌​‍​‌​​‌​​​‌​​​‌‌​‍​‌​‌‌​​​‌‌​​​​​‍​​‌​‌‌‌‌‌‌‌‌​​​‍​‌‌​​‌‌‌​‌‌​​‌‌‌‍​‌‌​​​‌‌‌​​​‌​‌‍​​‌‌‌‌‌‌‌‌​​‌‌‍‌​‌‌​​‌‌‍‌​​‌​​​​‍‌​​‌‌​​​‍‌​​​‌‌​​‍‌​​​‌​‌‌‍‌​​‌‌‌‌​‍‌​​​‌‌​​‍‌​​‌​‌‌‌‍​​​‌​​‌​‌‌‌‌​‌‌‍​​​​‌​​​‌‌‌‌‌‌‌‍​‌​‌‌​​‌​‌​‌​‌​‍​‌‌​‌‌‌​‌​‌‌​​‌​‍​​​​​​​‌​​‌​​​‌‍​‌‌​​​‌‌‌‌‌​​​‌‍​‌‌​​​​‌​​​​​​​‍​​​‌​‌​‌‌​‌​‌‌‌‍​​‌‌‌‌‌‌‌‌​​‌​‍​​​​​​​​‌‌‌‌​​‌‌‍​​​‌​‌​‌‌​​‌‌‌​‍‌​​‌‌‌‌​‍‌​​‌‌​‌‌‍‌​​‌​​‌​‍‌​​‌​‌‌​‍‌​​‌​​​‌‍​‌‌​​​‌​‌‌‌​​​‌‍‌‌​​‌‌​‌‍‌‌​​‌‌‌‌‍‌‌​​‌‌‌​‍‌‌​​​‌‌‌‍‌‌​‌​​‌​‍‌‌​​‌‌‌​‍‌‌​​‌‌‌‌‍‌‌​‌​​‌​‍‌‌​​‌‌​‌‍‌‌​​‌​‌​‍​‌​‌‌​‌‌‌‌​​‌​​‍​‌‌​​​​‌​‌​​​‌‌‍​​​​​​​​‌‌‌‌​​‌‌‍​‌​‌‌​​​‌‌​​​​​‍​​‌‌​‌​​‌‌‌‌​​​‍​‌​‌​​​‌‌​​‌‌‌‌‍​‌​‌​​​‌​‌‌‌‌‌‌‍​​​​​​​​‌‌‌​​‌​‌‍‌​​‌​‌‌‌‍‌​​​‌​‌‌‍‌​​​‌​‌‌‍‌​​​‌‌‌‌‍‌​​​‌‌​​‍‌‌​​​‌​‌‍‌​‌​​​‌‌‍‌​‌​​​‌‌‍‌​​‌​‌‌​‍‌​​‌​‌​​‍‌​​‌​‌‌​‍‌​​​‌​​​‍‌​​‌​‌‌​‍‌‌​‌​​​‌‍‌​​‌​​‌​‍‌​​‌‌​‌​‍‌​‌​​​‌‌‍‌​​‌‌‌‌​‍‌​​​‌‌​‌‍‌​​‌‌‌​​‍‌​​‌​‌‌‌‍‌​​‌​‌‌​‍‌​​​‌​​‌‍‌​​‌‌​‌​‍‌​​​‌‌​​‍‌​‌​​​‌‌‍‌‌​​‌​‌​‍‌‌​​​‌‌​‍‌‌​​‌‌‌​‍‌‌​‌​​​‌‍‌​​‌​‌‌‌‍‌​​​‌​‌‌‍‌​​‌​​‌​‍‌​​‌​​‌‌

注意:Logstash 需要 Java 8 ,目前不支持 Java 9 。

安装过程

Ⅰ、在线安装

可以使用官方仓库进行安装。

1. APT管理器

  • Debian 平台,以Ubuntu16.04为例
wget -O - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
cat>>/etc/apt/sources.list <<'EOF'
deb https://artifacts.elastic.co/packages/6.x/apt stable main
EOF
apt-get install apt-transport-https
apt-get update
apt-get install logstash

2. YUM管理器

  • Redhat 平台,以CentOS7.4为例
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat>/etc/yum.repos.d/logstash.repo<<'EOF'
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
yum clean all
yum install logstash -y

Ⅱ、离线安装

目前可以从 http://www.elasticsearch.org/overview/elkdownloads/ 下载这两个包的源代码或者二进制版本。

1. RPM包(RedHat体系)

  • Redhat 平台
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.3.rpm
rpm -ivh logstash-6.4.3.rpm

2. DEB包(Debian体系)

  • Debian 平台
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.3.deb
sudo dpkg -i logstash-6.4.3.deb 

Ⅲ、二进制安装

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.3-linux-x86_64.tar.gz
tar -zxvf logstash-6.4.3-linux-x86_64.tar.gz -d /opt/logstash

启动与运行

使用下面的命令启动logstash:

$ bin/logstash -e 'input{stdin{}} output{stdout{}}'

在终端输入 helloworld,回车看返回结果:

2018-09-10T19:11:54.751+0000 localhost helloworld

继续输入:

$ bin/logstash -e 'input{stdin{}} output{stdout{codec=>rubydebug}}'

在终端输入 Hello World,回车看返回结果:

{
       "message" => "Hello World",
      "@version" => "1",
    "@timestamp" => "2018-09-10T19:14:16.105Z",
          "host" => "localhost"
}

简单配置举例

以收集nginx日志为例。(提前搭建好elk环境)

  • 安装nginx
[root@localhost ~]# rpm -ivh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
[root@localhost ~]# yum install nginx -y
[root@localhost ~]# service nginx start
  • 配置nginx
[root@localhost conf.d]# vim /etc/nginx/conf.d/elk-nginx.conf
 
server {
 listen 80;
 server_name _;
 
location / {
    proxy_pass http://172.16.2.100:5601;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 }
 access_log /tmp/elk_access.log main;
 }
 
[root@localhost conf.d]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
 
[root@localhost conf.d]# service nginx restart
  • 配置logstash
[root@localhost conf.d]# vim /etc/logstash/conf.d/elk-nginx.conf
input {
 file {
 path => "/tmp/elk_access.log"
 start_position => "beginning"
 type => "nginx"
 }
}
filter {
 grok {
 match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
 }
 geoip {
 source => "clientip"
 }
}
output {
 stdout { codec => rubydebug }
 elasticsearch {
 hosts => ["172.16.2.100:9200"]
 index => "nginx-test-%{+YYYY.MM.dd}"
 }
}
  • 检查配置文件并重启服务
[root@localhost conf.d]# cd /opt/logstash/bin
[root@localhost bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/elk-nginx.conf --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
[root@localhost ~]# systemctl restart logstash
  • 检查es是否生成引索
[root@localhost elasticsearch]# curl '172.16.2.100:9200/_cat/indices?v'
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   %{[@metadata][beat]}-2018.10.23 8qC9mHcMSvCpiZibrHFLnQ   5   1        134            0    227.1kb        227.1kb
yellow open   nginx-test-2018.10.23           odAimBA9QUSsO0dhIfa6YQ   5   1        134            0    182.5kb        182.5kb
green  open   .kibana                         gaJns019QVy3ojdU-o6ECw   1   0          2            0     10.3kb         10.3kb
  • 访问kibana界面,已经成功获取到日志:
文章目录