如何使用Elastic+logstash+filebeat做Nginx日志分析
这篇文章将为大家详细讲解有关如何使用Elastic+logstash+filebeat做Nginx日志分析,小编觉得挺实用的,因此分享给大家做个参考,希望大家阅读完这篇文章后可以有所收获。
鲤城ssl适用于网站、小程序/APP、API接口等需要进行数据传输应用场景,ssl证书未来市场广阔!成为成都创新互联公司的ssl证书销售渠道,可以享受市场价格4-6折优惠!如果有意向欢迎电话联系或者加微信:13518219792(备注:SSL证书合作)期待与您的合作!
一、Elasticserach安装
1、Installation(elastic 6.3.2 版本 依赖 java JDK8)
下载合适的版本:
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
tar -xvf elasticsearch-6.4.0.tar.gz
cd elasticsearch-6.4.0/bin
./elasticsearch
2、集群健康检查
curl -XGET http://127.0.0.1:9200/_cat/health?v 集群健康查看
curl -XGET http://127.0.0.1:9200/_cat/nodes?v 节点状态查看
curl -XGET http://127.0.0.1:9200/_cat/indices?v 查看索引
curl -XPUT http://127.0.0.1:9200/customer?pretty 添加customer索引
curl -XDELETE http://127.0.0.1:9200/customer?pretty 删除索引
3、Config
cluster.name: 集群名
node.name: 节点名
path.data: 数据存储路径
path.logs: 日志存储路径
network.host: 监听地址
http.port: 监听端口
JVM 配置、logging配置 参考官方文档
重要参数:
Path settings
Cluster name
Node name
Network host
Discovery settings
Heap size
Heap dump path
GC logging
Temp directory
重要的系统参数:
Disable swapping
Increase file descriptors
Ensure sufficient virtual memory
Ensure sufficient threads
JVM DNS cache settings
4、Running as daemon
./bin/elasticsearch -d -p pid 启动
kill cat pid
停止
5、Set up X-Pack
参考x-pack破解方法
二、kibana安装及配置
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz
tar -xzf kibana-6.4.0-linux-x86_64.tar.gz
cd kibana-6.4.0-linux-x86_64/
1、启动 ./bin/kibana 2、config:
server.port: 5601
server.host: "192.168.12.81"
kibana.index: ".kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "dinpay"
三、logstash安装及配置
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz
配置 filebeat 收集Nginx日志: (Nginx.conf)
input {
beats {
port => 5044
codec => "json"
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "test1-nginx-access-%{+YYYY.MM.dd}"
template_overwrite => true
user => elastic
password => dinpay
}
}
配置logstash.yml elastic安装在本机(没有配置https)
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: dinpay
xpack.monitoring.elasticsearch.url: ["http://127.0.0.1:9200"]
四、filebeat安装及配置
wget https://artifacts.elastic.co/downloads/beats/filebeat/https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.0-linux-x86_64.tar.gz
tar -xf filebeat-6.4.0-linux-x86_64.tar.gz
cd filebeat-6.4.0-linux-x86_64
配置 filebeat.yml
type: log
paths:/var/log/nginx/access.log
output.logstash:
hosts: ["192.168.12.81:5044"]
xpack.monitoring:
enabled: true
elasticsearch:
hosts: ["http://192.168.12.81:9200", "http://192.168.12.81:9200"]
username: elastic
password: dinpay
五、Nginx 日志格式配置:
log_format json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$clientRealIp",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"http_user_agent":"$http_user_agent",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status",'
'"ss":"$upstream_status"}';
access_log /var/log/nginx/access.log json; map $http_x_forwarded_for $clientRealIp { "" $remote_addr; ~^(?P[0-9\.]+),?.*$ $firstAddr; }
关于“如何使用Elastic+logstash+filebeat做Nginx日志分析”这篇文章就分享到这里了,希望以上内容可以对大家有一定的帮助,使各位可以学到更多知识,如果觉得文章不错,请把它分享出去让更多的人看到。
本文名称:如何使用Elastic+logstash+filebeat做Nginx日志分析
文章网址:http://scyanting.com/article/ppcosp.html