Logstash基础操作-Filter-创新互联
Grok配置案例:
成都创新互联公司服务项目包括宿豫网站建设、宿豫网站制作、宿豫网页制作以及宿豫网络营销策划等。多年来,我们专注于互联网行业,利用自身积累的技术优势、行业经验、深度合作伙伴关系等,向广大中小型企业、政府机构等提供互联网行业的解决方案,宿豫网站推广取得了明显的社会效益与经济效益。目前,我们服务的客户以成都为中心已经辐射到宿豫省份的部分城市,未来相信会继续扩大服务区域并继续获得客户的支持与信任!##启动文件配置: # Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { stdin{} } filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] } } output { stdout{ codec => "rubydebug" } } ##输出文件内容 172.16.213.132 [07/Feb/2018:16:24:19 +0800] "GET / HTTP/1.1" 403 5039 ##显示内容 { "@version" => "1", "@timestamp" => 2019-11-10T06:02:42.865Z, "host" => "localhost.localdomain", "message" => "172.16.213.132 [07/Feb/2018:16:24:19 +0800] \"GET / HTTP/1.1\" 403 5039", "timestamp" => "07/Feb/2018:16:24:19 +0800", "bytes" => "5039", "response" => "403", "clientip" => "172.16.213.132", "referrer" => "\"GET / HTTP/1.1\"" }Grok 过滤重复字段
## 配置文件 # Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { stdin{ } } filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } } output { stdout{ codec => "rubydebug" } }Grok搭配Date时间插件配置
# Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { stdin{ } } filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } } output { stdout{ codec => "rubydebug" } }Date 过滤重复得字段配置
# Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { stdin{ } } filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate { remove_field => [ "timestamp" ] } } output { stdout{ codec => "rubydebug" } }综合练习配置参数
# Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { stdin{ } } filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ rename => {"response" => "response_new"} gsub => ["referrer", "\"", ""] remove_field => [ "timestamp" ] split => ["clientip", "."] } } output { stdout{ codec => "rubydebug" } }Geoip 地理位置插件操作方式
# Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { stdin{ } } filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ remove_field => [ "timestamp" ] } geoip { source => "clientip" database => "/usr/local/include/GeoLite2-ASN_20191105/GeoLite2-ASN.mmdb" } } output { stdout{ codec => "rubydebug" } }Geoip输出指定属性值
# Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { stdin{ } } filter { grok { match => ["message","%{IP:clientip}\ \[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\ %{NUMBER:response}\ %{NUMBER:bytes}"] remove_field => ["message"] } date { match => ["timestamp", "dd/MMMM/yyyy:HH:mm:ss Z"] } mutate{ remove_field => [ "timestamp" ] } geoip { source => "clientip" #database => "/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb" database => "/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb" fields => ["city_name", "region_name", "country_name", "ip", "latitude", "longitude", "timezone"] } } output { stdout{ codec => "rubydebug" } } 模拟数据: 36.7.152.182 [07/Feb/2018:16:24:19 +0800] "GET / HTTP/1.1" 403 5039综合实战
# Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { stdin{} } filter{ grok{ match => {"message" => "%{TIMESTAMP_ISO8601:localtime}\|\~\|%{IP:clientip} \|\~\|%{GREEDYDATA:http_user_agent}\|\~\|%{GREEDYDATA:url} \|\~\|%{GREEDYDATA:mediaid}\|\~\|%{GREEDYDATA:osid}"} remove_field => [ "message" ] } date { match => ["localtime", "yyyy-MM-dd'T'HH:mm:ssZZ"] target => "@timestamp" } mutate { remove_field => ["localtime"] } geoip { source => "clientip" #database => "/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb" database => "/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb" fields => ["city_name", "region_name", "country_name", "ip", "latitude", "longitude", "timezone"] } } output { stdout { codec => "rubydebug" } } 示例:2018-02-09T10:57:42+08:00|~|123.87.240.97|~|Mozilla/5.0 (iPhone;CPU iPhone OS 11_2_2 like Mac OS X) AppleWebKit/604.4.7 Version/11.0 Mobile/15C202 Safari/604.1 |~|http://m.sina.cn/cm/ads_ck_wap.html |~|12434785489009|~|DF45566587855P另外有需要云服务器可以了解下创新互联cdcxhl.cn,海内外云服务器15元起步,三天无理由+7*72小时售后在线,公司持有idc许可证,提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案,具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势,专为企业上云打造定制,能够满足用户丰富、多元化的应用场景需求。
网站题目:Logstash基础操作-Filter-创新互联
文章源于:http://scyanting.com/article/dcepoj.html