以nginx錯(cuò)誤日志為例，演示日志處理流程

隆陽(yáng)ssl適用于網(wǎng)站、小程序/APP、API接口等需要進(jìn)行數(shù)據(jù)傳輸應(yīng)用場(chǎng)景，ssl證書(shū)未來(lái)市場(chǎng)廣闊！成為創(chuàng)新互聯(lián)建站的ssl證書(shū)銷(xiāo)售渠道，可以享受市場(chǎng)價(jià)格4-6折優(yōu)惠！如果有意向歡迎電話聯(lián)系或者加微信：13518219792（備注：SSL證書(shū)合作）期待與您的合作！

filebeat--logstash--es

filebeat--kafka--logstash--es

#filebeat使用systemd管理 /usr/lib/systemd/system/filebeat.service? [Unit] Description=Filebeat Documentation=http://www.elastic.co Wants=network-online.target After=network-online.target [Service] ExecStart=/usr/local/filebeat/filebeat?-c?/usr/local/filebeat/filebeat.yml Restart=always [Install] WantedBy=multi-user.target

#logstash使用systemd管理 #如果有多個(gè)logstash配置文件,可以使用-f指定目錄 /usr/lib/systemd/system/logstash.service? [Unit] Description=logstash Documentation=http://www.elastic.co Wants=network-online.target After=network-online.target [Service] Environment=JAVA_HOME=/usr/java/jdk1.8.0_211 ExecStart=/usr/local/logstash/bin/logstash?-f?/usr/local/logstash/config/logstash.conf?-l?/usr/local/logstash/logs Restart=always [Install] WantedBy=multi-user.target

#啟動(dòng)nginx容器，映射日志目錄 docker?run?-d?--name=nginx?--net=host?-v?/tmp/nginx_log:/var/log/nginx?nginx

#nginx錯(cuò)誤日志：

2019/09/21 17:00:08 [error] 7#7: *9 open() "/usr/share/nginx/html/api" failed (2: No such file or directory), client: 192.168.3.102, server: localhost, request: "GET /api HTTP/1.1", host: "192.168.3.100"

• filebeat--logstash--es示例

#filebeat輸出logstash示例 /usr/local/filebeat/filebeat.yml? filebeat.inputs: -?type:?log ??paths: ????-?/tmp/nginx_log/error.log ?? ??multiline.pattern:?^\d{4}/\d{2}/\d{2}\s\d{2}:\d{2}:\d{2} ??#匹配nginx日志時(shí)間格式??2019/09/21?17:00:08 ??multiline.negate:?true ??multiline.match:?after ??exclude_files:?[".gz$"] ??tail_files:?true ?? ??#增加輸出字段，tags為數(shù)組形式，fields.id為鍵值對(duì)形式 ??tags:?["nginx-100"] ??fields: ????id:?"nginx-100" output.logstash: ??hosts:?["192.168.3.100:5044","192.168.3.101:5044"] ??loadbalance:?true #輸出到單個(gè)logstash #output.logstash: #??hosts:?["127.0.0.1:5044"]

#logstash輸出到es示例；根據(jù)fileds.id來(lái)劃分索引 /usr/local/logstash/config/logstash.conf? input?{ ??beats?{ ????port?=>?5044 ??} } output?{ ??elasticsearch?{ ????hosts?=>?["http://192.168.3.100:9200","http://192.168.3.101:9200","http://192.168.3.102:9200"] ????index?=>?"%{[fields][id]}-%{+YYYY.MM.dd}" ????user?=>?"elastic" ????password?=>?"HkqZIHZsuXSv6B5OwqJ7" ??} }

• filebeat--kafka--logstash--es示例

#filebeat輸出到kafka示例 /usr/local/filebeat/filebeat.yml? filebeat.inputs: -?type:?log ??paths: ????-?/tmp/nginx_log/error.log ?? ??multiline.pattern:?^\d{4}/\d{2}/\d{2}\s\d{2}:\d{2}:\d{2} ??#匹配nginx日志時(shí)間格式??2019/09/21?17:00:08 ??multiline.negate:?true ??multiline.match:?after ??exclude_files:?[".gz$"] ??tail_files:?true ?? ??#增加輸出字段，tags為數(shù)組形式，fields.id為鍵值對(duì)形式 ??tags:?["nginx-kafka-100"] ??fields: ????id:?"nginx-kafka-100" output.kafka: ??hosts:?["192.168.3.100:9092",?"192.168.3.101:9092",?"192.168.3.102:9092"] ??topic:?'%{[fields.id]}' ??partition.round_robin: ????reachable_only:?false ??required_acks:?1 ??compression:?gzip ??max_message_bytes:?1000000

#kafka輸出到es示例 /usr/local/logstash/config/logstash.conf input?{ ????kafka?{ ??????group_id?=>?"logstash" ??????topics?=>?["nginx-kafka-100"] ??????bootstrap_servers?=>?"192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092" ??????consumer_threads?=>?"1" ??????fetch_max_bytes?=>?"26214400" ??????codec?=>?plain ??} } filter?{ ??json?{ ????source?=>?"message" ??} } output?{ ??elasticsearch?{ ????hosts?=>?["http://192.168.3.100:9200","http://192.168.3.101:9200","http://192.168.3.102:9200"] ????index?=>?"%{[fields][id]}-%{+YYYY.MM.dd}" ????user?=>?"elastic" ????password?=>?"HkqZIHZsuXSv6B5OwqJ7" ??} }

參考：

https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html

https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)cdcxhl.cn，海內(nèi)外云服務(wù)器15元起步，三天無(wú)理由+7*72小時(shí)售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國(guó)服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡(jiǎn)單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢(shì)，專(zhuān)為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場(chǎng)景需求。

本文題目：filebeat日志收集-創(chuàng)新互聯(lián)
當(dāng)前路徑：http://weahome.cn/article/dojhgc.html