如何使用Spark分析網(wǎng)站日志，相信很多沒(méi)有經(jīng)驗(yàn)的人對(duì)此束手無(wú)策，為此本文總結(jié)了問(wèn)題出現(xiàn)的原因和解決方法，通過(guò)這篇文章希望你能解決這個(gè)問(wèn)題。

創(chuàng)新互聯(lián)網(wǎng)站建設(shè)提供從項(xiàng)目策劃、軟件開(kāi)發(fā)，軟件安全維護(hù)、網(wǎng)站優(yōu)化(SEO)、網(wǎng)站分析、效果評(píng)估等整套的建站服務(wù)，主營(yíng)業(yè)務(wù)為成都網(wǎng)站設(shè)計(jì)、網(wǎng)站建設(shè)，成都app軟件開(kāi)發(fā)公司以傳統(tǒng)方式定制建設(shè)網(wǎng)站，并提供域名空間備案等一條龍服務(wù)，秉承以專(zhuān)業(yè)、用心的態(tài)度為用戶(hù)提供真誠(chéng)的服務(wù)。創(chuàng)新互聯(lián)深信只要達(dá)到每一位用戶(hù)的要求，就會(huì)得到認(rèn)可，從而選擇與我們長(zhǎng)期合作。這樣，我們也可以走得更遠(yuǎn)！

郁悶從昨天開(kāi)始個(gè)人網(wǎng)站不斷的發(fā)出告警504錯(cuò)誤，登錄機(jī)器看了一下是php-fpm報(bào)錯(cuò)，這個(gè)錯(cuò)誤重啟php-fpm后，幾個(gè)小時(shí)就告警，快一年了都沒(méi)什么問(wèn)題，奇怪

[28-Sep-2016 11:53:19] NOTICE: ready to handle connections
[28-Sep-2016 11:53:19] NOTICE: systemd monitor interval set to 10000ms
[28-Sep-2016 11:53:26] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[28-Sep-2016 13:46:35] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[28-Sep-2016 13:49:32] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

以為是這個(gè)值設(shè)置的太小了，所以修改了配置修改大了值

[28-Sep-2016 15:51:43] NOTICE: fpm is running, pid 28179
[28-Sep-2016 15:51:43] NOTICE: ready to handle connections
[28-Sep-2016 15:51:43] NOTICE: systemd monitor interval set to 10000ms
[28-Sep-2016 15:52:12] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 7 total children
[28-Sep-2016 16:15:58] WARNING: [pool www] server reached pm.max_children setting (20), consider raising it
[28-Sep-2016 16:52:32] WARNING: [pool www] server reached pm.max_children setting (20), consider raising it
[28-Sep-2016 16:53:05] WARNING: [pool www] server reached pm.max_children setting (20), consider raising it
[28-Sep-2016 16:55:17] WARNING: [pool www] server reached pm.max_children setting (20), consider raising it

結(jié)果后來(lái)還是一樣，幾個(gè)小時(shí)之后再次504告警，再看nginx的日志，發(fā)現(xiàn)一些奇怪的ip訪問(wèn)量非常大。。。有懷疑是有惡意ip的訪問(wèn)，看來(lái)有必要查查訪問(wèn)日志中的ip訪問(wèn)量

root@iZ28bhfjhgkZ:/var/log/nginx# vim access.log
121.42.53.180 - - [25/Sep/2016:06:26:29 +0800] "POST /wp-cron.php?doing_wp_cron=1474755989.0131719112396240234375 HTTP/1.0" 499 0 "-" "WordPress/4.3.1; http://zhwen.org"
182.92.148.207 - - [25/Sep/2016:06:26:29 +0800] "GET / HTTP/1.1" 200 41253 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)"
203.208.60.226 - - [25/Sep/2016:06:28:55 +0800] "GET /?p=675 HTTP/1.1" 200 8204 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.226 - - [25/Sep/2016:06:28:57 +0800] "GET /wp-content/themes/sparkling/inc/css/font-awesome.min.css?ver=4.3.1 HTTP/1.1" 200 26711 "http://zhwen.org/?p=675" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.226 - - [25/Sep/2016:06:28:57 +0800] "GET /wp-content/plugins/wp-pagenavi/pagenavi-css.css?ver=2.70 HTTP/1.1" 200 374 "http://zhwen.org/?p=675" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.208.60.226 - - [25/Sep/2016:06:28:58 +0800] "GET /wp-content/plugins/yet-another-related-posts-plugin/style/widget.css?ver=4.3.1 HTTP/1.1" 200 771 "http://zhwen.org/?p=675" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
121.43.107.174 - - [25/Sep/2016:06:29:18 +0800] "GET / HTTP/1.1" 200 41253 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)"
115.28.189.208 - - [25/Sep/2016:06:29:33 +0800] "GET / HTTP/1.1" 200 41253 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)"
42.156.139.59 - - [25/Sep/2016:06:30:58 +0800] "GET /?paged=14 HTTP/1.1" 200 11164 "-" "YisouSpider"
182.92.148.207 - - [25/Sep/2016:06:31:29 +0800] "GET / HTTP/1.1" 200 41253 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)"
61.135.169.81 - - [25/Sep/2016:06:34:14 +0800] "GET /?p=articles/cscope-tags HTTP/1.1" 200 10681 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12) AppleWebKit/602.1.50 (KHTML, like Gecko)"
61.135.169.81 - - [25/Sep/2016:06:34:14 +0800] "GET /apple-touch-icon-precomposed.png HTTP/1.1" 404 151 "-" "Safari/12602.1.50.0.10 CFNetwork/807.0.4 Darwin/16.0.0 (x86_64)"

所以對(duì)訪問(wèn)日志的ip做了一個(gè)簡(jiǎn)單統(tǒng)計(jì)：
1）先把ip取出來(lái)（為了減少數(shù)據(jù)量，其實(shí)也可以直接壓縮后下載到本地），再下載到本地
root@iZ28bhfjhgkZ:/var/log/nginx# cat access.log|awk ‘{print $1}’ > tt

在sparkshell中執(zhí)行下面的代碼：

val line = sc.textFile("/data1/data/t1")
line.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
.map(e => (e._2, e._1)).reduceByKey(_+","+_)
.sortByKey(true,1).saveAsTextFile("/data1/data/t3")

2）最后的結(jié)果t3的內(nèi)容如下，發(fā)現(xiàn)這幾個(gè)ip的訪問(wèn)量非常大，尤其

191.96.249.53
。。。。。
(855,182.92.148.207)
(3100,121.8.136.75)
(3889,61.135.169.81)
(53513,191.96.249.53)

3）再搞一個(gè)iptables限制，搞定。spark做這種統(tǒng)計(jì)分析還是非常簡(jiǎn)單的，就是一行代碼搞定分析。

root@iZ28bhfjhgkZ:/var/log# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        
root@iZ28bhfjhgkZ:/var/log# iptables -A INPUT -s 191.96.249.53 -j DROP
root@iZ28bhfjhgkZ:/var/log# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        
DROP       all  --  DEDICATED.SERVER     anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination        
root@iZ28bhfjhgkZ:/var/log# 

看完上述內(nèi)容，你們掌握如何使用Spark分析網(wǎng)站日志的方法了嗎？如果還想學(xué)到更多技能或想了解更多相關(guān)內(nèi)容，歡迎關(guān)注創(chuàng)新互聯(lián)行業(yè)資訊頻道，感謝各位的閱讀！