宝塔屏蔽垃圾搜索引擎蜘蛛以及采集扫描工具教程（网页显示宝塔防火墙）

目前除了我们常见的搜索引擎如百度、Google、Sogou、360等搜索引擎之外，还存在其他非常多的搜索引擎，通常这些搜索引擎不仅不会带来流量，因为大量的抓取请求，还会造成主机的CPU和带宽资源浪费，屏蔽方法也很简单，按照下面步骤操作即可，原理就是分析指定UA然后屏蔽。

宝塔面板下使用方法如下：1、找到文件目录/www/server/nginx/conf文件夹下面，新建一个文件命名：agent_deny.conf 你也可以随意起名，创建完文件后，点击编辑这个文件，把下面的代码放进去保存。

（天辰重新收集整理，是为止目前也是最全的，最完善的代码）#禁止Scrapy等工具的抓取 " N0 @. s' Y' H8 ^ if ($http_user_agent ~* (Scrapy|Curl|HttpClient|crawl|curb|git|Wtrace)) { @5 n) x7 s: c' c- [7 A & G h! O0 G$ S' }( a4 i& N return 403;& w" |' q( t& u' z/ w. R 9 r, c/ u( t9 }$ g8 Q5 u0 C. B# q% J } 3 F& Q5 \# U" r3 B( ^ G* t( D1 J( L0 x _* [7 k$ Z #禁止指定UA及UA为空的访问) ?9 Q' V+ ^( k# Q+ @- d if ($http_user_agent ~* "CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|SemrushBot|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|Jullo|Feedly|YandexBot|oBot|FlightDeckReports|Linguee Bot|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms|^$" ) { : ?8 k- v* x$ D- Q6 E, R! ]' ^7 _9 U0 F return 403; Y9 r( P" S0 E4 r# b 2 R4 c7 F! _9 I5 h. b' ^ } 5 c/ C3 e4 h! {1 I' B6 H1 Y, D. H #禁止非GET|HEAD|POST方式的抓取 , w" b0 ]: `2 w8 w if ($request_method !~ ^(GET|HEAD|POST)$) {; d2 s" D0 F5 B$ D ! l7 n9 E. L* `+ m9 X/ H- @( e return 403;- e* g5 E3 T6 n7 L# C2 D) s) K: ` / Q6 ~$ c1 m3 ~ }4 e& R7 A" y9 r' k* Y

2、找到【网站】-【设置】点击左侧 【配置文件】选项卡，在第7-8行左右 插入代码：

include agent_deny.conf;* i& a+ _0 _0 O0 ], s 添加完毕后保存，重启nginx即可，这样这些蜘蛛或工具扫描网站的时候就会提示403禁止访问注意：如果你网站使用火车头采集发布，使用以上代码会返回403错误，发布不了的。

如果想使用火车头采集发布，请使用下面的代码：#禁止Scrapy等工具的抓取# y, _, A( V5 D if ($http_user_agent ~* (Scrapy|Curl|HttpClient|crawl|curb|git|Wtrace)) { 8 |: e+ q* R9 a ) V( ^/ Q5 W4 k* m- z return 403;2 u: k$ w$ }: l0 N* M A 8 e- X- P. ?8 T5 |; \) ` }& B" H W; \* P$ j7 d8 \% M' ~6 ] ' i% A' {$ m) z J% R #禁止指定UA及UA为空的访问; }# s# t2 Z3 Y: Q if ($http_user_agent ~* "CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|SemrushBot|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|Jullo|Feedly|YandexBot|oBot|FlightDeckReports|Linguee Bot|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms ) {" V! p/ f5 m& r5 m! C / j( Q t: @5 \ return 403;/ m) v$ d) Z7 ?, @" @& v; j $ U: E2 o; ?! D( O/ {9 r( H) x I, r o } 1 L s+ C4 z8 T: c0 s l7 H+ x) v& h* K5 r8 x6 E6 t #禁止非GET|HEAD|POST方式的抓取$ a% b& G/ j$ F! h if ($request_method !~ ^(GET|HEAD|POST)$) { 1 ~7 @2 l C6 {3 V* V7 J V2 R$ _! f- j- l, A; n return 403; / t3 |5 D5 ?4 n5 t: Y! {$ [! B8 `6 h # J( j* j: p' L0 X1 ^7 ^+ ]1 p) ` }

设置完了可以用模拟爬去来看看有没有误伤了好蜘蛛，说明：以上屏蔽的蜘蛛名不包括以下常见的6大蜘蛛名：百度蜘蛛：Baiduspider谷歌蜘蛛：Googlebot必应蜘蛛：bingbot搜狗蜘蛛：Sogou web spider

360蜘蛛：360Spider神马蜘蛛：YisouSpider爬虫常见的User-Agent如下：FeedDemon 内容采集 3 |! ^+ a) b# A0 _ BOT/0.1 (BOT for JCE) sql注入 : W- v9 J* x: D" D! q# B CrawlDaddy sql注入8 l6 b5 J& j3 C* C$ n! D Java 内容采集 : S/ [; `3 Z5 H$ W Jullo 内容采集 7 q) @) \ M" V. h6 W Feedly 内容采集 " F' j& c1 e* s! c# v1 B' A. y" } UniversalFeedParser 内容采集 + r3 L" i" C( D( X! M8 x1 [6 p0 ^ ApacheBench cc攻击器 : I. M: B) y2 z4 S* r Swiftbot 无用爬虫! U7 b7 L. y& |* b( I q YandexBot 无用爬虫 ' m$ ?, Z- Y' w5 d, ]* [( J AhrefsBot 无用爬虫/ \- c- Q$ ]$ p. z& ? jikeSpider 无用爬虫 ! X& A* p, X6 x5 S8 N! ~$ Q9 y MJ12bot 无用爬虫 / |, ^5 B+ Y$ w! \* z6 ` ZmEu phpmyadmin 漏洞扫描8 ~) r; p, ]2 Z$ B6 m! O( P' D+ R0 | WinHttp 采集cc攻击% m2 V( K- Y5 m9 D) I5 w: ]. j EasouSpider 无用爬虫% g o- T4 j+ }9 R1 h, a HttpClient tcp攻击 ( _1 q- i+ w8 } Microsoft URL Control 扫描 , p; x7 Y* |0 u5 O YYSpider 无用爬虫$ D6 I3 J3 K9 f jaunty wordpress爆破扫描器 $ w. Y5 ^0 S) {* J6 D# }- d+ m oBot 无用爬虫 5 o' U2 E/ [, z6 l' c" ]* W Python-urllib 内容采集4 n# S5 A2 ^( O Indy Library 扫描 % A' D, ~) B2 @: i" N! U FlightDeckReports Bot 无用爬虫 & p9 E9 U7 H% P Linguee Bot 无用爬虫

来源：BT宝塔屏蔽垃圾搜索引擎蜘蛛以及采集扫描工具教程