宝塔屏蔽垃圾搜索引擎蜘蛛以及采集扫描工具教程（网页显示宝塔防火墙）

目前除了我们常见的搜索引擎如百度、Google、Sogou、360等搜索引擎之外，还存在其他非常多的搜索引擎，通常这些搜索引擎不仅不会带来流量，因为大量的抓取请求，还会造成主机的CPU和带宽资源浪费，屏蔽方法也很简单，按照下面步骤操作即可，原理就是分析指定UA然后屏蔽。

宝塔面板下使用方法如下：1、找到文件目录/www/server/nginx/conf文件夹下面，新建一个文件命名：agent_deny.conf 你也可以随意起名，创建完文件后，点击编辑这个文件，把下面的代码放进去保存。

（天辰重新收集整理，是为止目前也是最全的，最完善的代码）#禁止Scrapy等工具的抓取 6 s+ }- O$ N% o) L! k( A4 z if ($http_user_agent ~* (Scrapy|Curl|HttpClient|crawl|curb|git|Wtrace)) {+ d3 A% u* C$ E! v 2 D3 P- o0 @4 Y' s- [ return 403;* B! `. [4 E$ n1 q4 u L2 ]% G. w8 l- A }" p' j6 q# f, N! I3 s % X1 n6 |* Y" _/ v* H0 z% k #禁止指定UA及UA为空的访问; B' S, P: G9 [ if ($http_user_agent ~* "CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|SemrushBot|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|Jullo|Feedly|YandexBot|oBot|FlightDeckReports|Linguee Bot|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms|^$" ) { 7 _# X; n m+ ]: l/ y" _ ]1 v7 a! h" H& W return 403; % I: n6 n* Q5 d# P; j- V9 ]5 A/ U2 ]) ?# ~: m) q) M* m }) o, F/ d% c! l9 Q! h2 O5 Y # \" N7 [8 J# U: Q" S" A #禁止非GET|HEAD|POST方式的抓取6 L: I5 _ z/ _ if ($request_method !~ ^(GET|HEAD|POST)$) {) w# {1 a% @6 ?# G e ' Y6 ^- Z1 U" [/ X- o return 403; 8 \1 z' c# z& R K% o% M. D& a7 |1 p } * E4 S, t. N$ E

2、找到【网站】-【设置】点击左侧 【配置文件】选项卡，在第7-8行左右 插入代码：

include agent_deny.conf; : N" y3 |$ i* ~9 G% F2 g 添加完毕后保存，重启nginx即可，这样这些蜘蛛或工具扫描网站的时候就会提示403禁止访问注意：如果你网站使用火车头采集发布，使用以上代码会返回403错误，发布不了的。

如果想使用火车头采集发布，请使用下面的代码：#禁止Scrapy等工具的抓取 ' K+ G1 h3 w- ~) S* z+ w if ($http_user_agent ~* (Scrapy|Curl|HttpClient|crawl|curb|git|Wtrace)) { + b0 f# G2 j8 a0 I/ f8 o6 { , q5 E5 R7 \4 N$ ]) k P$ c return 403;( \5 L0 o z) j. O* V* d# y - b8 L ^+ }0 ]* W8 k } " l; F t" C2 {. a: V0 }* V( j 5 I/ K2 w- p( N; k ~ #禁止指定UA及UA为空的访问 6 G% P7 e- i) n1 g6 n6 v4 | if ($http_user_agent ~* "CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|SemrushBot|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|Jullo|Feedly|YandexBot|oBot|FlightDeckReports|Linguee Bot|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms ) {4 k% O# ~$ U- n2 w7 }% a ( C" s# r; m- d( ]- \: H7 Q return 403; $ z+ a" ^0 A: l | ) D- s' f- W6 M0 f, ?. W$ w }, e6 h; W4 T( L5 v # Q) x8 t/ p: h7 @6 ~4 H- R #禁止非GET|HEAD|POST方式的抓取: H, A. f9 w" K6 B/ a) B if ($request_method !~ ^(GET|HEAD|POST)$) {4 k; T; Y% S( P$ I/ C 6 \$ g% }8 {6 l6 |7 |1 I# T/ ]5 n0 V return 403; 1 |' q7 t4 u9 s7 g * y7 q0 C* c2 v+ G* b. ~9 \ }

设置完了可以用模拟爬去来看看有没有误伤了好蜘蛛，说明：以上屏蔽的蜘蛛名不包括以下常见的6大蜘蛛名：百度蜘蛛：Baiduspider谷歌蜘蛛：Googlebot必应蜘蛛：bingbot搜狗蜘蛛：Sogou web spider

360蜘蛛：360Spider神马蜘蛛：YisouSpider爬虫常见的User-Agent如下：FeedDemon 内容采集 + M* w6 l7 d+ w; E BOT/0.1 (BOT for JCE) sql注入 - y; \( e5 `2 d. b d CrawlDaddy sql注入 ( i6 G; f" j& j+ t8 x Java 内容采集* S# S. C) I/ o7 } Jullo 内容采集7 @; C6 j3 @& `7 x* x- X Feedly 内容采集" W2 R1 a/ [% b% F, G; G2 {8 J UniversalFeedParser 内容采集 - y L! h x1 p/ K- Q# h) e+ Y2 Q ApacheBench cc攻击器( Y3 k: D$ m8 B7 K9 _/ Y0 [- O) N Swiftbot 无用爬虫 6 q7 z1 J, G! t b( i. | YandexBot 无用爬虫! l* u. ?" ?/ P/ Q9 u* Y AhrefsBot 无用爬虫: T1 P3 C% l$ ? jikeSpider 无用爬虫 . G5 v2 S- Z$ ?. d MJ12bot 无用爬虫 $ P7 Z7 T0 I" x$ v. ?+ F ZmEu phpmyadmin 漏洞扫描9 m+ E2 Y' W6 F( p% U5 V WinHttp 采集cc攻击9 T, V( h' X |/ d* l EasouSpider 无用爬虫 ( @$ Y* N0 B, I6 P$ J) z7 Q HttpClient tcp攻击3 t$ y' d0 J9 x8 A6 L Microsoft URL Control 扫描1 c# @$ X/ d0 j% f) f* ]$ K0 t- I YYSpider 无用爬虫 + h$ t. L! f8 A jaunty wordpress爆破扫描器* ~& M% |0 ]8 u oBot 无用爬虫3 o5 \7 U' ]2 Y; `# Z# x Python-urllib 内容采集) S3 [* g0 C1 d& x3 Z* D Indy Library 扫描 8 k2 B6 k [2 R) |$ {1 f. j FlightDeckReports Bot 无用爬虫 ! H, |& [6 n B7 B+ T8 ^; c. ?6 N Linguee Bot 无用爬虫

来源：BT宝塔屏蔽垃圾搜索引擎蜘蛛以及采集扫描工具教程