实用Nginx和Apache环境屏蔽YisouSpider蜘蛛爬虫方法
|
![]() |
![]() |
![]() |
| 【性价之王】 | 【线路之王】 | 【价格之王】 | 【配置之王】 |
| 【免费之王】 | 【香港首推】 | 【梯子之王】 | 【独服之王】 |
今天晚上有网友找到蜗牛,说自己新搭建的博客还没有多少流量,网站就打开非常慢,需要刷新很久才可以慢慢打开,而且自己使用的是阿里云VPS主机,速度不至于这么慢,于是蜗牛帮助看看到底是什么问题。我们先通过TOP命令看看当前VPS的负载情况,果然占用很高的CPU资源,php-fpm基本上占用CPU在60%+,再去看看日志。
我们看到了什么?有很多的YisouSpider爬虫,这个蜘蛛其实也没有多少用途,是来自阿里旗下的一搜视频和神马搜索蜘蛛,占据很小的搜索流量,我们可以屏蔽掉爬虫。这里我们可以参考前天发布的\”通过设定.htaccess和nginx.conf禁止恶意User Agent网页爬虫\”来解决问题。这里蜗牛单独针对YisouSpider爬虫来解决掉我们的困扰。
第一、Nginx环境网站
如果我们采用的是Nginx环境的网站,就相对简单很多,直接在当前站点的.conf文件中添加脚本。
if ($http_user_agent ~* \”YisouSpider\”) {
return 403;
}
替换之后重启Nginx,我们就可以解决这个问题,只要过来,就返回403.
第二、Apache环境网站
这个也很简单,我们可以直接在.htaccess根目录文件中屏蔽掉来访的YisouSpider蜘蛛爬虫的IP地址。
Order allow,deny
Allow from all
Deny from 42.156.139.54
红色的IP地址就是我们需要屏蔽的YisouSpider爬虫IP地址,这里蜗牛整理到当前大部分的YisouSpider蜘蛛地址,我们都添加进去。
Deny from 42.156.136.95
Deny from 42.156.137.67
Deny from 42.156.137.62
Deny from 42.156.137.63
Deny from 42.156.137.60
Deny from 42.156.138.113
Deny from 42.120.161.37
Deny from 42.120.161.32
Deny from 42.120.161.65
Deny from 42.156.139.95
Deny from 42.156.136.113
Deny from 42.156.136.44
Deny from 42.156.136.42
Deny from 42.156.136.40
Deny from 42.156.139.51
Deny from 42.156.139.53
Deny from 42.156.139.55
Deny from 42.156.139.54
Deny from 42.156.139.57
Deny from 42.156.138.63
Deny from 42.156.138.62
Deny from 42.156.138.61
Deny from 42.156.138.60
Deny from 42.156.138.67
Deny from 42.120.161.62
Deny from 42.156.138.65
Deny from 42.120.160.67
Deny from 42.120.161.95
Deny from 42.120.160.113
Deny from 42.120.160.63
Deny from 42.120.160.62
Deny from 42.120.160.61
Deny from 42.120.160.60
Deny from 42.156.138.81
Deny from 42.156.138.80
Deny from 42.156.138.83
Deny from 42.156.138.82
Deny from 42.156.136.23
Deny from 42.156.137.53
Deny from 42.156.137.51
Deny from 42.156.137.57
Deny from 42.156.138.104
Deny from 42.156.139.5
Deny from 42.156.139.4
Deny from 42.120.161.23
Deny from 42.120.161.24
Deny from 42.156.136.107
Deny from 42.156.137.61
Deny from 42.156.136.50
Deny from 42.156.136.51
Deny from 42.156.136.53
Deny from 42.156.136.54
Deny from 42.156.136.55
Deny from 42.156.136.57
Deny from 42.156.139.42
Deny from 42.156.138.75
Deny from 42.156.139.40
Deny from 42.156.138.77
Deny from 42.156.138.74
Deny from 42.120.160.74
Deny from 42.120.160.75
Deny from 42.120.160.109
Deny from 42.120.160.77
Deny from 42.156.137.40
Deny from 42.156.138.95
Deny from 42.120.160.16
Deny from 42.120.160.17
Deny from 42.120.160.15
Deny from 42.120.160.13
Deny from 42.120.160.10
Deny from 42.120.160.28
Deny from 42.156.139.77
Deny from 42.120.161.57
Deny from 42.156.139.74
Deny from 42.120.161.51
Deny from 42.120.161.53
Deny from 42.120.161.18
Deny from 42.156.136.83
Deny from 42.156.136.27
Deny from 42.120.160.81
Deny from 42.120.160.80
Deny from 42.120.160.83
Deny from 42.120.160.82
Deny from 42.156.136.28
Deny from 42.120.161.63
Deny from 42.156.137.32
Deny from 42.156.137.37
Deny from 42.120.160.23
Deny from 42.156.136.82
Deny from 42.156.136.81
Deny from 42.156.136.80
Deny from 42.120.160.24
Deny from 42.156.136.4
Deny from 42.156.136.5
Deny from 42.156.136.2
Deny from 42.120.160.9
Deny from 42.120.160.4
Deny from 42.120.160.5
Deny from 42.120.160.2
Deny from 42.156.139.60
Deny from 42.156.139.61
Deny from 42.156.139.62
Deny from 42.156.139.63
Deny from 42.120.161.42
Deny from 42.120.161.40
Deny from 42.156.139.67
Deny from 42.156.138.20
Deny from 42.156.138.16
Deny from 42.156.138.17
Deny from 42.156.138.15
Deny from 42.156.136.37
Deny from 42.120.160.95
Deny from 42.156.136.32
Deny from 42.156.136.31
Deny from 42.156.136.24
Deny from 42.156.137.23
Deny from 42.156.137.20
Deny from 42.120.160.37
Deny from 42.120.160.32
Deny from 42.156.137.28
Deny from 42.156.139.82
Deny from 42.156.139.83
Deny from 42.156.139.80
Deny from 42.156.139.81
Deny from 42.120.161.4
Deny from 42.156.138.107
Deny from 42.156.138.24
Deny from 42.156.138.23
Deny from 42.120.161.74
Deny from 42.120.160.19
Deny from 42.156.138.28
Deny from 42.156.139.15
Deny from 42.156.139.17
Deny from 42.156.139.19
Deny from 42.156.139.18
Deny from 42.120.160.40
Deny from 42.120.160.42
Deny from 42.120.161.20
Deny from 42.156.138.19
Deny from 42.120.161.68
Deny from 42.156.137.2
Deny from 42.120.161.66
Deny from 42.120.161.67
Deny from 42.120.161.60
Deny from 42.120.161.61
Deny from 42.156.137.5
Deny from 42.156.137.4
Deny from 42.156.138.32
Deny from 42.156.138.68
Deny from 42.156.136.18
Deny from 42.156.136.19
Deny from 42.156.138.37
Deny from 42.156.136.15
Deny from 42.156.136.16
Deny from 42.156.136.17
Deny from 42.156.136.13
Deny from 42.156.139.28
Deny from 42.120.160.53
Deny from 42.120.160.50
Deny from 42.120.160.51
Deny from 42.120.161.5
Deny from 42.120.160.57
Deny from 42.120.160.55
Deny from 42.120.161.9
Deny from 42.120.161.19
Deny from 42.156.139.107
Deny from 42.156.139.22
Deny from 42.120.161.15
Deny from 42.120.161.17
Deny from 42.156.137.24
Deny from 42.120.161.83
Deny from 42.156.136.61
Deny from 42.156.136.60
Deny from 42.156.136.63
Deny from 42.156.136.62
Deny from 42.156.136.67
Deny from 42.156.137.107
Deny from 42.156.139.32
Deny from 42.156.139.31
Deny from 42.120.161.81
Deny from 42.156.139.37
Deny from 42.156.138.40
Deny from 42.156.137.19
Deny from 42.156.138.42
Deny from 42.156.137.17
Deny from 42.156.137.15
Deny from 42.156.137.13
Deny from 42.156.137.10
Deny from 42.120.161.55
Deny from 42.120.161.113
Deny from 42.120.161.77
Deny from 42.156.137.74
Deny from 42.156.137.77
Deny from 42.156.137.95
Deny from 42.156.138.2
Deny from 42.156.138.4
Deny from 42.156.138.5
Deny from 42.156.139.113
Deny from 42.120.160.20
Deny from 42.156.137.80
Deny from 42.156.137.81
Deny from 42.156.137.82
Deny from 42.156.137.83
Deny from 42.156.137.113
Deny from 42.156.139.71
Deny from 42.156.139.24
Deny from 42.156.139.20
Deny from 42.156.136.77
Deny from 42.156.136.74
Deny from 42.156.139.23
Deny from 42.120.161.82
Deny from 42.156.138.57
Deny from 42.120.161.80
Deny from 42.156.138.55
Deny from 42.156.138.53
Deny from 42.156.138.50
Deny from 42.156.138.51
这样我们基本上可以解决问题,如果还有漏网之鱼的IP地址,我们再从日志文件中找出来添加进去。如果我们网站采用到CDN加速,这里需要注意到返回的IP地址不是蜘蛛地址,而是CDN的返回地址,我们需要用到非加速状态的获取真实IP地址。
总结,这里蜗牛上面分享到Nginx和Apache环境下的,屏蔽YisouSpider爬虫的抓取导致网站访问变慢和卡死的问题。
[YisouSpider]历史优惠活动内容
猜你可能想看的VPS
- V5.NET:香港/美国云服务器7折月付42港元起;香港物理服务器月付3 美国VPS[主机]
- 天上云:香港CN2弹性云服务器;1核/1G/20G/;促销月付24元起( 香港VPS[主机]
- HTML5绿色有机食品电商网站模板 - Organtio 全球[VPS测评]
- Linux系统下安装Java JDK 全球[VPS测评]
- 云服务器都是有哪些特点? 全球[VPS测评]
- 创意IT服务公司主页网站模板 - Softets 全球[VPS测评]
- 快速云:怎么连接vps?VPS的用途有哪些? 全球[VPS测评]
- 晚点独家丨拼多多9月上线跨境电商平台,首站美国 美国VPS[主机]
- 拼多多正筹建跨境电商平台:密集挖角SHEIN员工,零佣金招商入驻 全球[VPS测评]
- 言車新车|大阳VPS125混动踏板,长续航,代步通勤9980元起 全球[VPS测评]
- 跨境电商拔节生长关键期,宁波政协委员月谈会精准助力! 全球[VPS测评]
- 咖啡主机:享受7、8折促销 美国vps仅需15.4元/月 香港vps仅需 美国VPS[主机]
- RangCloud:香港云主机,1核1G内存/15G SSD/2Mbps 香港VPS[主机]
- 悦智网络:香港cn2 VPS阿里云线路,四川德阳高防vps,大连高防vp 香港VPS[主机]
- 孤狼云:香港沙田CN2/安畅CN2云服务器仅9.9元/月_香港虚拟主机2 香港VPS[主机]
- 硅云:香港免备案云服务器仅232元/年;香港2核2G2M仅482元/年 香港VPS[主机]
- 弘速科技:香港安畅CN2+BGP线路1核2G/8M带宽秒杀款120元/年 香港VPS[主机]
- 41云:香港bgp云服务器低至14元/月;香港三网CN2 GIA VPS 香港VPS[主机]
- 速云互联:香港GT大带宽vps/香港CN2云服务器/洛杉矶200G高防v 香港VPS[主机]
- 香港云服务器对于建站有什么便捷的地方? 香港VPS[主机]
- 香港的云服务器值得租用吗?有什么好处? 香港VPS[主机]
- 2.45亿、成都公安(大数据基座-服务器)采购:神码、虹信、海康、超聚变 全球[VPS测评]
- 2022年最新的15个美容院抖音文案,美容院文案怎么写 全球[VPS测评]
- 云计算的大门依旧没向雷军敞开 全球[VPS测评]
- 快速云:科普云服务器服务器的概念和作用2022-08-2313:26来源 全球[VPS测评]
- tudcloud,香港沙田大带宽vps特惠7折,CN2+BGP直连,10 香港VPS[主机]
- VPSPlayer,超便宜香港cn2线路小带宽VPS,2核4G内存仅30 香港VPS[主机]
- 傲闪云vps,1核1G低至10元起,国内高防/香港CN2/美国CN2/美 美国VPS[主机]
- 恒创科技:国庆中秋双节同庆,香港1核2G云服务器,年付378元 香港VPS[主机]
- 岚云:香港安迅/沙田CN2直连,75折起,1核/1G/30G SSD/1 香港VPS[主机]
转载请注明原文地址: http://140.238.13.167:12355/read-39245.html











