Baidu spider

晚上9:17


I have no idea why Baidu spider didn't follow robot.txt rules, finally I use iptables rules to block in and out ip address.

[Iptable rules]

iptables -p tcp -I INPUT -j DROP -s 119.63.192.x
iptables -p tcp -I INPUT -j DROP -s 123.125.64.x
iptables -p tcp -I INPUT -j DROP -s 180.76.0.x
iptables -p tcp -I INPUT -j DROP -s 220.181.0.x

[Reference]
http://blog.indeepnight.com/2012/03/how-to-block-web-spider-or-crawler.html

http://tools.dynamicdrive.com/userban/#.UphVwcQW2So

http://www.robotstxt.org/robotstxt.html

https://support.google.com/webmasters/answer/156449?hl=zh-Hant

http://forums.oscommerce.com/topic/382923-baiduspider-using-multiple-user-agents-how-to-stop-them/

http://baike.baidu.com/view/1847001.htm?noadapt=1#4

http://www.weithenn.org/cgi-bin/wiki.pl?Mysql_Apache_PHP-%E9%BB%83%E9%87%91%E6%9E%B6%E7%AB%99%E7%B5%84%E5%90%88#Heading43

You Might Also Like

0 意見