欢迎光临
我们一直在努力
当前位置：IDC资讯中心 > 站长之家 > SEO网站优化 > 正文

世界各大搜索引擎的蜘蛛名称列表_seo网站优化

2007-08-24 分类：SEO网站优化 阅读(2239) 评论(0)
建站超值云服务器，限时71元/月

本文记录了全世界比较出名的Robots.txt 列表需要设置的搜索蜘蛛。如何设置那个目录不想被搜索引擎收录的可参照下去设置。


当然也必须从Robots.txt 去设置



下列为比较出名的搜索引擎蜘蛛名称：


Google的蜘蛛： Googlebot
百度的蜘蛛：baiduspider
Yahoo的蜘蛛：Yahoo Slurp
MSN的蜘蛛：Msnbot
Altavista的蜘蛛：Scooter
Lycos的蜘蛛： Lycos_Spider_(T-Rex)
Alltheweb的蜘蛛： FAST-WebCrawler/
INKTOMI的蜘蛛： Slurp


如需要参考的可以参照本文：
User-agent（用户代理设置）：(蜘蛛名字)
拒绝：(文件名字)


 


User-agent: Black Hole
Disallow: /


User-agent: Titan
Disallow: /


User-agent: WebStripper
Disallow: /


 


User-agent: NetMechanic
Disallow: /


User-agent: CherryPicker
Disallow: /


User-agent: EmailCollector
Disallow: /


 


User-agent: EmailSiphon
Disallow: /


 


User-agent: WebBandit
Disallow: /


 


User-agent: EmailWolf
Disallow: /


 


User-agent: ExtractorPro
Disallow: /


User-agent: CopyRightCheck
Disallow: /


 


User-agent: Crescent
Disallow: /


User-agent: NICErsPRO
Disallow: /



User-agent: Wget
Disallow: /


User-agent: SiteSnagger
Disallow: /


 


User-agent: ProWebWalker
Disallow: /


User-agent: CheeseBot
Disallow: /


User-agent: mozilla/4
Disallow: /


User-agent: mozilla/5
Disallow: /



User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow: /


 


User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow: /


 


User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 9
Disallow: /


 


User-agent: ia_archiver
Disallow: /


 


User-agent: ia_archiver/1.6
Disallow: /


 


User-agent: Alexibot
Disallow: /


User-agent: Teleport
Disallow: /


 


User-agent: TeleportPro
Disallow: /


User-agent: Wget
Disallow: /


User-agent: MIIxpc
Disallow: /



User-agent: Telesoft
Disallow: /



User-agent: Website Quester
Disallow: /


User-agent: WebZip
Disallow: /



User-agent: moget/2.1
Disallow: /


 


User-agent: WebZip/4.0
Disallow: /


User-agent: WebStripper
Disallow: /


User-agent: WebSauger
Disallow: /


User-agent: WebCopier
Disallow: /


User-agent: NetAnts
Disallow: /


User-agent: Mister PiX
Disallow: /


 


User-agent: WebAuto
Disallow: /


User-agent: TheNomad
Disallow: /



User-agent: WWW-Collector-E
Disallow: /


 


User-agent: RMA
Disallow: /


 


User-agent: libWeb/clsHTTPDisallow: /


User-agent: asterias
Disallow: /



User-agent: turingos
Disallow: /


 


User-agent: spanner
Disallow: /


 


User-agent: InfoNaviRobot
Disallow: /



User-agent: Harvest/1.5
Disallow: /



User-agent: ExtractorPro
Disallow: /



User-agent: Bullseye/1.0
Disallow: /


 


User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /


User-agent: Crescent Internet ToolPak HTTPOLE Control v.1.0
Disallow: /


 


User-agent: CherryPickerSE/1.0
Disallow: /


User-agent: CherryPickerElite/1.0
Disallow: /


User-agent: WebBandit/3.50
Disallow: /


User-agent: NICErsPRO
Disallow: /



User-agent: Microsoft URL Control – 5.01.4511
Disallow: /



User-agent: DittoSpyder
Disallow: /


User-agent: Foobot
Disallow: /


User-agent: WebmasterWorldForumBot
Disallow: /


User-agent: SpankBot
Disallow: /


User-agent: BotALot
Disallow: /



User-agent: lwp-trivial/1.34
Disallow: /


User-agent: lwp-trivial
Disallow: /



User-agent: BunnySlippers
Disallow: /


 


User-agent: Microsoft URL Control – 6.00.8169
Disallow: /


User-agent: URLy Warning
Disallow: /


User-agent: Wget
Disallow: /



User-agent: Wget/1.5.3
Disallow: /


User-agent: LinkWalker
Disallow: /


User-agent: cosmos
Disallow: /


User-agent: moget
Disallow: /


User-agent: hloader
Disallow: /


 


User-agent: humanlinks
Disallow: /


User-agent: LinkextractorPro
Disallow: /


 


User-agent: Offline Explorer
Disallow: /


编辑：Windear
首发：SEO、SEM密谋、站长资源信息网
地址：http://www.seoplot.com，/
转载请出处和保留连接！否则必究



User-agent: Mata Hari
Disallow: /


 


User-agent: LexiBot
Disallow: /


 


User-agent: Offline Explorer
Disallow: /



User-agent: Web Image Collector
Disallow: /


User-agent: The Intraformant
Disallow: /


 


User-agent: True_Robot/1.0
Disallow: /


User-agent: True_Robot
Disallow: /


User-agent: BlowFish/1.0
Disallow: /


 


User-agent: JennyBot
Disallow: /


User-agent: MIIxpc/4.2
Disallow: /


 


User-agent: BuiltBotTough
Disallow: /



User-agent: ProPowerBot/2.14
Disallow: /


User-agent: BackDoorBot/1.0
Disallow: /



User-agent: toCrawl/UrlDispatcher
Disallow: /


User-agent: WebEnhancer
Disallow: /



User-agent: TightTwatBot
Disallow: /


 


User-agent: suzuran
Disallow: /


User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /


User-agent: VCI
Disallow: /


User-agent: Szukacz/1.4
Disallow: /


User-agent: QueryN Metasearch
Disallow: /


User-agent: Openfind data gathere
Disallow: /


User-agent: Openfind
Disallow: /


 


User-agent: Xenus Link Sleuth 1.1c
Disallow: /


 


User-agent: Xenus
Disallow: /


User-agent: Zeus
Disallow: /


User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /


User-agent: RepoMonkey
Disallow: /


User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /


User-agent: Webster Pro
Disallow: /


User-agent: EroCrawler
Disallow: /


User-agent: LinkScan/8.1a Unix Disallow: /



User-agent: Kenjin Spider
Disallow: /


User-agent: Keyword Density/0.9
Disallow: /


 


User-agent: Kenjin Spider
Disallow: /


 


User-agent: Cegbfeieh
Disallow: /


Different:


User-agent: larbin
Disallow: /


User-agent: b2w/0.1
Disallow: /


 


User-agent: Copernic
Disallow: /



User-agent: psbot
Disallow: /



User-agent: Python-urllib
Disallow: /


 



User-agent: NetMechanic
Disallow: /


User-agent: URL_Spider_Pro
Disallow: /


User-agent: CherryPicker
Disallow: /


编辑：Windear
首发：SEO、SEM密谋、站长资源信息网
地址：http://www.seoplot.com，/
转载请出处和保留连接！否则必究


User-agent: EmailCollector
Disallow: /


User-agent: EmailSiphon
Disallow: /



User-agent: WebBandit
Disallow: /



User-agent: EmailWolf
Disallow: /


User-agent: ExtractorPro
Disallow: /


User-agent: CopyRightCheck
Disallow: /


 


User-agent: Crescent
Disallow: /


 


User-agent: SiteSnagger
Disallow: /


User-agent: ProWebWalker
Disallow: /


User-agent: CheeseBot
Disallow: /


 


User-agent: LNSpiderguy
Disallow: /



User-agent: Mozilla
Disallow: /


User-agent: mozilla
Disallow: /


User-agent: mozilla/3
Disallow: /


User-agent: mozilla/4
Disallow: /



User-agent: mozilla/5
Disallow: /



User-agent: WebAuto
Disallow: /


 


User-agent: TheNomad
Disallow: /


 


User-agent: WWW-Collector-E
Disallow: /



User-agent: RMA
Disallow: /


 


User-agent: libWeb/clsHTTP
Disallow: /


User-agent: httplib
Disallow: /


User-agent: turingos
Disallow: /


User-agent: InfoNaviRobot
Disallow: /


User-agent: Harvest/1.5
Disallow: /


 


User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /



User-agent: CherryPickerSE/1.0
Disallow: /


 


User-agent: CherryPickerElite/1.0
Disallow: /



User-agent: WebBandit/3.50
Disallow: /


 


User-agent: NICErsPRO
Disallow: /


User-agent: DittoSpyder
Disallow: /


User-agent: Foobot
Disallow: /


User-agent: BotALot
Disallow: /


 


User-agent: lwp-trivial/1.34
Disallow: /


User-agent: lwp-trivial
Disallow: /



User-agent: URLy Warning
Disallow: /



User-agent: hloader
Disallow: /



User-agent: humanlinks
Disallow: /


User-agent: LinkextractorPro
Disallow: /


User-agent: Offline Explorer
Disallow: /


 


User-agent: Mata Hari
Disallow: /


User-agent: LexiBot
Disallow: /


User-agent: Web Image Collector
Disallow: /



User-agent: The Intraformant
Disallow: /



User-agent: True_Robot/1.0
Disallow: /


User-agent: True_Robot
Disallow: /


User-agent: BlowFish/1.0
Disallow: /



User-agent: JennyBot
Disallow: /


User-agent: MIIxpc/4.2
Disallow: /



User-agent: BuiltBotTough
Disallow: /



User-agent: ProPowerBot/2.14
Disallow: /


User-agent: BackDoorBot/1.0
Disallow: /


User-agent: toCrawl/UrlDispatcher
Disallow: /


 


User-agent: WebEnhancer
Disallow: /



User-agent: suzuran
Disallow: /


 


User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /



User-agent: VCI
Disallow: /


User-agent: Szukacz/1.4
Disallow: /


 


User-agent: QueryN Metasearch
Disallow: /


User-agent: Openfind data gathere
Disallow: /


User-agent: Openfind
Disallow: /



User-agent: Xenus Link Sleuth 1.1c
Disallow: /



User-agent: Xenus
Disallow: /


 


User-agent: Zeus
Disallow: /


User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /



User-agent: RepoMonkey
Disallow: /


 


User-agent: Openbot
Disallow: /



User-agent: URL Control
Disallow: /


 


User-agent: Zeus Link Scout
Disallow: /


User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /


User-agent: EroCrawler
Disallow: /


User-agent: LinkScan/8.1a Unix
Disallow: /



User-agent: Keyword Density/0.9
Disallow: /


User-agent: Kenjin Spider
Disallow: /


User-agent: Iron33/1.0.2
Disallow: /



User-agent: Bookmark search tool
Disallow: /


User-agent: GetRight/4.2
Disallow: /


User-agent: FairAd Client
Disallow: /



User-agent: Gaisbot
Disallow: /


User-agent: Aqua_Products
Disallow: /


User-agent: Radiation Retriever 1.1
Disallow: /


User-agent: WebmasterWorld Extractor
Disallow: /



User-agent: Flaming AttackBot
Disallow: /


User-agent: Oracle Ultra Search
Disallow: /


User-agent: MSIECrawler
Disallow: /


User-agent: PerMan
Disallow: /


 


User-agent: searchpreview
Disallow: /


User-agent: naver
Disallow: /


 


User-agent: dumbot
Disallow: /


User-agent: Hatena Antenna
Disallow: /


User-agent: grub-client
Disallow: /


User-agent: grub
Disallow: /



User-agent: larbin
Disallow: /


User-agent: b2w/0.1
Disallow: /



User-agent: Copernic
Disallow: /


 


User-agent: psbot
Disallow: /


 


User-agent: Python-urllib
Disallow: /



User-agent: EmailWolf
Disallow: /



User-agent: ExtractorPro
Disallow: /



User-agent: CopyRightCheck
Disallow: /


User-agent: Crescent
Disallow: /


User-agent: SiteSnagger
Disallow: /


User-agent: ProWebWalker
Disallow: /


 


User-agent: CheeseBot
Disallow: /



User-agent: LNSpiderguy
Disallow: /


User-agent: Mister PiX
Disallow: /


User-agent: WebAuto
Disallow: /


User-agent: TheNomad
Disallow: /


User-agent: WWW-Collector-E
Disallow: /


User-agent: RMA
Disallow: /


User-agent: httplib
Disallow: /



User-agent: turingos
Disallow: /


User-agent: InfoNaviRobot
Disallow: /


User-agent: Harvest/1.5
Disallow: /


User-agent: Bullseye/1.0
Disallow: /


User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /



User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /



User-agent: CherryPickerSE/1.0
Disallow: /


User-agent: CherryPickerElite/1.0
Disallow: /



User-agent: NICErsPRO
Disallow: /


User-agent: URLy Warning
Disallow: /


 


User-agent: humanlinks
Disallow: /



User-agent: Web Image Collector
Disallow: /


User-agent: The Intraformant
Disallow: /


User-agent: True_Robot/1.0
Disallow: /



User-agent: True_Robot
Disallow: /


User-agent: BlowFish/1.0
Disallow: /


User-agent: JennyBot
Disallow: /


User-agent: MIIxpc/4.2
Disallow: /


 


User-agent: BuiltBotTough
Disallow: /


User-agent: ProPowerBot/2.14
Disallow: /


 


User-agent: BackDoorBot/1.0
Disallow: /



User-agent: toCrawl/UrlDispatcher
Disallow: /


User-agent: WebEnhancer
Disallow: /



User-agent: suzuran
Disallow: /


User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /



User-agent: VCI
Disallow: /



User-agent: Szukacz/1.4
Disallow: /


User-agent: QueryN Metasearch
Disallow: /


User-agent: Openfind data gathere
Disallow: /


User-agent: Openfind
Disallow: /


User-agent: Xenus Link Sleuth 1.1c
Disallow: /



User-agent: Xenus
Disallow: /


User-agent: Zeus
Disallow: /


 


User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /


 


User-agent: RepoMonkey
Disallow: /


User-agent: Microsoft URL Control
Disallow: /


User-agent: Openbot
Disallow: /


User-agent: URL Control
Disallow: /



User-agent: Zeus Link Scout
Disallow: /



User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /


User-agent: Webster Pro
Disallow: /


 


User-agent: EroCrawler
Disallow: /


User-agent: LinkScan/8.1a Unix
Disallow: /


User-agent: Keyword Density/0.9
Disallow: /



User-agent: Kenjin Spider
Disallow: /



User-agent: Iron33/1.0.2
Disallow: /


User-agent: Bookmark search tool
Disallow: /


 


User-agent: GetRight/4.2
Disallow: /


User-agent: FairAd Client
Disallow: /



User-agent: Gaisbot
Disallow: /


User-agent: Aqua_Products
Disallow: /



User-agent: Radiation Retriever 1.1
Disallow: /


 


User-agent: WebmasterWorld Extractor
Disallow: /


 


User-agent: Flaming AttackBot
Disallow: /



User-agent: Oracle Ultra Search
Disallow: /


 


User-agent: MSIECrawler
Disallow: /


User-agent: PerMan
Disallow: /



User-agent: searchpreview
Disallow: /


User-agent: sootle
Disallow: /


User-agent: es
Disallow: /


 


User-agent: Enterprise_Search/1.0
Disallow: /


 


User-agent: Enterprise_Search
Disallow: /

赞(0)
版权申明：本站文章部分自网络，如有侵权，请联系：west999com@outlook.com 特别注意：本站所有转载文章言论不代表本站观点！ 本站所提供的图片等素材，版权归原作者所有，如需使用，请与原作者联系。未经允许不得转载：IDC资讯中心 » 世界各大搜索引擎的蜘蛛名称列表_seo网站优化
分享到： 更多 (0)

相关推荐

热门标签

ldquo (1370)win (1324)活动 (1166)ios (1162)rdquo (818)微信 (747)iphone (728)小米 (722)windows (706)手机 (698)酷跑 (686)飞车 (540)天天 (492)qq (491)网站 (457)全民 (442)discuz (408)充值 (389)安装 (368)更新 (321)华为 (300)浏览器 (292)支付宝 (289)三星 (266)note (264)教程 (262)预览版 (261)小编 (258)越狱 (254)linux (248)

热门文章

分类目录