加拿大华人论坛 加拿大百科Robots.txt
在加拿大
For information on blocking any of these robots using the Robots.txt exclusion standard, see http://www.robotstxt.org/wc/exclusion.html AltaVista Ask Jeeves FAST GoogleInktomi LycosMSN Yahoo Company AltaVista User Agent Scooter-3.0.3 (Many variations. Most contain the word Scooter) Robot.txt Identifier User-agent: Scooter Details Company Ask Jeeves User Agent Mozilla/2.0 (compatible; Ask Jeeves) Robot.txt Identifier User-agent: directhit User-agent: teomaagent1 Details Company Fast Search and Transfer ASA User Agent FAST-WebCrawler/3.4/Nirvana) AKA - Mozilla/4.0 (compatible; FastCrawler3, [email protected]) Robot.txt Identifier User-agent: fast Details http://fast.no/support/crawler.asp Powers Alltheweb.com, Lycos and many smaller search engines Company Google User Agent Googlebot/2.1 (+http://www.googlebot.com/bot.html) AKA: Wget/1.5.3 AKA: Googlebot-image (+http://www.googlebot.com/bot.html) Robot.txt Identifier User-agent: googlebot Details http://www.googlebot.com/bot.html Company Inktomi User Agent Slurp (Slurp.so/1.0 ([email protected]; http://www.inktomi.com/slurp.html) Robot.txt Identifier User-agent: slurp Details http://www.inktomi.com/slurp.html Powers AOL, MSN and many others Company Lycos User Agent Lycos_Spider_(modspider) AKA: T-Rex Robot.txt Identifier User-agent: lycos Details Lycos is powered by the search engine at Fast, we don't know why they continue to operate their own spider Company Microsoft / MSN User Agent MSNbot / MSRbot Robot.txt Identifier N/A Details MSNbot is the development Bot for their new search engine MSRbot is supposed to be a Research Bot Page topic: Search engine optimization, What Are Search Engine Robots?Related Content Free Search Engine Positioning Report
评论
User-agent: *Disallow: /User-agent: googlebotDisallow: cheese.htm# robots.txt for http://www.example.com/User-agent: *Disallow: /cyberworld/map/ # This is an infinite virtual URL spaceDisallow: /tmp/ # these will soon disappearDisallow: /foo.html
评论
http://www.oyster-web.co.uk/inf_robots.htm
·中文新闻 2024年巴西G20峰会：直言不讳的前外交官山上真吾指责总理安东尼
·中文新闻 在内皮恩河寻找失踪男子
