CCBot/1.0 (+http://www.commoncrawl.org/bot.html). Just seen this bot come by today, obeyed Robots.txt enough to take it read it and leave ... www.webmasterworld.com/search_engine_spiders/3612195.htm - Similar pages
How does the bot identify itself? Our bot will identify itself with the following User-Agent string: CCBot/1.0 (+http://www.commoncrawl.org/bot.html) ... www.commoncrawl.org/faq.htm - 8k - Cached - Similar pages
CCBot/1.0 (+http://www.commoncrawl.org/bot.html). User Agent Vitals:. Bot or Browser: This User Agent is a known bot. This User Agent is a Bot! ... www.botsvsbrowsers.com/details/158107/index.html - 21k - Cached - Similar pages
HTTP User Agent, CCBot/1.0 (+http://www.commoncrawl.org/bot.html). Exclusion. The website reference, given below, confirms support for the robots.txt ... www.janetsystems.co.uk/Articles/WebRobots/tabid/82/itemid/1731/modid/ 450/Default.aspx - 43k - Cached - Similar pages