New Search Engine Blekko Launches Beta
SeoflexForum - Free Ad Forum - Post a FREE Ad - Business - SEO :: SEARCH ENGINES :: Search Engines General
Page 1 of 1
New Search Engine Blekko Launches Beta
New Search Engine Blekko Launches Beta
http://blekko.com
http://blekko.com
--------------------------------------------------------------------------------
| Seoflex Forum |
About the ScoutJet web crawler - Add url ScoutJet
About the ScoutJet web crawler
ScoutJet web crawler
ScoutJet is the web crawler for blekko, a new Silicon Valley based search engine created by the founders of DMOZ and Topix.
We are developing next generation search technology, and kindly request that you permit ScoutJet access to your site so that we may refine our relevance algorithms with the broadest variety of content available from the Internet.
ScoutJet obeys robots.txt
You can prevent ScoutJet from indexing all or part of your site by including the following lines in your http://www.yoursite.com/robots.txt file:
# Allow only specific directories
User-agent: ScoutJet
Disallow: /
Allow: /public
You can also limit the rate at which ScoutJet crawls your page using the Crawl-delay directive:
# Limit ScoutJet's crawl rate (example is to crawl no more than 1 page every 5 seconds)
User-agent: ScoutJet
Crawl-delay: 5
In addition, ScoutJet understands wildcards and Allow.
ScoutJet crawls from the following IP ranges:
64.13.159.*
38.99.96.*, 38.99.97.*, 38.99.98.*, 38.99.99.*
ScoutJet tries its best to crawl politely. But if you do experience a problem with ScoutJet, please let us know at crawler (at) blekko (dot) com.
http://www.scoutjet.com/
ScoutJet web crawler
ScoutJet is the web crawler for blekko, a new Silicon Valley based search engine created by the founders of DMOZ and Topix.
We are developing next generation search technology, and kindly request that you permit ScoutJet access to your site so that we may refine our relevance algorithms with the broadest variety of content available from the Internet.
ScoutJet obeys robots.txt
You can prevent ScoutJet from indexing all or part of your site by including the following lines in your http://www.yoursite.com/robots.txt file:
# Allow only specific directories
User-agent: ScoutJet
Disallow: /
Allow: /public
You can also limit the rate at which ScoutJet crawls your page using the Crawl-delay directive:
# Limit ScoutJet's crawl rate (example is to crawl no more than 1 page every 5 seconds)
User-agent: ScoutJet
Crawl-delay: 5
In addition, ScoutJet understands wildcards and Allow.
ScoutJet crawls from the following IP ranges:
64.13.159.*
38.99.96.*, 38.99.97.*, 38.99.98.*, 38.99.99.*
ScoutJet tries its best to crawl politely. But if you do experience a problem with ScoutJet, please let us know at crawler (at) blekko (dot) com.
http://www.scoutjet.com/
--------------------------------------------------------------------------------
| Seoflex Forum |
Similar topics
» Search engine Blekko to rely on the human touch
» How do I add my site to Active Search Results Search Engine
» Local Search Engine - Way to Search product & Services
» Yahoo! Search will always be a Search Engine
» What is your favorite Search Engine
» How do I add my site to Active Search Results Search Engine
» Local Search Engine - Way to Search product & Services
» Yahoo! Search will always be a Search Engine
» What is your favorite Search Engine
SeoflexForum - Free Ad Forum - Post a FREE Ad - Business - SEO :: SEARCH ENGINES :: Search Engines General
Page 1 of 1
Permissions in this forum:
You cannot reply to topics in this forum
|
|