Ezoicbot is the generic name for Ezoics web crawler. Ezoicbot is the general name for two different types of crawlers: a desktop crawler that simulates a user on desktop browser, and a mobile crawler that simulates a user on a mobile device.
Your website will probably be crawled by both Ezoicbot Desktop and Ezoicbot Mobile. You can identify the subtype of Ezoicbot by looking at the user agent string in the request. However, both crawler types obey the same product token (user agent token) in robots.txt, and so you cannot selectively target either Ezoicbot mobile or Ezoicbot desktop using robots.txt.
What is Ezoicbot and why is it crawling my site?
Ezoic is a technology platform for digital publishers. You can learn more about what Ezoic does here.
Ezoicbot is our web crawler designed to extract valuable information about how the internet, search engines, and websites all work together. Ezoicbot can help publishers and all web publishers better understand how their sites work. This includes the ability for search engines, like Google, to index and rank their content.
How Ezoicbot accesses your site
For most sites, Ezoicbot shouldn’t access your site more than once every few seconds on average. However, due to delays it’s possible that the rate will appear to be slightly higher over short periods.
Ezoicbot was designed to be run simultaneously by thousands of machines to improve performance and scale as the web grows. Also, to cut down on bandwidth usage, we run many crawlers on machines located near the sites that they might crawl. Therefore, your logs may show visits from several machines at ezoic.com, all with the user-agent Ezoicbot. Our goal is to crawl pages from your site without overwhelming your server’s bandwidth.
Blocking Ezoicbot from visiting your site
It’s almost impossible to keep a web server secret by not publishing links to it. For example, as soon as someone follows a link from your “secret” server to another web server, your “secret” URL may appear in the referrer tag and can be stored and published by the other web server in its referrer log. Similarly, the web has many outdated and broken links. Whenever someone publishes an incorrect link to your site or fails to update links to reflect changes in your server, Ezoicbot will try to crawl an incorrect link from your site.
If you want to prevent Ezoicbot from crawling content on your site, you have a number of options. Be aware of the difference between preventing a page from being accessible at all by both crawlers or users. Changes to your robots.txt file will usually allow you to prevent Ezoicbot from crawling a site or its pages.
Ezoicbot and most bots will respect the directives in robots.txt.