Tracking Good Bots and Crawlers with Distil
Articles,  Blog

Tracking Good Bots and Crawlers with Distil


Although your site lies prey to any number
of bad bots—such as ones that attempt scraping, brute force attacks, competitive data mining,
brownouts, account hijacking, and more, the good news is that not all bots are bad. Good bots include search engine crawlers from
Google, Bingbot, Yahoo Slurp, Baidu, Lycos, and Yandex. Like those, social media crawlers
from Facebook, LinkedIn, Twitter, and Google+ help humans find and access your site. By using the good bot report, you can understand
the traffic patterns and origin sources of the benevolent bots visiting your site. This
information may be helpful in optimizing your website to cater to these bots if needed.
The Good Bots Report displays a snapshot of the top good bots crawling your site, including: Date Filter – Specific date range highlighted
by the Good Bots Report. Bot Name – Name of the good bot crawling
your site (Googlebot, Bingbot, Facebook, and Twitterbot, et al.).
# of Requests – Total number of requests attempted by good bots on the specific date.
Add to Graph – Click the third-column box of any good bot row in the table to add the
good bot’s data to the Daily Good Bot Requests graph. Click the box again to remove the good
bot’s data from the graph. Daily Good Bot Requests – Visual breakdown
of the top good bots crawling your site. Select a good bot below the graph to remove the good
bot’s data from the graph. For more information on Distil, check out
www.distilnetworks.com

Leave a Reply

Your email address will not be published. Required fields are marked *