• Tracking Good Bots and Crawlers with Distil
    Articles,  Blog

    Tracking Good Bots and Crawlers with Distil

    Although your site lies prey to any number of bad bots—such as ones that attempt scraping, brute force attacks, competitive data mining, brownouts, account hijacking, and more, the good news is that not all bots are bad. Good bots include search engine crawlers from Google, Bingbot, Yahoo Slurp, Baidu, Lycos, and Yandex. Like those, social media crawlers from Facebook, LinkedIn, Twitter, and Google+ help humans find and access your site. By using the good bot report, you can understand the traffic patterns and origin sources of the benevolent bots visiting your site. This information may be helpful in optimizing your website to cater to these bots if needed. The Good…