• Robots: How to Influence Crawling and Indexing on Google | SEO COURSE 2020 【Lesson #29】
    Articles,  Blog

    Robots: How to Influence Crawling and Indexing on Google | SEO COURSE 2020 【Lesson #29】

    In SEO terms, the crawling phase occurs when Googlebot accesses a page and analyzes it, while the indexing occurs when the webpage appears to be suitable for inclusion in the search engine index. Since the 1990s, webmasters around the world have used a robots.txt file in the root of their websites in order to provide any bot with some instructions on how to access their content. In this very simple text file, a Disallow directive is inserted, containing the paths of the pages or folders that the bot must not scan, in order not to overload the resources of our server. There is also a User-agent for referring to a…