Don’t block Googlebot from crawling JavaScript and CSS
Articles,  Blog

Don’t block Googlebot from crawling JavaScript and CSS


Hi everybody. Matt Cutts here. I just have a short public
service announcement rather than a question today. And the PSA is this. If you block Googlebot from
crawling JavaScript or CSS, please take a few minutes and
just take that out of the robots.txt. And let us crawl the JavaScript,
let us crawl the CSS, and get a better idea of
what’s going on on the page. A lot of people block it because
they think, this is going to be resources that
I don’t want to have the bandwidth or something. But Googlebot is pretty smart
about not crawling stuff too fast. And a lot of people will
do things like, they’ll check for flash, but then they’re
including some JavaScript. And they don’t realize,
including that Javascript, the Javascript is blocked. And so we’re not able to crawl
the site as effectively as we would like. In addition, Google is getting
better at processing JavaScript. It’s getting better at things
like looking at CSS to figure out what’s important
on the page. So if you do block Googlebot,
I would ask, please take a little bit of time, go ahead and
remove those blocks from the robots.txt, so you can let
Googlebot in, get a better idea of what’s going on with
your site, get a better idea of what’s going on
with your page. And then that just helps
everybody, in terms of if we can find the best search
results, we’ll be able to return them higher to users. So thanks if you can
take the chance. I know it’s kind of a common
idiom for people to just say, I’m going to block JavaScript
and CSS. But you don’t need
to do that now. So please, in fact, actively
let Googlebot crawl things like Javascript and
CSS if you can. Thanks.

6 Comments

  • Shreyansh Modi

    Thanks for the share, i always asked myself this question is CSS & JS of any importance to google bot. I got my answer today. Now we have entered the Quality SEO Age..!!

  • Chris Rempel

    It seems the more we "give" Google the less benefit we have as webmasters. Not that it's directly relevant to this video, but I personally think Google is on a dark path. With G Hosted interstitial pages from search, "answering questions" directly and preferential SERP treatment for its own services (YouTube), our site content is just fodder for their own channel. And unless you're a brand, RIP organic traffic…

    (Oh, but I'm sure we can buy Adwords to make up for it. Wait…)

  • Laurent Bourrelly

    Ok like GoogleBot respects robots.txt !
    I've observed it going anywhere it pleases, despite robots.txt, noindex, rel nofollow.

  • Ty Whalin

    Not sure what to think about this change. Curious to how many much this could impact my site. Waiting for some changes currently going on before deciding on this one.

  • Mark Hallam

    I have put Allow: for everything and still have issues! I have been to the forums and implemented their solutions including speeding up my sites….it is all to no avail! I guess Google will never be human enough to help out webmasters with real answers. This video doesn't even use the robots.txt to demonstrate??

Leave a Reply

Your email address will not be published. Required fields are marked *