Robots.txt Formatting | Affilorama
Articles,  Blog

Robots.txt Formatting | Affilorama

the robots.txt file formatting is when you start is indexed by the search
engines it scrolled by the search engine spiders
google bot yellow slip in Bing bot in order to find
all the content on your site so that other people can find it but
what if you could sections a view website that you don’t want indexed the bot’s dumbly index whatever
they can find they don’t know that for example those photos on the hidden part of your
site a strictly friends and family only or that there a certain pages in your
website that you’d really rather not have popping up in the
search engine listings or being archived by that piece key internet I caught book robots.txt formatting like your long expired special offers in this lesson we look at robots dot txt
robots dot T ext telling the search engines
what they can and cannot index what is the robots don’t text file robots not txt is a small text document
that lives in the router view website intel’s the robots visiting your website
which pages they can and cannot Nexus win one of these robots visits your site
the first thing they do is go looking for the robot start text file robots.txt date listen to your requests in one
visit pages that you’ve disallowed so how do you make a robot’s far will
decide which the reserve your website you want the
spiders to index in which ones you don’t want them crawling through and decide if there are any parts that
you would rather not have crawled through your site at all open up your plain text editor choice credit new blank text file and save it
as robots doc txt then write this information into the
file to block all spiders from you entire
website write this two late goals by to see all content on
the website robots.txt see this certain directories you can follow this into blocks inspired him to allow a certain spot a while blocking others his some tips you must use a new line
for each instruction blank lines are used to show separate
groups instructions there’s in the last example the S tricks in the user agent lynn has
a special meaning in robot stop txt’n can be used as a wildcard if you were to
disallow all gift images on your website you couldn’t
just go do disallow x2x gif that won’t work you file must be called robots don’t txt all lower cases can you file must be
located in the root directory view robots.txt formatting and this is website for example dub dub dub dot you’re so I dot com forward slash robots
don’t txt that’s where the spot is look when they
visit your site and they won’t find it if you put it anywhere else now simply save your file
and upload it to your website robots.txt you robot file and your XML sign if you’ve already seen Allison on
creating XML site names you know that your robots dot takes
fathers a really handy place to live the search engines know where that is all you have to do is leave a blank line
after the last command in your robots file and then paste this little line here if
you’ve got more than one site meant you can lead to more than one line this way you need to specifically tell
each and every search engine with a confined you cite me they’ll see
it as soon as they look for your robots.txt file formatting palm which every polite but will do when they
visit your site anyway because some things you need to know not
all spiders on a robot’s dot T ext polite spot as such as those belonging
to the major search engines up a lot and want index items you listed
a new robots farm however not all robots up a lot for
example from small search engines or general data scraping boats tell also collect any and all content
anyway your robots dot takes far is actually
publicly accessible don’t try to use the robots file to hide
content on your site the robots file is able to be viewed by
anybody robots.txt files simply by typing dub dub dub you’re so I
dot com forward slash robot star text into the browser so anybody can see the things you’ve
said you don’t want indexed if these content on your website that
you really really don’t want anybody else seeing you be speed is to password-protected
directory they will usually be a tool to help you
do this in your hosting control panel see panel similar know that password
protecting your comment.if done right will also prevent the un
polite bart’s from exiting the content in time for today’s lesson summary in
this lesson with looked robots don’t text what it is what it’s used for how to
create one we’ve looked at certain things you can
do with robots to takes including blocking your entire site from being
index blocking certain directories blocking
certain parts and identifying the location of the site
map anyway that’s all we’ve got time for
today thanks so much for watching about robots.txt file formatting, and we’ll see you again soon bye for now
the free affiliate marketing training video lessons

One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *