» » » » » » » » Settings Robots.txt in Blogger / Blogspot & Website

Settings Robots.txt in Blogger / Blogspot & Website

, , , , , ,
What is the robots.txt?
Robots.txt is a command for search engine robots to browse or do not browse a page on our blog. Robots.txt is arguably filter our blog from search engines.

Does every blog has a robots.txt?
Yes, all blogs already have a robots.txt given by bloggers. By default robots.txt on a blog like this:


User-agent: Mediapartners-GoogleDisallow:

User-agent:*

Disallow: / searchAllow: /

Sitemap: http: // bloggerseo786 / feeds / posts / default? Orderby = UPDATED

To view the default robots.txt, please type dibrowser http://bloggerseo786.blogspot.com/robots.txtWhat is the meaning of the codes above?User-agent: Mediapartners-GoogleGoogle Adsense robot crawls blog

Disallow:
Which is not allowed no

User-agent: *
All robot Search Engine / Search engine

Disallow: / search
Not allowed to crawl the search folder and setrusnya, like .... / search / label and ... search / search? Updated ...

Allow: /
Allow all pages to be crawled, except written on Disallow above. Mark (/) or less means that the name of the blog.

Sitemap: 
http: // bloggerseo786/ feeds / posts / default? Orderby = UPDATED

sitemap or feed address our blog.
Why not label incorporated into searches?
By writing the default code Disallow: / search, it means crawling to label such http://namablog.com/search/label/Tutorial address, will not be crawled by search engines because the label is not the real URL toward a specific page.
How to prevent the robot on a particular page?
To prevent google crawl certain pages on your blog, tell my friend about me does not want pages indexed by the search engines. For the URL about me on your blog eg http://namablogsobat.blogspot.com/p/about.htmlSo to robots.txt, please copy the code defaut above, tambahakan pages that are not allowed, the result is like this:

User-agent: Mediapartners-GoogleDisallow:

User-agent: *

Disallow: / search

Disallow: /p/about.html

Allow: /Sitemap: http: // bloggerseo786/ feeds / posts / default? Orderby = UPDATED

Where is the way to edit robots.txt?
To add or editing robots.txt, go to Settings> Preferences Search, on Robots.txt click Edit and then click Yes, then write the desired code robots.txt> Save Changes
I never permutations robots.txt, how?
It's okay, your blog will still be crawled by search engine robots because as I mentioned before, every blog already possessed default robots.txt.Hati2 with the use of robots.txt, if the keywords of your blog can be ignored by the search engine.