What is the purpose of the robots.txt file in terms of Google and SEO?
Question: -
Q: To keep the general site human-friendly, where would you put the robots.txt file? In your sitemap?
Q: In the robots.txt file, would you want to indicate that non-product related pages (staff bios, resources for existing customers) should not be indexed? Do those types of pages dilute your density? Or is it best to incorporate keywords into those pages as much as possible?
Answer: -
The robots.txt file is decidedly not for the humans, hence its name (robots.txt). The universally accepted location to put it is in your top directory, as in http://www.jm-seo.org/robots.txt or http://www.jasonmcdonald.org/robots.txt. You do NOT link to it anywhere on your website, as it is only for the robots.
That said, what are its functions?
First and foremost, it tells the robots (a.k.a., Googlebot, Slurp (The Yahoo Spider), etc.) what not to look at. Following the disallow command you tell these indexing spiders what to IGNORE on your website. The syntax is as follows:
User-agent: * This means this applies to all robots
Allow: / this means index everything, just to be clear, except what is disallowed belo
Disallow: /1 this means do not index the '1' directory
Disallow: /password this means do not index the 'password' directory
Finally, the robots.txt file also indicates the location of your sitemap.xml file. And remember that it is (in a sense) the opposite. What you put IN the sitemap.xml file is what you DO want them to index.
And, for extra credit, I recommend that you exclude your PPC / AdWords ad landing pages, because you want 100% paid traffic flowing into these pages so as to not confuse your metrics!
As for the second question in this group, I would leave in your staff pages, etc., as they make your website appear more natural. I would only exclude very unrelated pages from the indexing process.
Hope this helps!
- Jason McDonald - info@jm-seo.org
|
