Why should you exclude your PPC / AdWords Landing Pages via robots.txt?
Question: -
Q: If one excludes all on-website links to specific landing pages, so only the advertisement has a link to those pages, then why do we need to include those pages in robots.txt?
Answer: -
You would think that if something is 'not linked to' from your home page, then no one can find it. Generally, this is probably the case. But in some cases, you may advertise in a magazine or other publication and they have a website. That, then gets picked up by Google and indexed via a cross link. Even AdWords direct links seem to be indexed by Google at times, even though you would think that this would not happen.
I have seen Google pick up all sorts of stuff from a website that even is explicitly forbidden via robots.txt. It's a messy world and sometimes Google and the GoogleBot seem to just pick stuff up.
So, I think you should try your best to ensure that these AdWords / PPC landing pages are not picked up - which means be redundant in how you try to exclude them from organic search. Finally, you can always you the 'site:yourwebsite.com/landingpage.html' syntax to see if a specific page has found its way into the index, and then go to Webmaster tools to request an exclusion if you do not want it indexed.
Hope this helps!
- Jason
|
