What is a Robots.txt File?
In simple words a Robots.txt is at the root folder of any website and has a message for search engines bots to index or not to index the website pages for inclusion in web directory. This small, but very important aspect of SEO can make or break the SEO efforts of a company website.
SEO & Robots.txt file
When it comes to SEO, most people understand that a website must be “search engine friendly” site architecture/HTML, and meta data, title tags and meta descriptions.
Another meta element, if implemented incorrectly, that can also trip up websites is robots.txt. I was recently reminded of this while reviewing the website of a big ompany that had spent considerable money on building a mobile version of their website, on a sub-directory. That’s fine, but having a negative statement in their robots.txt file meant that the website wasn’t accessible to search engines.