Robots.txt files can be used to tell search engine crawlers and other web robots what you want them to do with your content or site.
In essence, they are telling robots where they can go on your site and where they can’t go.
Robots.txt files are simple text documents that need to adhere to a strict syntax in order for them to work properly with the various programs that use them. The robots.txt is primarily analyzed in many Technical SEO services offerings.
Besides specifying the directories and pages that bots should not crawl. The most common use is for preventing crawlers from accessing parts of a website that shouldn’t be publicly available, such as login pages, directories without any indexable content like error pages or web archives, etc.
Now that you know a little bit about the robots.txt file, let’s visit some interesting facts about it.
Where the Robots.txt File Lives
The robots.txt file should always be in the root directory of your website and the filename should always be “robots.txt”. This enables you to control all directories and subdirectories under the root directory without having to list them all out individually in the robot’s instructions.
The files location is typically exampledomain.com/robots.txt.
Should You Block Pages from Google?
When you block pages from the search engine crawls, it means that they can’t be indexed. This means that they won’t appear in the search results at all. But, because of this, you lose some SEO value and traffic to these pages. Even though no one will ever see these blocked pages in a search engine result page (SERP), they are still available on your website, so blocking them is not necessary.
There really isn’t a reason to be blocking pages from Google. Google wants to view what users do, so you really shouldn’t be blocking Google or other search engines from something such as a login page. Plus what if someone Google searched “your brand + login”?
Your Sitemap Should be Called in Your Robots.txt File
it is important to include the sitemap of your site in the robots.txt file. This will prevent automated bots from indexing parts of your website that you do not want them to index (if you have any that make sense to block).
If you have a large website, then adding many sitemaps into the robots.txt file can be a time-consuming job. However, there are services such as Yoast’s SEO plugin or Raven for WordPress, which can automatically generate your sitemap for you and make it available in the robots.txt file within minutes without any manual work required on their part at all.
