Robots.txt files can be used to tell search engine crawlers and other web robots what you want them to do with your content or site.
In essence, they are telling robots where they can go on your site and where they can't go.
Robots.txt files are simple text documents that need to adhere to a…
