Generate a robots.txt file for your site.
A robots.txt file tells search engine crawlers which pages to index and which to skip. It sits at your website root (example.com/robots.txt).
