Robots.txt Generator

The robots.txt file is used to tell crawlers, mostly search engines, the path of directories or files that can be accessed on your website.