Create and customize your robots.txt file in seconds with our professional SEO-optimized tool
Create your robots.txt file in seconds with our intuitive interface and guided configuration.
Define which areas of your site should be indexed and which shouldn't to optimize SEO.
Access and use our tool from any device, anytime.
The robots.txt file is a text file that is part of the Robots Exclusion Protocol (REP) and is used to tell search engine crawlers which pages or files they can or cannot request from your site.
A well-configured robots.txt can:
The most common directives are:
User-agent: Specifies which robot the rules apply toDisallow: Indicates which paths should not be crawledAllow: Indicates which paths can be crawled (overrides Disallow)Sitemap: Specifies the location of the XML sitemapCrawl-delay: Sets the delay between crawl requestsHost: Indicates the preferred domain (unofficial directive)User-agent: * to apply rules to all robots
Manage your cookie preferences. You can enable or disable different types of cookies below.
These cookies are essential for the website to function and cannot be switched off.
These cookies allow the website to remember choices you make (like language or region) and provide enhanced features.
These cookies help us understand how visitors interact with our website by collecting anonymous information.
These cookies are used to track visitors across websites to display relevant advertisements.
Under the GDPR, you have the right to access, rectify, and erase your personal data. You can also object to processing and request data portability.