How to Create a Robots.txt File in WordPress

A properly configured robots.txt file helps search engines understand which parts of your WordPress website they should crawl. For bloggers, business owners, and website managers, learning how to create a WordPress robots.txt file is an important step toward maintaining a healthy technical SEO setup.

In this guide, you'll learn what robots.txt does, how to create or edit it in WordPress, which rules to use, and how to check for common mistakes.

What Is a WordPress Robots.txt File?

A robots.txt file is a plain-text file located in your website's root directory. It provides instructions to search engine crawlers about which URLs they should or should not crawl.

For example, you can access the file by visiting:

https://yourwebsite.com/robots.txt

WordPress can generate a virtual robots.txt file automatically if a physical file does not exist. You can customize the rules when your website requires more specific crawling instructions.

Remember that robots.txt is not a security tool. It does not prevent visitors from accessing restricted pages, and blocked URLs may still appear in search results.

Why Is Robots.txt Important for WordPress SEO?

A well-managed WordPress robots.txt file helps you control crawling and avoid unnecessary restrictions.

Its main benefits include:

• Manage crawling: Discourage crawlers from visiting areas that do not need to be crawled.
• Improve crawl efficiency: Help search engines focus on useful website URLs.
• Prevent crawling mistakes: Avoid accidentally blocking important pages or resources.
• Support technical SEO: Maintain appropriate crawling rules as your website grows.

Robots.txt does not directly guarantee higher rankings. Its primary purpose is to provide crawling instructions to compatible search engine bots.

How to Create a Robots.txt File in WordPress

There are three common ways to manage your WordPress robots.txt file.

Method 1: Check Your Default Robots.txt File

Before creating a new file, check whether WordPress already generates one.

1. Open your web browser.
2. Visit https://yourwebsite.com/robots.txt.
3. Review the displayed instructions.
4. Check whether important website sections remain accessible to search engines.

If the existing configuration meets your needs, you may not need to make any changes.

Method 2: Use an SEO Plugin

SEO plugins such as Yoast SEO and Rank Math may provide tools for managing robots.txt, depending on the plugin version and configuration.

Follow these steps:

1. Log in to your WordPress dashboard.
2. Open your SEO plugin's settings or tools.
3. Look for a robots.txt editor, if available.
4. Review the existing rules carefully.
5. Make the necessary changes and save them.
6. Visit your website's /robots.txt URL to verify the result.

This method is convenient for beginners who prefer managing technical SEO settings through the WordPress dashboard.

Method 3: Create the File Manually

You can also create a physical robots.txt file through your hosting control panel.

1. Log in to your hosting account.
2. Open File Manager or connect using an SFTP client.
3. Navigate to your WordPress installation's root directory, commonly named public_html.
4. Check whether a robots.txt file already exists.
5. Create a plain-text file namedrobots.txt or carefully edit the existing file.
6. Add your crawling rules and save the changes.
7. Visit https://yourwebsite.com/robots.txt to confirm that the file is accessible.

Make sure the file is in your website's root directory, not inside a subfolder such as /wp-content/.

Example of a WordPress Robots.txt File

If you need a simple starting configuration, consider this example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourwebsite.com/sitemap_index.xml

Replace yourwebsite.com with your domain and ensure the sitemap URL matches your actual sitemap location.

Here is what each directive means:

• User-agent: * applies the following rules to all matching crawlers.
• Disallow: /wp-admin/ requests that crawlers avoid URLs under the WordPress admin directory.
• Allow: /wp-admin/admin-ajax.php explicitly permits crawling of the specified AJAX endpoint.
• Sitemap: identifies your XML sitemap for crawlers that support the directive.

This configuration is only an example. Review your website's requirements before applying these rules, and avoid blocking CSS or JavaScript resources that search engines need to render your pages.

How to Add a Sitemap to WordPress Robots.txt

An XML sitemap helps search engines discover important URLs on your website. Adding its location to robots.txt is a useful practice.

For example:

Sitemap: https://yourwebsite.com/wp-sitemap.xml

WordPress commonly provides a default sitemap at /wp-sitemap.xml, while SEO plugins may generate a sitemap index at /sitemap_index.xml.

To add your sitemap:

1. Check your SEO plugin's sitemap settings.
2. Open the sitemap URL in your browser.
3. Confirm that it loads correctly.
4. Add the complete URL to your robots.txt file.

A sitemap helps search engines discover URLs, but it does not guarantee that every page will be crawled or indexed.

Common Robots.txt Mistakes to Avoid

Incorrect rules can create crawling problems. Avoid these common mistakes:

• Blocking your entire website: The rule Disallow: / can prevent compliant crawlers from accessing all URLs.
• Blocking important content: Broad restrictions may accidentally block blog posts, landing pages, or products.
• Blocking CSS and JavaScript: Search engines may need these resources to understand your page layout.
• Using robots.txt to remove indexed pages: A blocked URL may still appear in search results. Use an appropriate noindex directive when suitable.
• Adding an incorrect sitemap URL: Always verify that your sitemap exists and is accessible.
• Treating robots.txt as security: Use authentication and access controls to protect private information.

Review your rules carefully before publishing changes.

How to Test Your WordPress Robots.txt File

After creating or editing your file, follow these steps to check that it works correctly.

Step 1: Open the file. Visit https://yourwebsite.com/robots.txt and confirm that your latest rules appear.

Step 2: Check for errors. Review directive names, directory paths, and sitemap URLs for spelling mistakes or unintended restrictions.

Step 3: Review Google Search Console. Use the available robots.txt reporting features and URL Inspection tool to investigate crawling problems.

Step 4: Test important pages. Check representative blog posts, landing pages, and product URLs to ensure your rules do not unintentionally restrict crawling.

If the file displays outdated content, investigate caching, hosting settings, and any physical robots.txt file that may override WordPress's virtual version.

Frequently Asked Questions

Does WordPress automatically create a robots.txt file?

Yes. WordPress can generate a virtual robots.txt file when a physical file is not present in the website's root directory.

Is a robots.txt file necessary for WordPress SEO?

WordPress websites can function with the default configuration. Custom rules help when you have specific crawling requirements.

Can robots.txt prevent a page from appearing on Google?

Not reliably. Google may index a blocked URL if it discovers it elsewhere. Use an appropriate noindex directive or access restrictions, depending on your goal.

How often should I update robots.txt?

Review it periodically and whenever your website structure, crawling requirements, or technical SEO strategy changes.

Final Thoughts

Creating a WordPress robots.txt file is straightforward when you choose the right method and understand the rules you are applying. You can use WordPress's default configuration, an SEO plugin, or your hosting File Manager.

Start with a simple setup, verify your sitemap URL, and test your changes before relying on them. For more practical guidance on technical SEO and website optimization, visit Free Local SEO Audit.