Update your WordPress robots.txt file to reduce index bloat and unnecessary crawl activity. These are the things that you should always include and never block.
The modest robots.txt file frequently runs in the background of a WordPress website. By default, it is really simple and, of course, doesn’t support any custom directives you might wish to use.
Let’s get right to what else you can add to improve it—no further introduction is necessary.
The WordPress Robots.txt File: Where Is It?
Simply upload a custom robots.txt file to the install’s root folder if you wish to utilize one.
Either an FTP program or a plugin, like Yoast SEO (SEO → Tools → File Editor), which has a robots.txt editor accessible from the WordPress admin area, can be used for this.
The Robots.txt File by Default in WordPress (And Why It’s Insufficient)
Incorporate your XML sitemap(s) at all times.
Ensure that every XML sitemap is stated explicitly, as this aids search engines in finding all pertinent URLs.
Some Things Not to Block
Some essential WordPress folders, such as /wp-includes/, /wp-content/plugins/, or even /wp-content/uploads/, have been suggested to be blocked in outdated versions. Don’t!
The following justifies not blocking them:
- Google is intelligent enough to reject irrelevant files. Blocking JavaScript and CSS can affect indexing and renderability.
- You may inadvertently block important images, movies, and other material, particularly those loaded from /wp-content/uploads/, which houses all uploaded media you want to be indexed.
Instead, allow crawlers to retrieve the images, JavaScript, and CSS required for correct rendering.
In charge of Staging Locations
Keeping staging sites safe from crawling is good for SEO and overall security.
The recommendation is to disallow the entire site.
The noindex meta tag should still be used, but it’s still a good idea to utilize both to ensure another layer is covered.
If Google finds links to other websites (typically from production calls to staging when migration isn’t perfect), it may still index those pages.
Important: Make sure to verify this setting again after moving to production to make sure that any disallowing or noindexing has been reverted.
Clean Up Some Non-Essential Core WordPress Paths
Although not all default paths should be blocked, many of them have little SEO value, like the one below:
Block Particular Query Parameters
There are situations when you want to prevent search engines from crawling URLs that contain known low-value query parameters, such as print versions, comment answers, or tracking parameters.
Low-Value Taxonomies and SERPs Are Prohibited
If your WordPress website has internal search results pages or tag archives that don’t provide anything useful, you may also block them:
Additionally, ensure that your internal linking structure eliminates internal linking to locations you do not want to index or crawl and supports your choice.
Monitor On Crawl Stats
Once your robots.txt file is set up, use Google Search Console to track crawl statistics:
- Check the Crawl Stats under Settings to determine whether bots are squandering resources.
- To find out if a prohibited URL is indexed or not, use the URL Inspection Tool.
- Verify sitemaps to ensure they only include the pages you want to be indexed and crawled.
Furthermore, in addition to Google, some server administration software, such as Plesk, cPanel, and Cloudflare, can offer incredibly extensive crawl statistics.
Lastly, Screaming Frog’s configuration override will simulate changes and investigate Yoast SEO’s crawl optimization settings, some of which deal with the previously described problems.
Final Thoughts
Though WordPress is an excellent content management system, it isn’t configured with crawl optimization or the best default robots.txt setting.
With just a few lines of code and less than half an hour of your time, you can prevent thousands of pointless crawl requests to your website that aren’t worth identifying. You can also prevent a future scaling problem.
