Cloudflare can block SEMrush crawlers from accessing your site by setting a Cloudflare rule. The problem occurs when the Cloudflare rules are set to “Block All” bots and not specific IPs. This is an issue for legitimate crawlers like SEMrush, but it’s also an issue for those that scrape content without permission or scraping malware. How do you fix this?

This usually happens when Cloudflare detects an attack from the IP address of your server or if Cloudflare thinks you are using too much bandwidth. Today we will show you two steps to help SEMrush crawl your website again and get started with SEO keyword tracking! Here is a quick tutorial on how to fix Cloudflare blocking SEMrush crawler.

SEMrush Audit Error

If you run SEMrush Audits, you might have experienced SEMrush being blocked from auditing a website; you will see an error code like this:

Site Audit Settings
Site Audit Settings

Sorry, auditing domain.com has failed

We encountered an error that stopped us from crawling your website:
Network connection was interrupted.
We couldn’t crawl your homepage due to connectivity issues on your site.

How to fix it
This issue is typically caused by your hosting provider blocking our crawler. Please download a log file and contact your website administrator to fix this error. See how to whitelist SEMrushBot on different hosting providers.

How to fix SEMrush bot from being blocked

CloudFlare Error Codes
Cloudflare Error Codes

  1. Identify the Error

    First, go to your web pages and make sure they work in your browser.

    If you have an Ahrefs account, see if you can run a test with that SEO tool.

    Then, Check the error code by downloading the SEMrush log (shown in the images above). The error code console pops up when the crawl fails and creates a notification.

     

  2. Confirm it’s Cloudflare

    The second step is to identify that Cloudflare is blocking the audit tool since numerous things could be blocking your crawl (e.g., Robots.TXT settings, WordPress set to noindex, a blocked user-agent).

  3. Whitelist SEMrush’s IPs

    After confirming it was a Cloudflare blockage, you will need to Whitelist the SEMrush’s IP range in your Hosting:

    46.229.173.68
    46.229.173.67
    46.229.173.66

    Find out how to whitelist your IP range in CloudFlare; see below.

How to Whitelist SEMrushBot in Cloudflare:

1. Log in to your Cloudflare.com account.

2. Select your domain name.

Select your Domain
Select your Domain

3. Click the Cloudflare Firewall >> Tools sub-menu settings.

tools

4. Under IP Access Rules, enter the following details in steps 5-6.

IP Access Rules
IP Access Rules

5. Enter the SEMrush Values. (46.229.173.66, 46.229.173.67, 46.229.173.68)

6. Select an Action >> Allow.

7. Select whether the rule applies to This website or All websites in the account.

8. Add a note (i.e., SEMrush).

9. Click Add.

10. Purge your cache and wait at least an hour before trying to audit your website again. This can be done by clicking on “Purge Everything” under the Caching tab.

Caching Tab
Caching Tab
Purge Cache
Purge Cache

After the cache has cleared for SEMrush, your audits should have no issues running.

I hope this was helpful in understanding how Cloudflare can block SEMrush crawlers from accessing your site. If you have any questions about this, please email me.

FAQs

Why is Cloudflare blocking me from websites?

Cloudflare will block your connection to websites if it determines you have a hostile IP address. This hostile assessment can be from performing aggressive SEO or Infosec Security Audits, a shared IP address with bad neighbors, a flagged VPN connection, black hat software on your computer, malware, a hacked computer, or participating in frivolous activity online.

What is Cloudflare?

Cloudflare is a Content Delivery Network (CDN), DNS provider, and DDoS mitigation service, acting as a reverse proxy for websites. This means it delivers your website across its edge locations globally to clients, making it faster for visitors and safer against hackers.

What is the SEMRush bot?

SEMRush is a bot that helps you to track your website’s search engine rankings. It does this by checking your website’s position in the SERPs (search engine results pages) for a given set of keywords. SEMRush also allows you to view your website’s traffic data, such as the number of visitors and the number of pageviews. This information can be useful for understanding how your website is performing in search engines. Additionally, SEMRush can help you to research your competition and find new opportunities for SEO (search engine optimization). Overall, SEMRush is a powerful tool that can help you to improve your website’s visibility in search engines.