CloudFlare blocking SEMrush crawler [how-to fix]

CloudFlare blocking SEMrush crawler [how-to fix]

If you run SEMrush Audits, you might have experienced SEMrush being blocked from auditing a website.

You will see an error code like this:

Site Audit Settings
Site Audit Settings
Sorry, auditing domain.com has failed

We encountered an error that stopped us from crawling your website:
Network connection was interrupted.
We couldn't crawl your homepage due to connectivity issues on your site.

How to fix it
This issue is typically caused by your hosting provider blocking our crawler. Please download a log file and contact your website administrator to fix this error. See how to whitelist SEMrushBot on different hosting providers.

How to fix SEMrush bot from being blocked

  1. Check your error code by downloading the log.
  2. Whitelist SEMrush's IP range in your Hosting, or remove any Robots.txt blockages.
  3. Purge your cache
CloudFlare Error Codes
CloudFlare Error Codes

As you can see above, Cloudflare is blocking my SEMrush Audit scan, find out how to fix it below:

How to whitelist SEMrushBot in Cloudflare:

Log in to your Cloudflare account.

Select your domain.

Select your Domain
Select your Domain

Click the Firewall >> Tools sub-menu settings.

Under IP Access Rules, enter the following details:

IP Access Rules
IP Access Rules
  • Enter the SEMrush Values. (46.229.173.66, 46.229.173.67, 46.229.173.68)
  • Select an Action >> Allow.
  • Select whether the rule applies to This website or All websites in the account.
  • Add a note (i.e., SEMrush).

Click Add.

Purge your cache and wait at least an hour before trying to audit your website again.

This can be done by clicking on "purge everything" under the caching tab.

Caching Tab
Caching Tab
Purge Cache
Purge Cache