The page opens normally on your phone, but a crawling tool gets a 403. This is possible when a CDN, WAF, server or plugin apply different rules depending on the request. Before editing the SEO text, establish which layer denies access and whether the affected visitor is actually a search crawler.
Match the exact URL, time and IP address of the failed request with the CDN and server logs. Verify the crawler's origin, find the specific rule and make a limited fix for the necessary public access. Then repeat the real test and check the HTTP status, the content and indexability. Don't allow access based on the User-Agent alone.
What HTTP 403 means in this check
403 means access denied. The status alone doesn't say who denied it or whether it's a mistake. For an admin address the denial may be correct; for a public product page and a legitimate search crawler it may be unwanted.
Also check 200 responses that contain only a security challenge instead of the page. A successful HTTP status doesn't prove the bot got the useful content. A screenshot from your browser doesn't represent the conditions of the automated request either.
Collect data before making a change
| Data | Why it's needed |
|---|---|
| Exact URL and method | Different paths and GET/HEAD can be handled differently |
| Time and time zone | For matching with the logs |
| IP and claimed User-Agent | For verifying the origin |
| HTTP status and body | To distinguish a denial from a challenge |
| Request identifier | For example the Ray ID in a Cloudflare response |
| Rule and action | Which mechanism blocked or challenged the request |
Keep at least one successful and one failed request to the same public page. Note the differences without assuming in advance that the cause is an "SEO setting".
How to verify that it's Googlebot
The string "Googlebot" in the User-Agent can be spoofed. Google publishes verification methods using current IP ranges or a reverse DNS lookup followed by a forward DNS lookup to the original IP. Match against the right category of crawler or fetcher: not all requests from Google have the same role.
For other platforms, use their official identification methods. Don't automatically apply a Google rule to every AI crawler. Different systems may perform search, retrieval on a user's request or other activities, for which the site owner has different access policies.
CDN, WAF or origin: where is the denial?
- Look for the request in the CDN's available security events and see the action and the matching rule.
- Compare with the server access and error logs.
- If the request was stopped before the server, start with the security layer.
- If it reaches the server and the server returns 403, check server rules, access permissions and security plugins.
- If the data is missing, check the logging scope, period and limits; a missing record is not final proof.
A technical check against the origin can be useful for an administrator with permitted access and the correct configuration. Don't reveal the origin address publicly, and don't change its protection as a general solution.
What to check in the security rules
Possible causes include a custom WAF rule, a country or network restriction, bot protection, rate limiting, a protected path or a server configuration. Establish which rule matched the real request. Having Cloudflare doesn't prove it is the source of the denial.
Cloudflare supports verified bot signals, but having them doesn't make all your own rules automatically compatible. If needed, use a limited exception for the right public path and the verified request type. The Skip action has a specific scope and order of execution; check which products and checks it can actually skip in your plan and configuration.
Don't create a global "Allow" based on text contained in a header. Don't turn off all protections for the site when the problem is one rule. Record the change, the expected result and a way to roll it back.
Robots.txt, noindex and canonical solve different tasks
An HTTP denial happens at the access stage. Robots.txt manages permitted crawling for crawlers that respect it; noindex and canonical have separate roles when content is processed. Changing the meta description doesn't fix a blocked request.
Once access works, check whether the public page itself has the desired robots directives and canonical, whether robots.txt is accessible and whether the main content is present. Allowing the crawler doesn't guarantee indexing, ranking or inclusion in an AI answer.
Check after the fix
- Repeat the affected real test for the same URL.
- Check the final status, redirects and the HTML received.
- Confirm in the logs that the rule no longer blocks the legitimate request.
- Check representative pages and robots.txt, not just the home page.
- Make sure the admin and sensitive paths remain protected.
- Monitor repeated denials and the subsequent crawling and indexing signals.
A local test with a spoofed User-Agent can show how a rule reacts, but it doesn't prove access for the real crawler from its infrastructure. Use it as an auxiliary diagnostic, not as final confirmation.
For the next checks see why the site isn't indexed in Google. DigitaL+ combines technical SEO with accessibility and indexing in AI search engines. The general approach to visibility is in AEO and GEO without promises of automatic ranking.
Frequently asked questions
If my browser opens the site, does the crawler have access?
Not necessarily. The browser may pass a challenge, have a cookie or come from a different network.
Should we allow all automated visitors?
You don't have to. Define which public resources you want to be accessible and for which verified categories of requests.
Is a 403 always the reason for missing indexing?
No. It's a specific access problem. After the fix, other technical or content causes may remain and must be checked separately.
