Does Let's Encrypt follow redirects to HTTPS?

Yes. HTTP-01 starts on port 80 but follows redirects to HTTP or HTTPS on ports 80 and 443. If the HTTPS side returns 403 for the challenge path, you get this error, so the exception must exist in both server blocks.

How can I test without using up retries?

Write a test file into the challenge directory and fetch it from outside. Then use certbot certonly --dry-run, which runs against the staging environment and does not count toward production limits.

What does 404 instead of 403 mean?

404 means the server allowed the request but did not find the file: the webroot passed to Certbot is not the directory the server serves for that name, or another server block answers. 403 means the request was refused.

Certbot "unauthorized ... Invalid response: 403" fix

This error means Let's Encrypt reached your web server on port 80, requested the HTTP-01 challenge file under /.well-known/acme-challenge/, and was answered with 403 Forbidden. Something in front of or inside the web server refuses that path: a "deny dotfiles" rule, a WAF or CDN, or file permissions.

Quick fix

An explicit allow rule for /.well-known/acme-challenge/ fixes most 403 failures.

Reproduce the request yourself, from outside your network:

sudo mkdir -p /var/www/html/.well-known/acme-challenge
echo ok | sudo tee /var/www/html/.well-known/acme-challenge/test
curl -i http://example.com/.well-known/acme-challenge/test

If you also get 403, add an explicit exception before other rules. In Nginx:

location ^~ /.well-known/acme-challenge/ {
    root /var/www/html;
    default_type "text/plain";
    allow all;
}

Reload (sudo nginx -t && sudo systemctl reload nginx), confirm curl returns ok, then run sudo certbot renew --dry-run.

Causes

A web server rule that denies paths starting with a dot is the most common cause.

1. A rule denies hidden paths

Many hardening snippets contain location ~ /\. { deny all; } (Nginx) or a <FilesMatch "^\."> / RedirectMatch 403 /\. rule (Apache). They block /.well-known/ too.

Confirm: curl above returns 403 from your server (check the Server: header), and the rule exists in nginx -T or apachectl -S output.
Fix: add the ^~ /.well-known/acme-challenge/ location shown above (the ^~ modifier stops regex locations from matching). In Apache:

<Location "/.well-known/acme-challenge/">
    Require all granted
</Location>

2. A WAF, CDN or security plugin blocks the request

Bot protection, country blocking and some CMS security plugins return 403 to unknown clients. Let's Encrypt validates from several network locations, so blocking any of them can fail validation.

Confirm: the 403 page or Server: header comes from the CDN or WAF, or the plugin's log shows the block.
Fix: add a rule that skips protection for /.well-known/acme-challenge/*, or use DNS-01.

3. File system permissions or SELinux

Confirm: the web server's error log shows Permission denied for the challenge file; on RHEL-family systems ls -Z shows a label other than httpd_sys_content_t.
Fix: make the directory traversable and the file readable by the web server user (chmod 755 for directories, 644 for files), and run sudo restorecon -Rv /var/www.

4. The HTTPS side or the application returns 403

Your port 80 server redirects to HTTPS, and the HTTPS server block or the application (a framework router, an authentication layer) denies the path.

Confirm: curl -iL shows a 301 to HTTPS, then 403.
Fix: add the same exception to the HTTPS server block, or serve the challenge on port 80 before the redirect.

5. IP allowlists or basic auth on a staging site

Sites protected with allow/deny lists or HTTP basic authentication return 401 or 403 to the CA.

Fix: exempt the challenge path, or use DNS-01 for non-public sites.

Fixes by environment

Every environment needs the same exception: /.well-known/acme-challenge/ must answer over plain HTTP without authentication or bot checks. acme.sh reports the same failure as a Verify error.

Certbot --nginx / --apache: the plugin adds a temporary rule, but a site-wide deny rule or an include processed earlier can still win; use --webroot with the explicit location above if it keeps failing.

Cloudflare and other proxies: disable "under attack" style challenges for the path or temporarily set the DNS record to DNS-only while validating.

WordPress and similar CMSs: some security plugins add .htaccess rules denying dot-paths; whitelist /.well-known/acme-challenge/.

Docker: make sure the container that answers port 80 shares the webroot volume Certbot writes to.

Don't do this

Remove all dotfile protections. They stop leaks of .git, .env and similar files. Add a narrow exception instead.
Retry repeatedly against production. Failed validations count toward the limit of 5 authorization failures per identifier per hour; see Let's Encrypt rate limits.
Turn off the WAF entirely. Exempt only the challenge path, or switch to DNS-01.