Skip to content

fix: persist Let's Encrypt certificates across container lifecycle#88

Open
iambaboucarr wants to merge 7 commits intomasterfrom
fix/letsencrypt-rate-limits
Open

fix: persist Let's Encrypt certificates across container lifecycle#88
iambaboucarr wants to merge 7 commits intomasterfrom
fix/letsencrypt-rate-limits

Conversation

@iambaboucarr
Copy link
Copy Markdown
Collaborator

@iambaboucarr iambaboucarr commented Feb 26, 2026

Summary

  • Replace the cert Docker named volume with a ./traefik/cert bind mount so ACME certificates persist through docker compose down --volumes, container recreation, and volume prune operations.
  • Add configurable LETSENCRYPT_ACME_CASERVER env var (defaults to production) so operators can switch to the Let's Encrypt staging server during testing to avoid rate limits.
  • Update the Ansible deploy role to create traefik/cert/ directory and traefik/cert/acme.json with correct ownership, aligning it with the new bind mount path.

Problem

Running the deployment multiple times (e.g. make clean && make launch) would destroy the cert named volume each cycle, wiping all cached certificates and ACME account data. Traefik then re-requested certificates from production Let's Encrypt, hitting the 5 duplicate certificates per week rate limit after a few iterations. Once rate-limited, Traefik could not obtain valid certificates and browsers showed certificate errors as encountered by @bobjolliffe during his last demo.

Test plan

  • Fresh deployment: verify ./traefik/cert/acme.json is created with correct permissions (600, nobody:nobody)
  • Restart cycle: run make clean and make launch to confirm that certificates are preserved
  • Staging server: set LETSENCRYPT_ACME_CASERVER to staging URL, verify Traefik uses staging CA
  • Ansible deploy: run playbook on a test host, confirm traefik/cert/acme.json is created at the correct path

@iambaboucarr iambaboucarr force-pushed the fix/letsencrypt-rate-limits branch from 138d2c8 to 2674db8 Compare February 26, 2026 21:55
Copy link
Copy Markdown
Contributor

@radnov radnov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the PR is good, if we were to accept the bind mount approach.

However, I'd rather we keep using a volume for the cert, unless a bind mount is really necessary.

I would go with changing make clean to something like:

  • make clean-all (with --volumes opt - destroys volumes, with confirmation prompt)
  • make clean (doesn't destroy volumes)

Also, the extra LetsEncrypt Staging env var is good idea in any cases. 👍

@iambaboucarr iambaboucarr requested a review from radnov March 17, 2026 09:23
Copy link
Copy Markdown
Contributor

@radnov radnov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 Well done!


set -eu

mkdir -p /cert
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this folder created automatically by Docker when we mount the volume?

@@ -62,7 +62,6 @@
file:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this whole task be removed since we're handling all of this in an init container now?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original task was introduced by you in a previous PR. We can discuss the removal in an upcoming call

# Ansible
.ansible/

traefik/acme.json
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this line is a leftover from when we initially used a bind mount for the acme file

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's important to retain the .ansible/ dir, as IDEs that have the Ansible extension installed (e.g on VSCode) will create a .ansible/ dir.

.gitignore Outdated
# IDE
.vscode/

# Ansible
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment seems a bit redundant 🙂

README.md Outdated

### Let's Encrypt Certificate Management

Traefik uses Let's Encrypt ACME and stores certificate data at `./traefik/cert/acme.json` on the host filesystem.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It really doesn't 😂

Copy link
Copy Markdown
Collaborator Author

@iambaboucarr iambaboucarr Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is nothing funny here. This is a stale text from the original PR that used bind mount

@@ -15,5 +15,8 @@ Thumbs.db
# IDE
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When did we starting adding these comments? At the top of this file we ignore .idea which is also from an IDE

Copy link
Copy Markdown
Collaborator Author

@iambaboucarr iambaboucarr Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had the comments from the very beginning of the project, although it was a bit inconsistent

@tonsV2 tonsV2 self-requested a review March 17, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants