Lets’ Encrypt revoked certificate! Did anyone experience this?

[nachham]

Hi

I woke up this morning and my 110-Student-Production-Tutor-Instance was not accessible! Guess what! Let’s Encrypt revoked the certificate! However, the renew is due about two months ahead.

I am on Lilac.So it’s Caddy who’s in charge. Running Quickstart didn’t help. Assigning a new P address didn’t help neither. Not cool as there is a due Quiz-assignment today!

I had to change the domain name – with a new IP, literally, and assign it to the instance…. Well this worked, but not the best solution as this is an on-flight change. Obviously, I had to inform students about this change. I regret that this happened without prior warning!!

Checking the news: Let’s encrypt seemed to revoke purposely millions of SSLs https://traefik.io/blog/how-to-force-update-lets-encrypt-certificates/

Obviously, my Tutor instance was one of them!

Did any one experienced, this?

What is the best way to tackle this issue if this ever happens again?, Changing the domain name, the way I did it, was an extreme move, well just to keep going!

Thanks for your help!

Cheers.

[regis]

Thanks for the heads-up. It appears that Caddy 2.4.2 is not affected: Questions about Renewing before TLS-ALPN-01 Revocations - #21 by mholt - Help - Let's Encrypt Community Support

Problem: Tutor is running Caddy 2.3.0. I have no explanation for the fact that it was not upgraded in Maple, other that it was an oversight.

I will now push an upgrade to fix the Caddy Docker image version. The new version will be published in a few minutes. You should upgrade to tutor v13.1.2 to get the fix:

pip install tutor==13.1.2
tutor local reboot

Users who may not be able to upgrade should manually set the Caddy Docker image:

tutor config save --set DOCKER_IMAGE_CADDY=docker.io/caddy:2.4.6
tutor local reboot

This is only a temporary fix. As soon as you upgrade, the Caddy Docker image version should be unpinned with tutor config save --unset DOCKER_IMAGE_CADDY.

If for some reason your certificates are still invalid, take the nuclear route by forcing revocation of all certificates:

tutor local dc down
tutor local run caddy rm -r /data/caddy
tutor local start -d

[nachham]

Thanks @regis

Keeping Lilac on production, (at least till the end of the semester), do you think revocation might happen again, (owing to the fact that the new domain is up-and-running, now)?

Or should i apply the fix, ASAP (tough i am reluctant to apply it to a production-running-instance)?

Thank you

HN

[HenryV]

Yes, we’re also experiencing this. We are still on 12.0.04, not yet ready to upgrade as the system is very busy this week.

[HenryV]

I am running Lilac (12.0.4). Will this manual fix work for it? (ie the DOCKER_IMAGE_CADDY version setting)

[regis]

I believe it should. If it doesn’t please report here.

[HenryV]

It worked perfectly with 12.0.4. Thank you so much, Regis.

[nachham]

Hi,

What is the command to check the running Caddy version? Thank you!

HN

[mlebeda]

Hello,
You can check Caddy version for example with docker ps -a or tutor local exec caddy caddy version or for example docker exec -it tutor_local_caddy_1 caddy version.

[nachham]

Thank you!
Cheers
HN