When reading Caddy documentation recently, I noticed that instead of something you can get TLS, Caddy is actually capable of being a good web server. If we’re using Caddy for TLS only, should we, in the long term, remove the use of Nginx, and use Caddy itself only? As two reverse proxy seems redundant.
What do you think? I’d be glad to work on this.
That’s a good question @iamCristYe. It was raised before here: Editing nginx lms and cms.conf - #5 by MaartenGH
Basically, I agree that Caddy and Nginx serve somewhat similar goals. But Caddy is a much thinner layer that does only SSL termination. Because users may want to use a different proxy for SSL termination, there is a simple mechanism to completely remove Caddy from a working Tutor deployment. But in that case, we will still need a web server to redirect requests to the various containers. So we decided to keep nginx. In addition, there is the question of backward compatibility of nginx patches and plugins.
Still, there might be a way to keep Caddy and remove Nginx, even in the case of custom SSL termination. I’m open to suggestions here.
Edit: This should actually be a reply to this thread. It’s still somewhat relevant to the discussion about custom SSL termination here, though.
Hi @regis and @iamCristYe, something that may be of interest is Caddy’s tls directive. I have had some success using this.
By changing the Caddyfile to something like:
your.lms.domain {
tls /etc/ssl/yourcert.cer /etc/ssl/yourcert.pem
reverse_proxy nginx:80 {
header_up X-Forwarded-Port 443
}
}
for each of your domains, you can keep the Nginx configuration the same, and still use custom certificates. You will have to mount your certificate files to the Caddy container at the appropriate locations.
Arguably this is a bit redundant since you could just have Nginx handle TLS at that point. Though, Caddy’s succinct configuration and security-focused updates mean you still get some of the benefits of Caddy (as security and encryption best-practices change over time).
One problem is that it isn’t possible to achieve this configuration, as far as I’m aware, using Tutor plugins’ template patching alone.
To partially address this (along with a few other use-cases), I’ve been developing a Tutor plugin, tutor-contrib-recon (short for “reconfigure”), which allows for direct overrides of settings and templates in your Tutor environment during development. The goal is to ultimately make it easy to roll such changes either into a plugin or a modification to Tutor.
Right now it is a work in progress, but the template-override functionality is more-or-less implemented (though unstable). After installing and enabling the plugin, you can do this:
tutor recon replace-template apps/caddy/Caddyfile
Tutor’s template for the Caddyfile is then copied into $(tutor config printroot)/env_overrides/templates/apps/caddy/Caddyfile) (note: $(tutor recon printroot) is short for $(tutor config printroot)/env_overrides, so you can use that here instead).
You can then change it to something like:
{{ LMS_HOST }} {
tls {{ TLS_CERTS.public.container }} {{ TLS_CERTS.private.container }}
reverse_proxy nginx:80 {
header_up X-Forwarded-Port 443
}
}
{{ PREVIEW_HOST }} {
reverse_proxy nginx:80
}
{{ CMS_HOST }} {
reverse_proxy nginx:80
}
{{ patch("caddyfile") }}
and add settings to Tutor’s config.yml like:
TLS_CERTS:
private:
container: /etc/ssl/yourcert.pem
local: /wherever/your/cert/is/locally/yourcert.pem
public:
container: /etc/ssl/yourcert.cer
local: /wherever/your/cert/is/locally/yourcert.cer
With that finished, you’ll just need to use tutor recon save to render your new template. You will need to continue using tutor recon save instead of tutor config save in the future to allow the correct template to be rendered.
You’d also want to modify your docker compose template(s) to bind-mount your cert files by adding to the volumes key in the Caddy service like:
- {{ TLS_CERTS.public.local }}:{{ TLS_CERTS.public.container }}
- {{ TLS_CERTS.private.local }}:{{ TLS_CERTS.private.container }}
If you want to try to do this, let me know how it goes! The project is definitely in early phases, so I’d be interested to see if you find it usable for this purpose.
Hi @thomas-skillup thanks a lot for your sharing! Your work for overriding default templates is amazing and I’m really looking forward to it. But I’d like to point out the following:
This is not quite right to my understanding. I think it could be easily done by adding a patch field in the file tutor/tutor/templates/apps/caddy/Caddyfile at master · overhangio/tutor · GitHub
Another thing is, if my understanding is correct, we’re talking about different things. I was actually talking about removal the use of Nginx completely. Maybe, an example Caddyfile would be better to facilitate understanding:
local.overhang.io:80 preview.local.overhang.io:80 {
@favicon_matcher {
path_regexp ^(.*)/favicon.ico$
}
rewrite @favicon_matcher /static/images/favicon.ico
request_body max_size 4MB
reverse_proxy lms:8000 {
header_up X-Forwarded-Port 80
}
}
studio.local.overhang.io:80 {
reverse_proxy cms:8000
}
Now I know that getting rid of Nginx would lead to backward compatibility: but as Caddyfile is easy to write, it wouldn’t be so difficult to migrate?
And if they are not using Caddy for TLS, users can simply set up an apache/nginx reverse proxy themselves. actually, maybe this is probably not what tutor should take care of.
To summarize, IMHO, we should provide Caddy only out of the box for a simple single-machine installation. If users are going to run multiple open edx on a single machine/host with other websites, we can provide some guidance on setting up the reverse proxy.
I would be glad to work on this. But of course, this could be a major change, and community’s voice do matter.
Ah, I read your question and @ashraf’s question around the same time and totally misread! My comment should have been added to that thread. Thank you for the kind words anyway.
A switch to Caddy only sounds promising! Simplifying the stack could definitely be beneficial.
Regarding patching the Caddyfile, I’m not sure how you’d do that without modifying the template, since there isn’t a patch field in all of the required places.
The template could be made more configurable via a contribution to Tutor, for instance, but creating said contribution would be a great reason to have the ability to quickly prototype changes. Quickly prototyping/developing template changes for this type of feature prior to contributing would be a solid use case for recon.
Thanks!
We can create a PR at the GitHub repo like:
{{ LMS_HOST }}{% if not ENABLE_HTTPS %}:80{% endif %} {
{{ patch("caddy_custom_certs") }}
reverse_proxy nginx:80 {
header_up X-Forwarded-Port {{ 443 if ENABLE_HTTPS else 80 }}
}
}
Yep, exactly! That change works, only difference would be you’d supply the whole line in the caddy_custom_certs patch.
Another reason for writing the plugin was that, while changing this template may be a viable contribution to Tutor, it’s possible that some types of changes may not be. So it’s convenient to have an automated way to handle one-off changes, at least temporarily. If you tried to just modify the template on your build, for instance, it would just be overwritten the next time you saved/modified your Tutor configuration through the CLI.
There is a lot of content in this thread! Let me try to address a few items.
On the topic of tutor-contrib-recon: while I do admire your Tutor plugin skills (you’ve really grokked the API!) I am a little bit concerned that people might actually start using recon too much. Let me explain: one of the biggest issues that Open edX platform administrators face is when they make changes to Open edX, and when a new release comes out they need to rebase their changes on top of the release branch. Rebasing is often very difficult because they made too many changes and now they no longer remember what each commit is supposed to do and its impact. Also, when they ask for help on the forums, they can’t find any, because people have no idea if the changes they made are causing the issue.
So, very early in the Tutor development process, I created this plugin mechanism which allowed anyone to add features on top of Tutor to avoid forks of the project. I had thought about a mechanism which would allow plugin developers to override any section of the templates (and not just the patch statements), but I did not implement it because I figured it would lead to unmaintainable forks – just like it does in the rest of Open edX.
Do you understand my position @thomas-skillup?
Anyway, the point of this conversation was to address the redundancy of Caddy and Nginx. Despite the fact that getting rid of nginx would lead to backward incompatibility, I would lean in favour of the simplification. However, I would implement the change differently:
RUN_CADDY (default: true) tutor setting.NGINX_HTTP_PORT to CADDY_HTTP_PORT (default: 80).CADDY_HTTPS_PORT (default: 443) setting. If this is set to None, Caddy will not perform SSL termination.{% if CADDY_HTTPS_PORT %}%{{ CMS_HOST }}:{{ CADDY_HTTPS_PORT }}, {% endif %}{{ CMS_HOST }}:{% CADDY_HTTP_PORT %} {
reverse_proxy cms:8000
{{ patch("caddy-cms")|indent(2) }}
}
In addition, we would have to add logging directives, to make sure that users can capture web analytics and errors. As this would be a breaking change, we would have to introduce it in a major release – in Maple, for instance.
What are your thoughts? If you agree we can create a Tutor Enhancement Proposal (#community:tep).
Hello @regis, thanks so much for the reply! I completely understand this position. Ultimately, keeping Tutor deployments as close to standard as possible is a major benefit for sanity and usability.
I do have a couple of additional pieces of information to add which might help to explain the rationale of the plugin.
First and foremost, the plugin was motivated by a desire to more easily change Edx app configuration and settings (l/cms.env.json and the settings.py files). It is certainly possible to affect these using template patches. However, especially for the *.env.json files, it felt to me that templating with Jinja wasn’t the best tool for my use-case.
If many plugins attempt to patch these files, it was hard for me to conceptualize how each plugin would interact. So, instead, I chose an approach that works like a dictionary “update”. Currently, only one such update can be applied (per the contents of a user’s env_overrides directory), though I’ve tentatively planned to add the ability to “stack” updates (and entire override directories) in a user-defined order.
Another point to note is that a user’s environment modifications are fully contained within their env_overrides directory. So, when maintaining modifications, only this folder needs to be checked into version control. I’ve also been planning to allow all such modifications to be “compactified” into a single file for production.
On the other hand, if a template is overridden, it is true that the user is now effectively responsible for maintaining a Tutor fork. I’d like to make it more clear that this should really be used as a last-resort option for admins with tight constraints on their deployment’s spec. Such admins will need to be willing to maintain these changes as Tutor makes changes prior to upgrading their platform.
It is still true, though, that only the env_overrides folder, which might include redefinitions of Tutor templates, needs to be tracked for this fork(-like repository).
To make tracking less difficult for overrides containing templates, I’ve toyed with a few different ideas. One is that overrides with template changes should have the ability to declare the version of Tutor with which the override is compatible. Prior to incrementing the version, the maintainer will need to ensure that the override is still compatible.
One way that recon can assist with this is to add an interactive merge feature that works by the individual template. Of course, simply merging Tutor’s template changes into your overrides, even if only through fast-forwarding, is not enough to guarantee that the plugin would still work. So active maintenance is required regardless.
In retrospect, overriding of settings and overriding of templates are two fairly distinct features though they can go hand-in-hand. The ability to define a way to combine multiple sets of overrides will be needed to allow these features to operate more independently. And the documentation should make the differences clear.
All in all, the hope is to make it straightforward to ‘resolve’ a stack of overrides. Once fully resolved, all overrides could be compacted into a “single source of truth” file which should be human-readable. Even templates could be included as strings in such a file. Meanwhile, the Tutor installation on a node can be left vanilla or can begin with traditional plugins (with the knowledge that any override in the final, resolved file takes highest precedence).
As for a move to using only Caddy, this looks like a great idea! Please let me know if I can assist with the contribution. This is definitely a circumstance where a contribution makes the most sense, and I look forward to the added simplicity 
.
a bit curious how you do know I have a different approach, as I haven’t talked about it
I’m not sure about the use case of this added setting. Are you proposing that Caddy should listen to a port other than 443 like 8443 for https? I feel like this is useless, let’s encrypt only verify the domain at 80 and 443, so in this case, user will need to provide a cert for their domain. Then why don’t they offload TLS at the reverse proxy in the front?
Additionally, I’m thinking about adding a variable HTTPS_MODE, which can be off, manual and auto. The problem with current ENABLE_HTTPS variable is that if I want to use my own cert behind a reverse proxy, I have to set ENABLE_HTTPS to false as Caddy cannot bind port 443, and that would lead to LMS_ROOT_URL to be http, which should start with https.
By adding the three modes, users get to choose if s/he want to have Caddy auto-provisioned https, or manage https themselves.
I’m not sure about this. Currently Nginx isn’t doing logging right? Could you please elaborate more on this? Highly appreciate this!
Totally understand and fully agree.
link is dead but i suppose it’s Tutor Enhancement Proposals (TEPs) - Overhang.IO?
Right, my comment was not very explicit. I referred to your proposal here: Why Caddy + Nginx? - #6 by iamCristYe
The introduction of a {{ patch(...) }} statement in Caddy will not by itself resolve the question of custom certificates. It is my impression that users who have custom certificates also need custom web proxies. Thus, in the solution that I suggest, Caddy would not take care at all of SSL termination when custom certificates are required. Caddy would then only listen on port 80, and users would be responsible of launching their own proxy (as they do today).
Again, my explanations were not quite clear, sorry about that. My point was not so much to enable users to customize the HTTPS port, but to disable SSL termination entirely, by setting CADDY_HTTPS_PORT=null. I guess most of that use case could be tackled with a more simple, clearer boolean setting, such as ENABLE_SSL_TERMINATION.
This is incorrect – and I realize that the docs are very misleading. Instead, to use your own certs, you must keep ENABLE_HTTPS=true and set RUN_CADDY=false. You should then run a custom web proxy, as described here.
I’ll open a PR to resolve the erroneous docs.
EDIT: here is the PR docs: clarify how to use custom ssl certificates by regisb · Pull Request #498 · overhangio/tutor · GitHub
Let me recap:
RUN_CADDY=false) and users would be responsible for running their own web proxy with SSL/TLS termination.RUN_CADDY=true) but would load its certificates from elsewhere?RUN_CADDY=true and ENABLE_HTTPS=true.Am I interpreting your proposal correctly? If yes, this is a little too complex for my taste. I’d rather avoid the scenario where Caddy must load custom certificates on the host. Instead, I think it’s preferable if users handle all the proxy config by themselves.
Currently, nginx logs all requests to stdout, which enables observability solutions (such as Cairn to automatically collect logs and later perform analysis.
yes – will edit my post.
True for the first sentence, but mounting the certs into the Caddy container would solve the problem. Nevertheless, I agree with you that if users are using custom certs, setting up a reverse proxy would be easier.
This makes sense. However, I think my approach of HTTPS_MODE is cleaner. see my explanation below.
You’re right. I find myself writing nonsense as well. I in fact set ENABLE_HTTPS to be true, so probably what I wanted to talk about is the misleading docs (setting true must use lets encrypt etc…). Thanks for fixing it.
Sorry for not making myself clear:
CADDY_HTTP_PORT only, probably 80. or it can be another port if the user want to setup some sort of proxy.CADDY_HTTP_PORT only. user will need to set up the proxy manually.I hope the above settings/wordings make sense. It’s kind of merging the current ENABLE_HTTPS and RUN_CADDY.
I think log (Caddyfile directive) — Caddy Documentation this one fits.
let me know your thoughts.
There are multiple issues with using HTTPS_MODE:
ENABLE_HTTPS, which is used extensively by all plugins.Nonetheless, I think that we both agree that Nginx could be dropped in favour of a single Caddy proxy. I’ll write a TEP to that end before the Maple release.
Obviously, we’ve got different ideas on how to implement this. Is it possible to include my approach in your TEP so maintainers can discuss it? or I can write the TEP
Sure, good idea. I’ll write a TEP and organize a meeting.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.