regis
January 6, 2022, 9:36am
5
Hey Ned, good to see you here
You are describing a real issue that has been partially discussed here in different places:
The following a mix between an issue / feature request / documentation update request / support request. It depends on what’s intended.
The documentation to upgrade to a new version of Open edX recommends to use tutor local quickstart. If you have customized your docker images, it is asked to rebuild them before running quickstart.
It is not specified explicitely in the documentation about custom images , but you should probably run tutor config save before building the images. Otherwise env/bu…
Hello,
got Ubuntu 18.04.6 and Docker and Python 3
tried to install Maple by binary or pip, but got the same error:
Creating tutor_local_lms-job_run ... done
2021/12/24 11:19:23 Waiting for: tcp://mysql:3306
2021/12/24 11:19:23 Connected to tcp://mysql:3306
Loading settings lms.envs.tutor.production
Traceback (most recent call last):
File "./manage.py", line 119, in <module>
startup = importlib.import_module(edx_args.startup)
File "/opt/pyenv/versions/3.8.6/lib/python3.8/importlib/__in…
opened 04:38PM - 22 Dec 21 UTC
closed 06:07PM - 08 Jan 22 UTC
bug
**Bug description**
Upgrading a Kubernetes-managed installation from Tutor to M… aple breaks, because Kubernetes doesn't like the `nodePort` update to Services (the change that landed in 7c1e85ef4ba94cce1d597a1a3ea69cedbc2bde49).
**How to reproduce**
* Install Tutor 12.1.7, run `tutor k8s quickstart`
* Install Tutor 13.0.2, run `tutor k8s quickstart` again
* Observe that all Services break with:
```
Service "mysql" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
Service "redis" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP'
```
**Environment**
This is on Ubuntu Focal, against Kubernetes 1.18.20.
```
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.20", GitCommit:"1f3e19b7beb1cc0110255668c4238ed63dadb7ad", GitTreeState:"clean", BuildDate:"2021-06-16T12:51:17Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
```
**Additional context**
<https://docs.tutor.overhang.io/k8s.html> doesn't cover Open edX release upgrades specifically, but <https://docs.tutor.overhang.io/install.html#upgrading> mentions re-running `tutor local quickstart`, so I am assuming that for Kubernetes, the same approach should work (with `tutor k8s quickstart`).
I understand that I can get around this with running `tutor k8s stop` while still on version 12, followed by `tutor k8s quickstart` on version 13 (because the persistent volumes persist, no data is lost this way), and people who ran Lilac on Tutor may want to do that when running the release upgrade. But the problem with that is that d8d0560b9e647d36bf3d596a559d41a6e01c06f6 only landed in version 13, meaning that doing so removes the Caddy service (and its external IP), so when people do that they will have to update their DNS records for the new IP external IP address they will receive. Meaning, the service interruption they incur may be longer than they expect.
Is there a way to add a follow-up change to 7c1e85ef4ba94cce1d597a1a3ea69cedbc2bde49, so that the Service change behaves nicely on a major release upgrade?
In my understanding, there are two problems:
tutor local upgrade does not run tutor config save prior to printing that message about building Docker images. This specific problem is addressed in this pending PR: fix: upgrade from Lilac on k8s by regisb · Pull Request #554 · overhangio/tutor · GitHub The upgrade message that is printed in the standard output is poorly phrased. I can attempt to improve that message in the same PR.
Also, and maybe this wasn’t clear either: the way to upgrade a local installation is to run tutor local quickstart. The upgrade command will be run as part of quickstart. The only cases when you should run tutor local upgrade is when:
The upgrade process failed for some reason and you want to restart it.
You accidentally overwrote your environment with tutor config save and tutor has no way to know that it should upgrade.
Maybe this wasn’t clear from the docs or the tutor upgrade command help?
