Memory requirements & mysql

Using both microk8s (k8s 1.16.0) and minikube on Ubuntu 18.04 to test drive I’m having to allocate >~ 4.7G of RAM in virtualbox (5.2.32) to get all the pods to launch - the LMS pod (not the worker) stays in pending. The docs say the min is 4096M, has that changed from the docs or is there something special about my setup?

I’m also getting failures with the waiting for mysql to be ready. The kubectl wait... is timing out (even if mysql is up) and I’m getting errors in both the CMS and LMS pods like the following:

... django.db.utils.OperationalError: (1045, "Access denied for user 'openedx'@'172.17.0.7' (using password: YES)")

Any pointers? Thanks.

Hi @antonovich! We did not try to evaluate the precise requirements for a k8s install. The memory requirements indicated in the docs are for a local install.

Did you try to run tutor k8s quickstart again?

Hi @regis. Yes, I tried running it again. And again. At one point it appeared that I got the tutor k8s quickstart to work but because I hadn’t assigned enough memory it never started. After giving it enough memory I wasn’t able to get that to finish again. I tried everything I could think of over several hours but couldn’t get things to work.

I’m also getting errors running the local install:

...
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

Error: Command failed with status 1: docker-compose -f /home/toto/.local/share/tutor/env/local/docker-compose.yml --project-name tutor_local exec lms sh -e -c dockerize -wait tcp://mysql:3306 -timeout 20s
./manage.py lms --settings=tutor.production migrate

./manage.py lms --settings=tutor.production create_oauth2_client \
    "http://androidapp.com" "http://androidapp.com/redirect" public \
    --client_id android --client_secret hPifcxPfrtVIh1nSrmlB7d9E \
    --trusted
    
# Fix incorrect uploaded file path
if [ -d /openedx/data/uploads/ ]; then
  if [ -n "$(ls -A /openedx/data/uploads/)" ]; then
    echo "Migrating LMS uploaded files to shared directory"
    mv /openedx/data/uploads/* /openedx/media/
    rm -rf /openedx/data/uploads/
  fi
fi

Is there any thinking about doing a helm chart at some point or would that defeat the purpose a bit? Thanks.

It appears that there was some cruft left in the config after attempting to do a tutor k8s quickstart. I guess that is the price to pay for “quickinstallers”. Removing the .local/share/tutor/ directory fixed things for a local install.

Hi @antonovich! I’m glad you managed to solve your issue. I believe the problem you faced may be related to activating/de-activating the MinIO plugin multiple times. See this github issue for reference: https://github.com/overhangio/tutor/issues/241

Hi @regis. I know k8s support is only beta but has anyone tested it lately? Given Kubernetes’ total dominance of container orchestration and Docker Inc’s likely impending demise, it would seem to be the way forward. Even Mesosphere looks like it is turning into a Kubernetes support company these days! I’m just evaluating OpenEDX at the moment but if I do decide to use it in my project I will almost certainly make a helm chart, which seems to have significant mindshare for packaging up complex systems like OpenEDX (and is declarative awesomeness, obviating issues like https://github.com/overhangio/tutor/issues/241).

I know k8s support is only beta but has anyone tested it lately?

Funny that you ask! I realised just yesterday that the “beta” label was still present in the tutor k8s --help output, and the next release will not have this label anymore. (see this commit )
To be clear: Kubernetes deployment is fully functional and supported. Some major players are deploying many platforms every day with tutor k8s quickstart.
That being said, as specified in the docs, deploying to Kubernetes requires extensive devops knowledge. As a consequence, we (the overhang.io company) are not going to provide extensive, free community support for k8s deployment. In order to ask for our help on k8s, you will have to buy into a yearly support subscription.

Does that make sense?

Thanks @regis. I was a Senior DevOps engineer at Criteo (rue Blanche) until a year ago, supporting their Mesos platform among other things, so I have some experience in the field. I understand you guys need to eat though. I read a post on the OpenEDX forums that suggests they are going to start looking at k8s themselves, so I’ll probably try and hack together a helm chart if I go ahead with OpenEDX and see whether they are interested.

Thanks for your help though, I managed to get things working via tutor so got to test - the free IBM trial on their cloud took three days to spin up and it is still impossible to log in two days after that!

How cool! I’m an ex-Criteo as well :slight_smile: (from rue Chapon and Place de l’Opéra)

I had a big conundrum when I first started work on the k8s integration: should I use Helm or not? In the end, I decided not to go with Helm for the following reasons (in decreasing order of importance):

  1. At the time, kubectl seemed sufficiently mature to do most of the work performed by Helm.
  2. Helm does a lot of templating that conflicts with tutor’s own templating system. We would have been forced to have both systems coexist, and that would have been quite impractical.
  3. Plain kubernetes manifests have a bigger mindshare/user base than Helm charts, simply because they do not require helm to be installed.
  4. I wanted to have a simple solution that could be extended by other users, if necessary. Simple Kubernetes manifest files filled this requirement.
  5. Writing Helm templates is an even bigger pain in the ass than writing Kubernetes manifests.

Finally, (and I’m not quite sure about this one) it seems like it’s not so easy to extend Helm charts using a plugin approach similar to the tutor plugin system.

That being said, I would be more than happy if someone (you?) developed a Helm chart for Open edX. Everybody wins when we improve the tooling around Open edX.

My understanding (but I admit I may very well be wrong) is that Helm charts are great for installing apps that do not have much modularity. With Open edX, there are so many moving pieces that’s it’s difficult to write a Helm chart that will work for everyone.

Can I ask what’s your use case for running Open edX? Feel free to PM me if you’d rather not discuss this publicly (in English or French).

I read a post on the OpenEDX forums that suggests they are going to start looking at k8s themselves

I am curious to see where this is going to go. That would mean that edX would start running containerized applications, and this would be quite a revolution. It’s no small feat to migrate the entire infrastructure of a 200-engineer team.

@regis, I can’t PM here yet unfortunately. I’m evaluating edX for use in my PhD thesis, which I’m doing to move forward an open source project I started - https://transcrob.es. It’s obviously overkill for what I want to do strictly in the thesis but it has a lot of functionality that will be useful down the road and it’s Django-based, which makes it a great fit for Transcrob.es (which is also Django). I would argue that helm is actually great for complex, modular apps (see Transcrob.es), and once you are over the initial hurdle of seeing how the templates fit together, it is quite easy to navigate. We can take this offline if you prefer - anton at transcrob.es!