OpenShift 4.6 (k8s 1.19) Install

Hi all,

Earlier this year I explored using Tutor to install OpenEdx onto a test OpenShift cluster. I was able to get everything running after asking our cluster admin to allow containers to run as root. Encouraging but not immediately helpful as my organization would not allow this config in a real cluster.

I recently checked back and noticed updates around k8s and root containers. I decided to give it another try and report my results. My cluster is v4.6, which is k8s v1.19.

Upon re-initializing my config, I ran tutor k8s start. The only pod that came up initially was caddy. Caddy’s log, however, ended with an error, which I haven’t yet looked into:

run: loading initial config: loading new config: http app module: start:
 tcp: listening on :80: listen tcp :80: bind: permission denied

All of the replica sets that had failed to bring up a pod had a similar error:

Error creating: pods "cms-857476898f-" is forbidden: unable to validate 
against any security context constraint: 
[spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must
 be in the ranges: [1001340000, 1001349999]]

I looked through the deployment yaml and noticed all the securityContext settings. I don’t have deep expertise around security context constraints but from what I can tell, Red Hat images (pre-packaged or those built on the cluster) do not alter securityContext. OpenShift manages the UID and it just works (the acceptable range is random and per-namespace so there is no way to specify a UID that will work for everyone).

I went through the yaml and removed these settings and upon re initialization, Minio and mongodb were fine. Exim starts up but all I see in the log is

 exim: permission denied

The pods for the OpenEdx variants made it further but the application logs were reporting trouble with read-only directories. E.g.,

PermissionError: [Errno 13] Permission denied: '/openedx/data/logs'

I added emtpyDir mounts for the following to see if I could get further:

CMS variant:

  • /openedx/media
  • /openedx/data/ora2

LMS variant:

  • /openedx/data/logs
  • /openedx/media
  • /openedx/data/ora2

After making these changes, the LMS/CMS pods along with their worker pods all came up and, based on their logs, appeared to be ready to go.

That’s all I have for now. Look forward to feedback from the community and to helping with efforts to get Tutor/OpenEdx working smoothly on OpenShift.

Michael

Aaaaargh, that’s really bad news! Let me summarize the two issues here:

This means that we cannot bind a load balancer to port 80? (and I suspect 443 as well?) How are we supposed to deploy load balancers then?

This second error is really annoying. It means that OpenShift assigns arbitrary user IDs to the running containers. I do not understand how to get this to work with our containers and with the exim container. For instance, how are we supposed to write to volumes? Containers also need write-access to their own filesystem, how can we achieve that?

This means that we cannot bind a load balancer to port 80? (and I suspect 443 as well?) How are we supposed to deploy load balancers then?

I don’t think that’s the case. After some reading and examination of my config and resource declarations, I noticed that I had turned off proxy so I didn’t have the LoadBalancer declaration for caddy. I regenerated my config from scratch and tried to deploy.

➭ oc get svc/caddy
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
caddy   LoadBalancer   172.30.179.19   <external IP>   80:30161/TCP,443:31422/TCP   17h

This appears to be correct. I don’t think there’s any issue using this approach with OpenShift.

Caddy however still fails with a similar message. I don’t know yet why it’s complaining about https now vs http before.

run: loading initial config: loading new config: http app module: start: tcp: listening on :443: listen tcp :443: bind: permission denied

If the container isn’t running with root privileges, Caddy won’t be able to bind to 80/443. I think that’s what is going on here.

LoadBalancer can specify a targetPort as well. Could we run Caddy on 8080 and 8443 and handle the forwarding in the load balancer?

Regarding the random uid, it’s discussed a bit here.

Volumes work perfectly fine out of the box. Here’s what I see in one of our other Django apps. I have not done anything in terms of permissions or mode in the volumeMounts section of my Deployment

(app-root) sh-4.4$ whoami
1001220000
(app-root) sh-4.4$ ls -lah /opt/app-root/src/<mysite>/site_media/
total 8.0K
drwxrwxr-x.  1 default root         19 Nov 10 11:27 .
drwxrwxr-x.  1 default root       4.0K Nov  8 12:07 ..
drwxrwxrwx.  2 root    1001220000    0 Oct 21 14:36 media
drwxrwxr-x. 20 default root       4.0K Nov  8 12:07 static

This is a nice writeup on some of the considerations to maintain compatibility between k8s and OpenShift. (In the case above the volume mount is actually owned by root.<uid> and not root.root. Maybe this was a typo in the write up but all volume mounts in a project look like this in terms of ownership. Certainly makes sense this way.)

Without knowing a little more about what you’ve tested and feel like you need for k8s I am hesitant to make changes but I’d like to help here. Let me know what I can do.