I installed Tutor on a quite solid server and I’m receiving quite high CPU loads on a simple quiz (part of the demo course).
I tested the same with the bitnami edx and pretty much the same…
Any idea if this is normal?
Hi @mslyon! How much RAM and CPU (cat /proc/cpuinfo) does your server have? You can take a look at which containers are using up the most resources with docker stats Troubleshooting — Tutor documentation
Hi regis, actually the server should be quite good - it is a 6 core Intel Xeon E5-1650V3, with 128 GB of memory.
The CPU spike happens only when a quiz is loaded - I tried with a simple quiz and it goes to 40% - with a random quiz it goes even to 70%…
I tested on 2 different servers.
Do you see the same spike when you run a quiz?
No I don’t… Again, you should check whether one of the containers use a lot of CPU with docker stats.
hmm very strange then…
here is what I did - I installed tutor on a brand new server - 1 single user.
I imported demo course.
Just clicked on edX exams (6 questions)
and spike is immediate…
I also used another server and same behaviour…
any idea where this could come from? P.S I tested this on 3 different servers and it always happens ONLY when quiz is started…
and of course before running the quiz all is pretty normal:
What are the logs from the spiking container? What’s the content of your quizz?
Hi regis,
actually this behavior happens on Native, Bitnami and tutor - bitnami support was also able to reproduce it (see here: High CPU OpenEdx on some tasks - OpenEdX - Bitnami Community).
It seems to me that this is a bug that affects entire edx code.
To reproduce it is very simple: you just run any quiz and you should the CPU going to 20% immediately…obviously this is not an issue with 1 user but when 60 start the quiz at the same time, the server dies…
Again, bitnami tech people were able to reproduce this in the native installation and they found the same…they said that I shoudl probably open a ticket with open edx directly.
thanks.
@mslyon I managed to reproduce the issue. I can confirm that loading a quiz causes a spike in CPU usage. Please bear with me because I have another question: why is that an issue? You mention that:
What do you mean by “the server dies”? In my understanding it’s perfectly fine if CPU usage goes to 100/200/400% for some time, as long as requests are correctly processed and we don’t exceed the available memory – in the latter case processes will be killed, which will definitely be an issue. But loading a quiz only affects CPU usage, and not memory, doesn’t it?
yes you are correct. Memory is fine. But what happens is this: 60 users loaded the quiz (at the same time) / memory remained stable…CPU was over 100%…it took to openedx cca 5 minutes to respond with the quiz to all users…so it became unresponsive for 5-6 minutes…
I replicated this on native, tutor, bitnami…
This is even more prononouced when quiz is loaded from libraries and when quiz pulls some random questions from the quiz library…this become even worse then…
I did some quick profiling but was unable to find an obvious culprit. I suggest that you report this performance issue in the Open edX forums: https://discuss.openedx.org/ Remember to link to this conversation as well as the Bitnami ticket. You should ping Dave Ormsbee who is passionate about all things performance.
In your case, the obvious (but not great) solution is to throw money at the problem and to scale your server, either horizontally (more CPU) or vertically (more servers, for instance with Kubernetes). But I hope we can get to a point where we the LMS can serve 60-100 users concurrently.
thanks - I will create the ticket then.
BTW I just reported it here:
3 Likes
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.