LMS worker Timeout issue

Tutor
1

hi
This is the same issue like “issue no. 145” but for me the reason is not the RAM. My System RAM is also 8GB but still getting this issue. Help me solve it.
I’m trying to start tutor with the following command:

Response Message:
    lms_1               | [2020-11-11 06:31:47 +0000] [69] [INFO] Booting worker with pid: 69
    lms_1               | [2020-11-11 06:31:47 +0000] [71] [INFO] Booting worker with pid: 71
    mongodb_1           | 2020-11-11T06:31:57.925+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46512 #532 (3 connections now open)
    mongodb_1           | 2020-11-11T06:31:57.930+0000 I NETWORK  [conn532] received client metadata from 172.18.0.9:46512 conn532: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:31:57.954+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46514 #533 (4 connections now open)
    mongodb_1           | 2020-11-11T06:31:57.962+0000 I NETWORK  [conn533] received client metadata from 172.18.0.9:46514 conn533: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:31:58.984+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46516 #534 (5 connections now open)
    mongodb_1           | 2020-11-11T06:31:58.985+0000 I NETWORK  [conn534] received client metadata from 172.18.0.9:46516 conn534: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:31:59.267+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46518 #535 (6 connections now open)
    mongodb_1           | 2020-11-11T06:31:59.268+0000 I NETWORK  [conn535] received client metadata from 172.18.0.9:46518 conn535: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:31:59.272+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46520 #536 (7 connections now open)
    mongodb_1           | 2020-11-11T06:31:59.273+0000 I NETWORK  [conn536] received client metadata from 172.18.0.9:46520 conn536: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:31:59.512+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46522 #537 (8 connections now open)
    mongodb_1           | 2020-11-11T06:31:59.517+0000 I NETWORK  [conn537] received client metadata from 172.18.0.9:46522 conn537: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    lms_1               | [2020-11-11 06:32:17 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:69)
    lms_1               | [2020-11-11 06:32:17 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:71)
    lms_1               | [2020-11-11 06:32:17 +0000] [71] [INFO] Worker exiting (pid: 71)
    lms_1               | [2020-11-11 06:32:17 +0000] [69] [INFO] Worker exiting (pid: 69)
    mongodb_1           | 2020-11-11T06:32:18.227+0000 I NETWORK  [conn535] end connection 172.18.0.9:46518 (7 connections now open)
    mongodb_1           | 2020-11-11T06:32:18.227+0000 I NETWORK  [conn534] end connection 172.18.0.9:46516 (6 connections now open)
    mongodb_1           | 2020-11-11T06:32:18.227+0000 I NETWORK  [conn536] end connection 172.18.0.9:46520 (5 connections now open)
    mongodb_1           | 2020-11-11T06:32:18.228+0000 I NETWORK  [conn532] end connection 172.18.0.9:46512 (4 connections now open)
    mongodb_1           | 2020-11-11T06:32:18.228+0000 I NETWORK  [conn537] end connection 172.18.0.9:46522 (3 connections now open)
    mongodb_1           | 2020-11-11T06:32:18.228+0000 I NETWORK  [conn533] end connection 172.18.0.9:46514 (2 connections now open)
    lms_1               | [2020-11-11 06:32:18 +0000] [99] [INFO] Booting worker with pid: 99
    lms_1               | [2020-11-11 06:32:18 +0000] [101] [INFO] Booting worker with pid: 101
    nginx_1             | 2020/11/11 06:32:21 [error] 6#6: *51 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 89.36.219.233, server: panel.fanoos.academy, request: "GET / HTTP/1.1", upstream: "http://172.18.0.9:8000/", host: "panel.fanoos.academy"
    nginx_1             | 89.36.219.233 - - [11/Nov/2020:06:32:21 +0000] https://panel.fanoos.academy "GET / HTTP/1.1" 504 578 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36" "-"
    mongodb_1           | 2020-11-11T06:32:29.572+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46530 #538 (3 connections now open)
    mongodb_1           | 2020-11-11T06:32:29.577+0000 I NETWORK  [conn538] received client metadata from 172.18.0.9:46530 conn538: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:32:30.155+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46532 #539 (4 connections now open)
    mongodb_1           | 2020-11-11T06:32:30.162+0000 I NETWORK  [conn539] received client metadata from 172.18.0.9:46532 conn539: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:32:30.811+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46534 #540 (5 connections now open)
    mongodb_1           | 2020-11-11T06:32:30.811+0000 I NETWORK  [conn540] received client metadata from 172.18.0.9:46534 conn540: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:32:31.051+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46536 #541 (6 connections now open)
    mongodb_1           | 2020-11-11T06:32:31.053+0000 I NETWORK  [conn541] received client metadata from 172.18.0.9:46536 conn541: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:32:31.437+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46538 #542 (7 connections now open)
    mongodb_1           | 2020-11-11T06:32:31.440+0000 I NETWORK  [conn542] received client metadata from 172.18.0.9:46538 conn542: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    mongodb_1           | 2020-11-11T06:32:31.648+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:46540 #543 (8 connections now open)
    mongodb_1           | 2020-11-11T06:32:31.653+0000 I NETWORK  [conn543] received client metadata from 172.18.0.9:46540 conn543: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
    lms_1               | [2020-11-11 06:32:48 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:99)
    lms_1               | [2020-11-11 06:32:48 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:101)
    lms_1               | [2020-11-11 06:32:48 +0000] [99] [INFO] Worker exiting (pid: 99)
    lms_1               | [2020-11-11 06:32:48 +0000] [101] [INFO] Worker exiting (pid: 101)
    mongodb_1           | 2020-11-11T06:32:49.451+0000 I NETWORK  [conn541] end connection 172.18.0.9:46536 (7 connections now open)
    mongodb_1           | 2020-11-11T06:32:49.451+0000 I NETWORK  [conn540] end connection 172.18.0.9:46534 (6 connections now open)
    mongodb_1           | 2020-11-11T06:32:49.451+0000 I NETWORK  [conn538] end connection 172.18.0.9:46530 (5 connections now open)
    mongodb_1           | 2020-11-11T06:32:49.454+0000 I NETWORK  [conn543] end connection 172.18.0.9:46540 (4 connections now open)
    mongodb_1           | 2020-11-11T06:32:49.454+0000 I NETWORK  [conn542] end connection 172.18.0.9:46538 (3 connections now open)
    mongodb_1           | 2020-11-11T06:32:49.454+0000 I NETWORK  [conn539] end connection 172.18.0.9:46532 (2 connections now open)
    lms_1               | [2020-11-11 06:32:49 +0000] [129] [INFO] Booting worker with pid: 129
    lms_1               | [2020-11-11 06:32:49 +0000] [131] [INFO] Booting worker with pid: 131

ping ip machine docker:

    ping 172.18.0.9
    PING 172.18.0.9 (172.18.0.9) 56(84) bytes of data.
    64 bytes from 172.18.0.9: icmp_seq=1 ttl=64 time=0.231 ms
    64 bytes from 172.18.0.9: icmp_seq=2 ttl=64 time=0.056 ms

Help me solve it
@regis
Server Information:
Ram: 8GB
Cpu: 6 core

2

@regis
I updated to the latest version 10.4.1 but there is still a problem, it is possible to help to solve the problem
when run cmd :

tutor local quickstart -I

output:

python manage.py install_es_indexes
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/openedx/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py", line 25, in execute_loader
    loader_class(*loader_args).ingest()
  File "/openedx/discovery/course_discovery/apps/course_metadata/data_loaders/__init__.py", line 31, in __init__
    self.username = self.get_username_from_client(self.api_client)
  File "/openedx/discovery/course_discovery/apps/course_metadata/data_loaders/__init__.py", line 41, in get_username_from_client
    token = client.get_jwt_access_token()
  File "/openedx/venv/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 270, in get_jwt_access_token
    self._ensure_authentication()
  File "/openedx/venv/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 254, in _ensure_authentication
    timeout=self._timeout,
  File "/openedx/venv/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 170, in get_and_cache_oauth_access_token
    timeout=timeout,
  File "/openedx/venv/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 112, in get_oauth_access_token
    timeout=timeout
  File "/openedx/venv/lib/python3.5/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='lms', port=8000): Read timed out. (read timeout=5)
CommandError: One or more of the data loaders above failed.
Error: Command failed with status 1: docker-compose -f /root/.local/share/tutor/env/local/docker-compose.yml -f /root/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /root/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm discovery-job sh -e -c make migrate
3

Hi @chavoshipour,
Did you activate tutor-discovery plugin ?
if it is enabled, i suggest disable it first start your platform, after that you can enable Discovery again ?

4

disabled tutor-discovery plugin end ecommerce but not resolve problem !!!
when tutor local quickstart -I
output:

bundle exec rake search:initialize
bundle exec rake search:rebuild_index
Creating tutor_local_forum-job_run ... done
Waiting for mongodb/elasticsearch...
2020/11/12 05:43:44 Waiting for: tcp://mongodb:27017
2020/11/12 05:43:44 Waiting for: http://elasticsearch:9200
2020/11/12 05:43:44 Connected to tcp://mongodb:27017
2020/11/12 05:43:44 Received 200 from http://elasticsearch:9200
/openedx/cs_comments_service/lib/tasks/deep_search.rake:7: warning: already initialized constant ROOT
/openedx/cs_comments_service/lib/tasks/flags.rake:6: warning: previous definition of ROOT was here
/openedx/cs_comments_service/lib/tasks/kpis.rake:7: warning: already initialized constant ROOT
/openedx/cs_comments_service/lib/tasks/deep_search.rake:7: warning: previous definition of ROOT was here
/openedx/cs_comments_service/vendor/bundle/ruby/2.5.0/gems/elasticsearch-transport-1.1.2/lib/elasticsearch/transport/transport/base.rb:52: warning: constant ::Fixnum is deprecated
/openedx/cs_comments_service/vendor/bundle/ruby/2.5.0/gems/elasticsearch-transport-1.1.2/lib/elasticsearch/transport/transport/base.rb:54: warning: constant ::Fixnum is deprecated
/openedx/cs_comments_service/models/constants.rb:2: warning: already initialized constant COURSE_ID
/openedx/cs_comments_service/lib/tasks/db.rake:28: warning: previous definition of COURSE_ID was here
W, [2020-11-12T05:43:57.083923 #17]  WARN -- : Overwriting existing field _id in class User.
W, [2020-11-12T05:43:57.184111 #17]  WARN -- : MONGODB | Unsupported client option 'max_retries'. It will be ignored.
W, [2020-11-12T05:43:57.184336 #17]  WARN -- : MONGODB | Unsupported client option 'retry_interval'. It will be ignored.
W, [2020-11-12T05:43:57.184418 #17]  WARN -- : MONGODB | Unsupported client option 'timeout'. It will be ignored.
I, [2020-11-12T05:43:57.252912 #17]  INFO -- : Skipping initialization. The 'content' alias already exists. If 'rake search:validate_index' indicates a problem with the mappings, you could either use 'rake s                             earch:rebuild_index' to reload from the db or 'rake search:initialize[true]' to force initialization with an empty index.
/openedx/cs_comments_service/lib/tasks/deep_search.rake:7: warning: already initialized constant ROOT
/openedx/cs_comments_service/lib/tasks/flags.rake:6: warning: previous definition of ROOT was here
/openedx/cs_comments_service/lib/tasks/kpis.rake:7: warning: already initialized constant ROOT
/openedx/cs_comments_service/lib/tasks/deep_search.rake:7: warning: previous definition of ROOT was here
/openedx/cs_comments_service/vendor/bundle/ruby/2.5.0/gems/elasticsearch-transport-1.1.2/lib/elasticsearch/transport/transport/base.rb:52: warning: constant ::Fixnum is deprecated
/openedx/cs_comments_service/vendor/bundle/ruby/2.5.0/gems/elasticsearch-transport-1.1.2/lib/elasticsearch/transport/transport/base.rb:54: warning: constant ::Fixnum is deprecated
/openedx/cs_comments_service/models/constants.rb:2: warning: already initialized constant COURSE_ID
/openedx/cs_comments_service/lib/tasks/db.rake:28: warning: previous definition of COURSE_ID was here
W, [2020-11-12T05:44:05.392945 #25]  WARN -- : Overwriting existing field _id in class User.
W, [2020-11-12T05:44:05.508757 #25]  WARN -- : MONGODB | Unsupported client option 'max_retries'. It will be ignored.
W, [2020-11-12T05:44:05.508980 #25]  WARN -- : MONGODB | Unsupported client option 'retry_interval'. It will be ignored.
W, [2020-11-12T05:44:05.509089 #25]  WARN -- : MONGODB | Unsupported client option 'timeout'. It will be ignored.
I, [2020-11-12T05:44:06.205116 #25]  INFO -- : Added mappings to index: content_20201112054405552.
I, [2020-11-12T05:44:06.205342 #25]  INFO -- : Created new index: content_20201112054405552.
I, [2020-11-12T05:44:06.287627 #25]  INFO -- : Catch up from 2020-11-12 05:39:05 UTC complete.
I, [2020-11-12T05:44:06.367523 #25]  INFO -- : Alias [content] now points to index [content_20201112054405552].
I, [2020-11-12T05:44:06.389515 #25]  INFO -- : Catch up from 2020-11-12 05:39:06 UTC complete.
I, [2020-11-12T05:44:06.389685 #25]  INFO -- : Rebuild index complete.
All services initialised.
The Open edX platform is now running in detached mode
Your Open edX platform is ready and can be accessed at the following urls:

and output last log after running service:

mongodb_1        | 2020-11-12T05:45:44.197+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:50052 #63 (6 connections now open)
mongodb_1        | 2020-11-12T05:45:44.199+0000 I NETWORK  [conn63] received client metadata from 172.18.0.9:50052 conn63: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux",                              architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
mongodb_1        | 2020-11-12T05:45:44.261+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:50054 #64 (7 connections now open)
mongodb_1        | 2020-11-12T05:45:44.268+0000 I NETWORK  [conn64] received client metadata from 172.18.0.9:50054 conn64: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux",                              architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
mongodb_1        | 2020-11-12T05:45:44.448+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:50058 #65 (8 connections now open)
mongodb_1        | 2020-11-12T05:45:44.450+0000 I NETWORK  [conn65] received client metadata from 172.18.0.9:50058 conn65: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux",                              architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
lms_1            | [2020-11-12 05:46:01 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:269)
lms_1            | [2020-11-12 05:46:01 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:271)
lms_1            | [2020-11-12 05:46:01 +0000] [269] [INFO] Worker exiting (pid: 269)
lms_1            | [2020-11-12 05:46:01 +0000] [271] [INFO] Worker exiting (pid: 271)
mongodb_1        | 2020-11-12T05:46:02.405+0000 I NETWORK  [conn60] end connection 172.18.0.9:50046 (7 connections now open)
mongodb_1        | 2020-11-12T05:46:02.406+0000 I NETWORK  [conn64] end connection 172.18.0.9:50054 (3 connections now open)
mongodb_1        | 2020-11-12T05:46:02.406+0000 I NETWORK  [conn61] end connection 172.18.0.9:50048 (2 connections now open)
mongodb_1        | 2020-11-12T05:46:02.405+0000 I NETWORK  [conn62] end connection 172.18.0.9:50050 (6 connections now open)
mongodb_1        | 2020-11-12T05:46:02.406+0000 I NETWORK  [conn65] end connection 172.18.0.9:50058 (5 connections now open)
mongodb_1        | 2020-11-12T05:46:02.406+0000 I NETWORK  [conn63] end connection 172.18.0.9:50052 (4 connections now open)
lms_1            | [2020-11-12 05:46:02 +0000] [299] [INFO] Booting worker with pid: 299
lms_1            | [2020-11-12 05:46:02 +0000] [301] [INFO] Booting worker with pid: 301
mongodb_1        | 2020-11-12T05:46:14.453+0000 I NETWORK  [listener] connection accepted from 172.18.0.9:50064 #66 (3 connections now open)
mongodb_1        | 2020-11-12T05:46:14.454+0000 I NETWORK  [conn66] received client metadata from 172.18.0.9:50064 conn66: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux",                              architecture: "x86_64", version: "4.4.0-184-generic" }, platform: "CPython 3.5.9.final.0" }
5

@qali
@regis

i’m find resolve problem time out worker
add argument timeout the Dockerfile and fix problem

6

It would be nice if we could configure this timeout in the LMS (or increase it a bit for everyone). I’m running into the timeout regularly on a 4 core / 8GB ram VPS when I run tutor local start -d.

FYI I hotfixed this now by adding the following to the cms: and lms: sections in the docker-compose file:

command: gunicorn -c /openedx/gunicorn_conf.py --name lms --bind=0.0.0.0:8000 --max-requests=1000 --max-requests-jitter=100 --access-logfile - lms.wsgi:application -t 120

and

command: gunicorn -c /openedx/gunicorn_conf.py --name cms --bind=0.0.0.0:8000 --max-requests=1000 --max-requests-jitter=100 --access-logfile - cms.wsgi:application -t 120
7

@chavoshipour @MaartenGH I do not understand what is the problem you are talking about.

@chavoshipour I think your problem is that you are getting [CRITICAL] WORKER TIMEOUT messages, right? Why is that an issue? Or is it that you are getting requests.exceptions.ReadTimeout while running discovery? These two problems look very different to me.

8

The problem is this:

$ tutor local logs lms -f
docker-compose -f /home/tutor/.local/share/tutor/env/local/docker-compose.yml -f /home/tutor/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local logs --follow lms
/usr/lib/python3/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.10) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Attaching to tutor_local_lms_1
lms_1            | [2020-12-04 15:33:47 +0000] [7] [INFO] Starting gunicorn 20.0.4
lms_1            | [2020-12-04 15:33:47 +0000] [7] [INFO] Listening at: http://0.0.0.0:8000 (7)
lms_1            | [2020-12-04 15:33:47 +0000] [7] [INFO] Using worker: sync
lms_1            | [2020-12-04 15:33:47 +0000] [10] [INFO] Booting worker with pid: 10
lms_1            | [2020-12-04 15:33:47 +0000] [12] [INFO] Booting worker with pid: 12
lms_1            | [2020-12-04 15:34:17 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:10)
lms_1            | [2020-12-04 15:34:17 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:12)
lms_1            | [2020-12-04 15:34:17 +0000] [12] [INFO] Worker exiting (pid: 12)
lms_1            | [2020-12-04 15:34:17 +0000] [10] [INFO] Worker exiting (pid: 10)
lms_1            | [2020-12-04 15:34:18 +0000] [24] [INFO] Booting worker with pid: 24
lms_1            | [2020-12-04 15:34:18 +0000] [25] [INFO] Booting worker with pid: 25
lms_1            | [2020-12-04 15:34:48 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:24)
lms_1            | [2020-12-04 15:34:48 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:25)
lms_1            | [2020-12-04 15:34:48 +0000] [25] [INFO] Worker exiting (pid: 25)
lms_1            | [2020-12-04 15:34:48 +0000] [24] [INFO] Worker exiting (pid: 24)
lms_1            | [2020-12-04 15:34:49 +0000] [50] [INFO] Booting worker with pid: 50
lms_1            | [2020-12-04 15:34:50 +0000] [51] [INFO] Booting worker with pid: 51
lms_1            | [2020-12-04 15:35:20 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:50)
lms_1            | [2020-12-04 15:35:20 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:51)
lms_1            | [2020-12-04 15:35:20 +0000] [50] [INFO] Worker exiting (pid: 50)
lms_1            | [2020-12-04 15:35:20 +0000] [51] [INFO] Worker exiting (pid: 51)
lms_1            | [2020-12-04 15:35:21 +0000] [76] [INFO] Booting worker with pid: 76
lms_1            | [2020-12-04 15:35:21 +0000] [78] [INFO] Booting worker with pid: 78

The gunicorn workers take a bit too long to start and then they hit a timeout. They get restarted, take a bit too long to start again, etc. etc. etc.

The solution (in my case) is to increase the timeout a bit. I have set it to 120 by adding -t 120 to the gunicorn command from the openedx Dockerfile. Then the gunicorn workers start without problems.

9

Ok, I did not understand from your initial post that the gunicorn workers kept restarting. Would you like to open a PR? If yes, please use the long-version --timeout argument.

FYI this will no longer be an issue in Koa, as Gunicorn and Whitenoise will be replaced by uwsgi.

10

Cool, yes I’ll to so as soon as I have time (somewhere next week, probably). Is it OK to hardcode the --timeout 120, or would you need it to be variable? At first sight I don’t see a problem with hardcoding it.

11

Yes, hard coding the timeout is fine. Thanks!

12

Here you go:

13

increase the server resources - CPU and Memory and the timeout issues of LMS will go away.