Re-indexing courses command

when i executed command below I got this error:

tutor local run discovery ./manage.py refresh_course_metadata --partner_code=dev

docker-compose -f /root/.local/share/tutor/env/local/docker-compose.yml -f /root/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm discovery ./manage.py refresh_course_metadata --partner_code=dev
Starting tutor_local_memcached_1 ... done
Starting tutor_local_smtp_1      ... done
Starting tutor_local_mysql_1         ... done
Starting tutor_local_elasticsearch_1 ... done
Starting tutor_local_mongodb_1       ... done
Starting tutor_local_forum_1         ... done
Starting tutor_local_lms_1           ... done
2020-06-30 10:13:27,167 INFO 1 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /openedx/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:116 - Command is not using threads to write data.
2020-06-30 10:13:27,171 INFO 1 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /openedx/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:153 - Executing Loader [http://lms.xxx.xxx:8000/api/courses/v1/]
2020-06-30 10:13:27,182 INFO 1 [course_discovery.apps.course_metadata.data_loaders.api] /openedx/discovery/course_discovery/apps/course_metadata/data_loaders/api.py:34 - Refreshing Courses and CourseRuns from http://lms.xxx.xxx:8000/api/courses/v1/...
2020-06-30 10:13:27,183 INFO 1 [course_discovery.apps.course_metadata.data_loaders.api] /openedx/discovery/course_discovery/apps/course_metadata/data_loaders/api.py:84 - Requesting course run page 1...
2020-06-30 10:13:27,189 ERROR 1 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /openedx/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:28 - CoursesApiDataLoader failed!
Traceback (most recent call last):
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1134, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1179, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1130, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 946, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 889, in send
    self.connect()
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f674ae6cda0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/openedx/venv/lib/python3.5/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='lms.ifham.online', port=8000): Max retries exceeded with url: /api/courses/v1/courses/?username=discovery&page_size=50&page=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f674ae6cda0>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.5/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/openedx/discovery/course_discovery/apps/course_metadata/data_loaders/api.py", line 86, in _make_request
    response = self.api_client.get(self.api_url + '/courses/', params=params)
  File "/openedx/venv/lib/python3.5/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/edx_rest_api_client/client.py", line 283, in request
    return super(OAuthAPIClient, self).request(method, url, **kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='lms.ifham.online', port=8000): Max retries exceeded with url: /api/courses/v1/courses/?username=discovery&page_size=50&page=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f674ae6cda0>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openedx/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py", line 25, in execute_loader
    loader_class(*loader_args).ingest()
  File "/openedx/discovery/course_discovery/apps/course_metadata/data_loaders/api.py", line 37, in ingest
    response = self._make_request(initial_page)
  File "/openedx/venv/lib/python3.5/site-packages/backoff/_sync.py", line 100, in retry
    if giveup(e) or max_tries_exceeded or max_time_exceeded:
  File "/openedx/discovery/course_discovery/apps/course_metadata/data_loaders/api.py", line 68, in _fatal_code
    return ex.response.status_code != 429 and ex.response.status_code != 504  # pylint: disable=no-member
AttributeError: 'NoneType' object has no attribute 'status_code'
CommandError: One or more of the data loaders above failed.
Error: Command failed with status 1: docker-compose -f /root/.local/share/tutor/env/local/docker-compose.yml -f /root/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm discovery ./manage.py refresh_course_metadata --partner_code=dev

What version of Tutor and plugins are you running? In the future, please always include in your question the output of:

tutor --version
tutor plugins list

tutor, version 10.0.8
discovery==10.0.1
ecommerce==10.0.2 (disabled)
lts==9.1.6 (disabled)
minio==10.0.0 (disabled)
notes==10.0.0 (disabled)
xqueue==10.0.0 (disabled)

What are the logs from the lms and nginx right after triggering this error? You can obtain these logs by running tutor local logs --tail=100 lms nginx.

Attaching to tutor_local_nginx_1, tutor_local_lms_1
lms_1               | 102.123.158.136 - - [03/Jul/2020:07:59:55 +0000] "POST /courses/course-v1:edX+DemoX+Demo_Course/instructor/api/list_instructor_tasks HTTP/1.0" 200 17 "https://xx.xx.xx/courses/course-v1:edX+DemoX+Demo_Course/instructor" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36"
lms_1               | 2020-07-03 08:00:15,881 INFO 27 [tracking] [user 3] logger.py:49 - {"ip": "102.123.158.136", "time": "2020-07-03T08:00:15.881577+00:00", "event_type": "/courses/course-v1:edX+DemoX+Demo_Course/instructor/api/list_instructor_tasks", "accept_language": "en,en-US;q=0.9,ar;q=0.8", "event_source": "server", "context": {"path": "/courses/course-v1:edX+DemoX+Demo_Course/instructor/api/list_instructor_tasks", "course_id": "course-v1:edX+DemoX+Demo_Course", "org_id": "edX", "user_id": 3, "course_user_tags": {}}, "referer": "https://xx.xx.xx/courses/course-v1:edX+DemoX+Demo_Course/instructor", "page": null, "username": "admin", "event": "{\"GET\": {}, \"POST\": {}}", "agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36", "host": "xx.xx.xx"}
lms_1               | 102.123.158.136 - - [03/Jul/2020:08:00:15 +0000] "POST courses/course-v1:edX+DemoX+Demo_Course/instructor/api/list_instructor_tasks HTTP/1.0" 200 17 "https://xx.xx.xx/courses/course-v1:edX+DemoX+Demo_Course/instructor" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36"
lms_1               | 2020-07-03 08:00:35,203 INFO 27 [tracking] [user 3] logger.py:49 - {"ip": "102.123.158.136", "time": "2020-07-03T08:00:35.202834+00:00", "event_type": "/courses/course-v1:edX+DemoX+Demo_Course/instructor/api/list_instructor_tasks", "accept_language": "en,en-US;q=0.9,ar;q=0.8", "event_source": "server", "context": {"path": "/courses/course-v1:edX+DemoX+Demo_Course/instructor/api/list_instructor_tasks", "course_id": "course-v1:edX+DemoX+Demo_Course", "org_id": "edX", "user_id": 3, "course_user_tags": {}}, "referer": "https://xx.xx.xx/courses/course-v1:edX+DemoX+Demo_Course/instructor", "page": null, "username": "admin", "event": "{\"GET\": {}, \"POST\": {}}", "agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36", "host": "lms.ifham.online"}
nginx_1             | 95.179.138.196 - - [03/Jul/2020:16:12:43 +0000] "GET http://www.zghzpscpt.cn:81/ HTTP/1.1" 301 186 "-" "undefined" "-"
nginx_1             | 95.179.138.196 - - [03/Jul/2020:16:12:43 +0000] "GET http://www.zghzpscpt.cn:81/ HTTP/1.1" 301 186 "-" "undefined" "-"
nginx_1             | 95.179.138.196 - - [03/Jul/2020:16:12:43 +0000] "GET http://www.zghzpscpt.cn:81/ HTTP/1.1" 301 186 "-" "undefined" "-"

I think I understand. This seems to be caused by the fact that you are loading development settings, and not production settings. Try instead to run:

./manage.py refresh_course_metadata --partner_code=openedx

I’ll change the plugin README.

thank you for reply
there is another error when run:
tutor local run lms ./manage.py lms cache_programs

2020-07-05 16:49:08,009 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:49 - populate-multitenant-programs switch is ON
2020-07-05 16:49:08,055 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:71 - Skipping site example.com. No configuration.
2020-07-05 16:49:08,092 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:142 - Requesting program UUIDs for xx.xx.xx.
2020-07-05 16:49:08,149 ERROR 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:145 - Failed to retrieve program UUIDs for site: xx.xx.xx.
Traceback (most recent call last):
  File "/openedx/edx-platform/openedx/core/djangoapps/catalog/management/commands/cache_programs.py", line 143, in get_site_program_uuids
    uuids = client.programs.get(**querystring)
  File "/openedx/venv/lib/python3.5/site-packages/slumber/__init__.py", line 155, in get
    resp = self._request("GET", params=kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/slumber/__init__.py", line 101, in _request
    raise exception_class("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
slumber.exceptions.HttpNotFoundError: Client Error 404: http://discovery:8000/programs/
2020-07-05 16:49:08,153 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:150 - Received 0 UUIDs for site xx.xx.xx
2020-07-05 16:49:08,153 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:177 - Requesting pathways for xx.xx.xx.
2020-07-05 16:49:08,279 ERROR 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:187 - Failed to retrieve pathways for site: xx.xx.xx.
Traceback (most recent call last):
  File "/openedx/edx-platform/openedx/core/djangoapps/catalog/management/commands/cache_programs.py", line 181, in get_pathways
    new_pathways = client.pathways.get(exclude_utm=1, page=next_page)
  File "/openedx/venv/lib/python3.5/site-packages/slumber/__init__.py", line 155, in get
    resp = self._request("GET", params=kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/slumber/__init__.py", line 101, in _request
    raise exception_class("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
slumber.exceptions.HttpNotFoundError: Client Error 404: http://discovery:8000/pathways/
2020-07-05 16:49:08,297 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:193 - Received 0 pathways for site xx.xx.xx
2020-07-05 16:49:08,300 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:100 - Caching UUIDs for 0 programs for site xx.xx.xx.
2020-07-05 16:49:08,304 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:107 - Caching ids for 0 pathways for site xx.xx.xx.
2020-07-05 16:49:08,323 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:71 - Skipping site studio.ifham.online. No configuration.
2020-07-05 16:49:08,333 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:111 - Caching details for 0 programs.
2020-07-05 16:49:08,334 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:114 - Caching details for 0 pathways.
2020-07-05 16:49:08,334 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:117 - Caching programs uuids for 0 courses.
2020-07-05 16:49:08,334 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:120 - Caching programs uuids for 0 catalog courses.
2020-07-05 16:49:08,335 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:123 - Caching program UUIDs by 0 program types.
2020-07-05 16:49:08,335 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:126 - Caching programs uuids for 0 organizations
Error: Command failed with status 1: docker-compose -f /root/.local/share/tutor/env/local/docker-compose.yml -f /root/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm lms ./manage.py lms cache_programs

i think i faced those errors because i did not find any documentation to create program. can you guide me to all steps required to create program??
thank you again for your reply.

I think there is an error in the docs to index the programs. You should create a catalog integration with the right url:

tutor local run lms ./manage.py lms create_catalog_integrations --enabled \
    --internal_api_url=http://discovery:8000/api/v1 \
    --service_username=lms_catalog_service_user

And then attempt to run tutor local run lms ./manage.py lms cache_programs. In my case I also had to delete all Site objects :sweat_smile: tutor local run ./manage.py lms shell -c "from django.contrib.sites.models import Site;Site.objects.all().delete()" RUN THIS AT YOUR OWN PERIL

I’ll update the docs of the discovery plugin (again).

same error again :face_with_head_bandage:

Unfortunately at this point I would need to log into the server to debug this further…

what dose this error message mean:

Traceback (most recent call last):
  File "/openedx/edx-platform/openedx/core/djangoapps/catalog/management/commands/cache_programs.py", line 143, in get_site_program_uuids
    uuids = client.programs.get(**querystring)
  File "/openedx/venv/lib/python3.5/site-packages/slumber/__init__.py", line 155, in get
    resp = self._request("GET", params=kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/slumber/__init__.py", line 101, in _request
    raise exception_class("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
slumber.exceptions.HttpNotFoundError: Client Error 404: http://discovery:8000/programs/
2020-07-06 16:06:32,257 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:150 - Received 0 UUIDs for site xx.xx.xx
2020-07-06 16:06:32,258 INFO 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:177 - Requesting pathways for xx.xx.xx.
2020-07-06 16:06:32,301 ERROR 1 [openedx.core.djangoapps.catalog.management.commands.cache_programs] [user None] cache_programs.py:187 - Failed to retrieve pathways for site: xx.xx.xx.
Traceback (most recent call last):
  File "/openedx/edx-platform/openedx/core/djangoapps/catalog/management/commands/cache_programs.py", line 181, in get_pathways
    new_pathways = client.pathways.get(exclude_utm=1, page=next_page)
  File "/openedx/venv/lib/python3.5/site-packages/slumber/__init__.py", line 155, in get
    resp = self._request("GET", params=kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/slumber/__init__.py", line 101, in _request
    raise exception_class("Client Error %s: %s" % (resp.status_code, url), response=resp, content=resp.content)
**slumber.exceptions.HttpNotFoundError: Client Error 404: http://discovery:8000/pathways/**

i think this error related to value that found in site value in Site configurations,when i remove this value ({“COURSE_CATALOG_API_URL”:“http://discovery:8000”}) no error released and vice versa.

1 Like

yes this is correct but I want to know, uses of this URL COURSE_CATALOG_API_URL
please help me how to use and how it’s work.