Can't enable 'discovery' after switching site domain then switching back - JWT mismatched user issue

I wanted to reconfigure my Tutor installation to a subdomain (courses.edunomic.academy) which worked fine simply by running tutor local quickstart again with the new domain.

However, I’ve decided to go back to the old domain (edunomic.academy, no subdomain) and am unable to do so with the discovery plugin enabled (it works fine when they’re not enabled). tutor local quickstart returns:

2021-12-03 17:37:13,883 INFO 141 [edx_rest_framework_extensions.auth.jwt.decoder] /openedx/venv/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/decoder.py:189 - Token decode failed due to mismatched issuer [https://courses.edunomic.academy/oauth2]
2021-12-03 17:37:13,883 INFO 141 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /openedx/venv/lib/python3.8/site-packages/backoff/_common.py:86 - Backing off run_loader(...) for 0.7s (jwt.exceptions.InvalidTokenError: https://courses.edunomic.academy/oauth2 is not a valid issuer.)
2021-12-03 17:37:14,615 INFO 141 [edx_rest_framework_extensions.auth.jwt.decoder] /openedx/venv/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/decoder.py:189 - Token decode failed due to mismatched issuer [**https://courses.edunomic.academy/oauth2**]
2021-12-03 17:37:14,616 INFO 141 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /openedx/venv/lib/python3.8/site-packages/backoff/_common.py:86 - Backing off run_loader(...) for 41.6s (jwt.exceptions.InvalidTokenError: **https://courses.edunomic.academy/oauth2** is not a valid issuer.)
2021-12-03 17:37:56,264 INFO 141 [edx_rest_framework_extensions.auth.jwt.decoder] /openedx/venv/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/decoder.py:189 - Token decode failed due to mismatched issuer [**https://courses.edunomic.academy/oauth2**]
2021-12-03 17:37:56,264 ERROR 141 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /openedx/venv/lib/python3.8/site-packages/backoff/_common.py:101 - Giving up run_loader(...) after 3 tries (jwt.exceptions.InvalidTokenError: **https://courses.edunomic.academy/oauth2** is not a valid issuer.)
2021-12-03 17:37:56,265 ERROR 141 [course_discovery.apps.course_metadata.management.commands.refresh_course_metadata] /openedx/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py:39 - CoursesApiDataLoader failed!
Traceback (most recent call last):
  File "/openedx/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py", line 36, in execute_loader
    run_loader()
  File "/openedx/venv/lib/python3.8/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/openedx/discovery/course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py", line 33, in run_loader
    return loader_class(*loader_args).ingest()
  File "/openedx/discovery/course_discovery/apps/course_metadata/data_loaders/__init__.py", line 32, in __init__
    self.username = self.get_username_from_client(self.api_client)
  File "/openedx/discovery/course_discovery/apps/course_metadata/data_loaders/__init__.py", line 43, in get_username_from_client
    decoded_jwt = configured_jwt_decode_handler(token)
  File "/openedx/venv/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/decoder.py", line 72, in configured_jwt_decode_handler
    return api_setting_jwt_decode_handler(token)
  File "/openedx/venv/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/decoder.py", line 63, in jwt_decode_handler
    decoded_token = _decode_and_verify_token(token, jwt_issuer)
  File "/openedx/venv/lib/python3.8/site-packages/edx_rest_framework_extensions/auth/jwt/decoder.py", line 190, in _decode_and_verify_token
    raise jwt.InvalidTokenError('%s is not a valid issuer.' % token_issuer)
jwt.exceptions.InvalidTokenError: **https://courses.edunomic.academy/oauth2** is not a valid issuer.
CommandError: One or more of the data loaders above failed.
Error: Command failed with status 1: docker-compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm discovery-job sh -e -c make migrate

# Development partners
./manage.py create_or_update_partner  \
  --site-id 1 \
  --site-domain discovery.edunomic.academy:8381 \
  --code dev --name "Open edX - development" \
  --lms-url="http://lms:8000" \
  --studio-url="http://cms:8000" \
  --courses-api-url "http://edunomic.academy:8000/api/courses/v1/" \
  --organizations-api-url "http://edunomic.academy:8000/api/organizations/v1/"

# Production partner
./manage.py create_or_update_partner  \
  --site-id 2 \
  --site-domain discovery.edunomic.academy \
  --code openedx --name "Open edX" \
  --lms-url="http://lms:8000" \
  --studio-url="http://cms:8000" \
  --courses-api-url "https://edunomic.academy/api/courses/v1/" \
  --organizations-api-url "https://edunomic.academy/api/organizations/v1/"

./manage.py refresh_course_metadata --partner_code=openedx
./manage.py update_index --disable-change-limit

I’ve gone a little frantic trying to resolve this, to the point of deleting the django_site records for the old domain, deleting all courses, all users created after the switch, and removing all discovery/ecommerce settings from config.yml, etc. Nothing seems to work, and discovery won’t forget the old config it seems.

Any ideas how I can clear out whatever it is discovery is complaining about here?

A couple other notes:

ubuntu@ip-172-31-38-207:~/.local/share/tutor$ tutor local run discovery ./manage.py shell -c "from django.conf import settings; print(settings.JWT_AUTH['JWT_ISSUER'])"
docker-compose -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.yml -f /home/ubuntu/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tut
Starting tutor_local_elasticsearch_1 ... done
Starting tutor_local_elasticsearch_1 ...
Starting tutor_local_mongodb_1       ... done
Starting tutor_local_mysql_1         ... done
Starting tutor_local_redis_1         ... done
Starting tutor_local_forum_1         ... done
Starting tutor_local_smtp_1          ... done
Starting tutor_local_lms_1           ... done
https://edunomic.academy/oauth2

The JWT_AUTH setting looks correct.

I found this thread which shows a possible solution on OpenEDX: Failed to execute refresh metadata course in discovery service - #2 by giovannicimolin - DevOps - Open edX discussions

But I have no idea how I’d adapt that solution to tutor. Any thoughts?

I ended up deleting all local docker containers, deleting all mysql data, and doing a quickstart from scratch. That finally allowed discovery to work.

It would be great to figure out where that prior domain was held in case someone runs into it again.

I am glad that you managed to resolve the situation because I have no idea what was actually happening :sweat_smile:

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.