Juniper Upgrade - LMS not rendering certain content due to handling of StudentModule.DoesNotExist exception

Tutor
1

Hi All,
I was able to upgraded from ironwood to Juniper. Barring a few hiccups, the upgrade was successful. However some of the content/courseware is not being rendered by the LMS. Please note that the same content is available on CMS/Studio.

The exception is in handling of StudentModule.DoesNotExist exception

Below is the error log:

lms_1             | 2020-06-18 10:50:14,182 ERROR 9 [edx.courseware] [user 5] views.py:699 - Error in /courses/course-v1:edX+DemoX+Demo_Course/courseware/interactive_demonstrations/19a30717eff543078a5d94ae9d6c18a5/2: user=shashi.kiran, effective_user=shashi.kiran, course=course-v1:edX+DemoX+Demo_Course
lms_1             | Traceback (most recent call last):
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/query.py", line 538, in get_or_create
lms_1             |     return self.get(**kwargs), False
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/query.py", line 408, in get
lms_1             |     self.model._meta.object_name
lms_1             | lms.djangoapps.courseware.models.StudentModule.DoesNotExist: StudentModule matching query does not exist.
lms_1             | 
lms_1             | During handling of the above exception, another exception occurred:
lms_1             | 
lms_1             | Traceback (most recent call last):
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/utils.py", line 166, in ensure_defaults
lms_1             |     conn = self.databases[alias]
lms_1             | KeyError: 'student_module_history'
lms_1             | 
lms_1             | During handling of the above exception, another exception occurred:
lms_1             | 
lms_1             | Traceback (most recent call last):
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/transaction.py", line 115, in mark_for_rollback_on_error
lms_1             |     yield
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/base.py", line 779, in save_base
lms_1             |     force_update, using, update_fields,
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/base.py", line 870, in _save_table
lms_1             |     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/base.py", line 908, in _do_insert
lms_1             |     using=using, raw=raw)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 82, in manager_method
lms_1             |     return getattr(self.get_queryset(), name)(*args, **kwargs)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/query.py", line 1186, in _insert
lms_1             |     return query.get_compiler(using=using).execute_sql(return_id)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/sql/query.py", line 288, in get_compiler
lms_1             |     connection = connections[using]
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/utils.py", line 198, in __getitem__
lms_1             |     self.ensure_defaults(alias)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/utils.py", line 168, in ensure_defaults
lms_1             |     raise ConnectionDoesNotExist("The connection %s doesn't exist" % alias)
lms_1             | django.db.utils.ConnectionDoesNotExist: The connection student_module_history doesn't exist
lms_1             | 
lms_1             | During handling of the above exception, another exception occurred:
lms_1             | 
lms_1             | Traceback (most recent call last):
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/utils.py", line 166, in ensure_defaults
lms_1             |     conn = self.databases[alias]
lms_1             | KeyError: 'student_module_history'
lms_1             | 
lms_1             | During handling of the above exception, another exception occurred:
lms_1             | 
lms_1             | Traceback (most recent call last):
lms_1             |   File "/openedx/edx-platform/lms/djangoapps/courseware/views/index.py", line 165, in get
lms_1             |     return self.render(request)
lms_1             |   File "/openedx/edx-platform/lms/djangoapps/courseware/views/index.py", line 267, in render
lms_1             |     return render_to_response('courseware/courseware.html', self._create_courseware_context(request))
lms_1             |   File "/openedx/edx-platform/lms/djangoapps/courseware/views/index.py", line 502, in _create_courseware_context
lms_1             |     courseware_context['fragment'] = self.section.render(self.view, section_context)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/core.py", line 202, in render
lms_1             |     return self.runtime.render(self, view, context)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1979, in render
lms_1             |     return self.__getattr__('render')(block, view_name, context)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1423, in render
lms_1             |     return super(MetricsMixin, self).render(block, view_name, context=context)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/runtime.py", line 829, in render
lms_1             |     frag = view_fn(context)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 293, in student_view
lms_1             |     return self._student_or_public_view(context, prereq_met, prereq_meta_info, banner_text)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 401, in _student_or_public_view
lms_1             |     params = self._get_render_metadata(context, display_items, prereq_met, prereq_meta_info, banner_text, view, fragment)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 371, in _get_render_metadata
lms_1             |     items = self._render_student_view_for_items(context, display_items, fragment, view) if prereq_met else []
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/seq_module.py", line 534, in _render_student_view_for_items
lms_1             |     rendered_item = item.render(view, context)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/core.py", line 202, in render
lms_1             |     return self.runtime.render(self, view, context)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1979, in render
lms_1             |     return self.__getattr__('render')(block, view_name, context)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1423, in render
lms_1             |     return super(MetricsMixin, self).render(block, view_name, context=context)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/runtime.py", line 829, in render
lms_1             |     frag = view_fn(context)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/vertical_block.py", line 118, in student_view
lms_1             |     return self._student_or_public_view(context, STUDENT_VIEW)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/vertical_block.py", line 84, in _student_or_public_view
lms_1             |     rendered_child = child.render(view, child_block_context)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/core.py", line 202, in render
lms_1             |     return self.runtime.render(self, view, context)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1979, in render
lms_1             |     return self.__getattr__('render')(block, view_name, context)
lms_1             |   File "/openedx/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1423, in render
lms_1             |     return super(MetricsMixin, self).render(block, view_name, context=context)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/runtime.py", line 832, in render
lms_1             |     block.save()
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/mixins.py", line 255, in save
lms_1             |     self.force_save_fields(fields_to_save)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/mixins.py", line 270, in force_save_fields
lms_1             |     self._field_data.set_many(self, fields_to_save_json)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/field_data.py", line 170, in set_many
lms_1             |     field_data.set_many(block, new_update_dict)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/field_data.py", line 170, in set_many
lms_1             |     field_data.set_many(block, new_update_dict)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/xblock/runtime.py", line 228, in set_many
lms_1             |     self._kvs.set_many(updated_dict)
lms_1             |   File "/openedx/edx-platform/lms/djangoapps/courseware/model_data.py", line 137, in set_many
lms_1             |     self._field_data_cache.set_many(kv_dict)
lms_1             |   File "/openedx/edx-platform/lms/djangoapps/courseware/model_data.py", line 858, in set_many
lms_1             |     self.cache[scope].set_many(set_many_data)
lms_1             |   File "/openedx/edx-platform/lms/djangoapps/courseware/model_data.py", line 420, in set_many
lms_1             |     pending_updates
lms_1             |   File "/openedx/edx-platform/lms/djangoapps/courseware/user_state_client.py", line 254, in set_many
lms_1             |     'module_type': usage_key.block_type,
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/manager.py", line 82, in manager_method
lms_1             |     return getattr(self.get_queryset(), name)(*args, **kwargs)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/query.py", line 541, in get_or_create
lms_1             |     return self._create_object_from_params(kwargs, params)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/query.py", line 575, in _create_object_from_params
lms_1             |     obj = self.create(**params)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/query.py", line 422, in create
lms_1             |     obj.save(force_insert=True, using=self.db)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/base.py", line 741, in save
lms_1             |     force_update=force_update, update_fields=update_fields)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/base.py", line 790, in save_base
lms_1             |     update_fields=update_fields, raw=raw, using=using,
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/dispatch/dispatcher.py", line 175, in send
lms_1             |     for receiver in self._live_receivers(sender)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/dispatch/dispatcher.py", line 175, in <listcomp>
lms_1             |     for receiver in self._live_receivers(sender)
lms_1             |   File "/openedx/edx-platform/lms/djangoapps/coursewarehistoryextended/models.py", line 59, in save_history
lms_1             |     history_entry.save()
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/base.py", line 741, in save
lms_1             |     force_update=force_update, update_fields=update_fields)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/models/base.py", line 779, in save_base
lms_1             |     force_update, using, update_fields,
lms_1             |   File "/opt/pyenv/versions/3.5.9/lib/python3.5/contextlib.py", line 77, in __exit__
lms_1             |     self.gen.throw(type, value, traceback)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/transaction.py", line 117, in mark_for_rollback_on_error
lms_1             |     connection = get_connection(using)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/transaction.py", line 20, in get_connection
lms_1             |     return connections[using]
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/utils.py", line 198, in __getitem__
lms_1             |     self.ensure_defaults(alias)
lms_1             |   File "/openedx/venv/lib/python3.5/site-packages/django/db/utils.py", line 168, in ensure_defaults
lms_1             |     raise ConnectionDoesNotExist("The connection %s doesn't exist" % alias)
lms_1             | django.db.utils.ConnectionDoesNotExist: The connection student_module_history doesn't exist
lms_1             | 2020-06-18 10:50:14,480 ERROR 9 [django.request] [user None] log.py:228 - Internal Server Error: /courses/course-v1:edX+DemoX+Demo_Course/courseware/interactive_demonstrations/19a30717eff543078a5d94ae9d6c18a5/2
2

Indeed. This bug is supposed to be fixed in v10.0.2. What version are you running?

(more generally, when you write a new topic asking for help please remember to post the version of Tutor that you are using along with the output of tutor plugins list)

1 Like
3

Thanks @regis for the response. The tutor version I am running is v10.0.1. I shall follow your suggestion about posting version and plugins list output from the next post onwards.

Below is the output of tutor plugins list. Please note that plugins ecommerce and discovery were enabled but were giving some issues on running tutor local quickstart. I disabled them for now. Will enable in 10.0.2 and check

discovery==10.0.0 (disabled)
ecommerce==10.0.0 (disabled)
lts==9.1.6 (disabled)
minio==0.1.5 (disabled)
notes==10.0.0 (disabled)
xqueue==10.0.0 (disabled)

4

@regis The Earlier error is now fixed after upgrading to v10.0.2.

I do get the following error when discovery and ecommerce plugins were enabled while running tutor local quickstart

Starting tutor_local_elasticsearch_1 ... done
Starting tutor_local_smtp_1          ... done
Starting tutor_local_mysql_1         ... done
Starting tutor_local_memcached_1     ... done
Starting tutor_local_forum_1         ... done
Starting tutor_local_lms_1           ... done
python manage.py migrate --noinput
Operations to perform:
  Apply all migrations: admin, auth, catalogs, contenttypes, core, course_metadata, django_comments, guardian, ietf_language_tags, journal, publisher, publisher_comments, sessions, sites, social_django, taggit, waffle
Running migrations:
  Applying core.0013_historicalpartner...Traceback (most recent call last):
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 239, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.ProgrammingError: (1146, "Table 'discovery.core_historicalpartner' doesn't exist")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/openedx/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.5/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.5/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/openedx/venv/lib/python3.5/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/openedx/venv/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 234, in handle
    fake_initial=fake_initial,
  File "/openedx/venv/lib/python3.5/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
    schema_editor.create_model(model)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 307, in create_model
    self.execute(sql, params or None)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 137, in execute
    cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File "/openedx/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/openedx/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/openedx/venv/lib/python3.5/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.5/site-packages/MySQLdb/connections.py", line 239, in query
    _mysql.connection.query(self, query)
django.db.utils.ProgrammingError: (1146, "Table 'discovery.core_historicalpartner' doesn't exist")
Makefile:74: recipe for target 'migrate' failed
make: *** [migrate] Error 1
Error: Command failed with status 2: docker-compose -f /home/raifox/.local/share/tutor/env/local/docker-compose.yml -f /home/raifox/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/raifox/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm discovery-job sh -e -c make migrate

# Development partners
./manage.py create_or_update_partner  \
  --site-id 1 \
  --site-domain discovery.local.overhang.io:8381 \
  --code dev --name "Open edX - development" \
  --lms-url="http://lms:8000" \
  --courses-api-url "http://local.overhang.io:8000/api/courses/v1/"
 
# Production partner
./manage.py create_or_update_partner  \
  --site-id 2 \
  --site-domain discovery.local.overhang.io \
  --code openedx --name "Open edX" \
  --lms-url="http://lms:8000" \
  --courses-api-url "http://local.overhang.io/api/courses/v1/"

./manage.py refresh_course_metadata --partner_code=openedx
./manage.py update_index --disable-change-limit
5

What is the output of tutor local run discovery ./manage.py showmigrations core?

6

I shall revert in a short while with the output response.

7

@regis Below is the output of tutor local run discovery ./manage.py showmigrations core ?

tutor local run discovery ./manage.py showmigrations core
docker-compose -f /home/raifox/.local/share/tutor/env/local/docker-compose.yml -f /home/raifox/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm discovery ./manage.py showmigrations core
Starting tutor_local_elasticsearch_1 … done
Starting tutor_local_smtp_1 … done
Starting tutor_local_mongodb_1 … done
Starting tutor_local_rabbitmq_1 … done
Starting tutor_local_memcached_1 … done
Starting tutor_local_mysql_1 … done
Starting tutor_local_forum_1 … done
Starting tutor_local_lms_1 … done
core
0001_squashed_0011_auto_20161101_2207 (11 squashed migrations)
0002_partner_studio_url
0003_auto_20170522_0856
0004_partner_site
0005_auto_20170830_1246
0006_partner_lms_url
0007_auto_20171004_1133
0008_auto_20181217_1957
0009_partner_lms_commerce_api_url
0010_partner_lms_coursemode_api_url
0011_remove_partner_lms_commerce_api_url
0012_partner_publisher_url
0013_historicalpartner
0014_historicalsalesforceconfiguration_salesforceconfiguration
0015_add_lms_admin_url
0016_add_case_record_type_id
0017_drop_oidc_fields
0018_auto_20200414_0739

8

This is really strange. Migrations crash on step 0013, complaining that the historicalpartner table does not exist, but that migration 0013 is precisely the one that is supposed to create that table. Let’s try the following commands. If one of those does not work, try the others:

tutor local init --limit=discovery
tutor local run discovery ./manage.py migrate core
tutor local run discovery ./manage.py migrate core 0013
9

Thanks for the response. I shall try these commands and revert.

10

I tried all the 3 commands. Still getting the same issue.

11

Are you creating a new platform or upgrading an existing one? I suspect that migrations crashed halfway, leaving the MySQL database in an unknown state.

12

I am upgrading an existing one. All the upgrades were successful with the exception of this one. I shall also check on a staging server and see if the same problem persists[during an upgrade].

13

Out of despair, can you please try this:

tutor local run discovery ./manage.py migrate core 0012
tutor local run discovery ./manage.py migrate core 0013
14

Running those commands. Will revert shortly.

15

Below is the response for the above instruction

$ tutor local run discovery ./manage.py migrate core 0012
docker-compose -f /home/raifox/.local/share/tutor/env/local/docker-compose.yml -f /home/raifox/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm discovery ./manage.py migrate core 0012
Starting tutor_local_elasticsearch_1 ... done
Starting tutor_local_smtp_1          ... done
Starting tutor_local_mysql_1         ... done
Starting tutor_local_rabbitmq_1      ... done
Starting tutor_local_mongodb_1       ... done
Starting tutor_local_memcached_1     ... done
Starting tutor_local_forum_1         ... done
Starting tutor_local_lms_1           ... done
Operations to perform:
  Target specific migration: 0012_partner_publisher_url, from core
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
16

Unfortunately I don’t know how to debug this further without connecting to the server. You should try to place break points (import pdb; pdb.set_trace()) in /openedx/venv/lib/python3.5/site-packages/django/db/backends/mysql/base.py (line 71) to understand what query is failing when running ./manage.py migrate core 0013.

17

Thanks for the feedback and help all along this thread. Will debug and if i find a solution , I shall revert to this thread with the same.