Search courseware_content index missing from elasticsearch

I upgraded from koa to lilac a few days ago. Everything seemed to have succeeded. However, I received a complaint from a student indicating that the search was not working. They were receiving the following error when searching in the course:

An error occurred when searching for “list”

Looking at the LMS log file, I found the following entry:

lms_1            | 2022-02-25 13:33:05,663 WARNING 6 [elasticsearch] [user 3] [ip XXX.XXX.XXX.XXX] base.py:285 - HEAD http://localhost:9200/courseware_content [status:N/A request:0.003s]
lms_1            | Traceback (most recent call last):
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/urllib3/connection.py", line 169, in _new_conn
lms_1            |     conn = connection.create_connection(
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
lms_1            |     raise err
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
lms_1            |     sock.connect(sa)
lms_1            | ConnectionRefusedError: [Errno 111] Connection refused

I verified that the elasticsearch container was running. I then checked the indices on the container:

curl -X GET "localhost:9200/_cat/indices"
yellow open comments_20220218152154501        QZKU09KfQ9aE4M9ApV_Jhw 1 1  0 0   208b   208b
yellow open comment_threads_20220218152158611 8On20Zx2Rjy7hxGPFIGOsA 1 1 25 0 18.1kb 18.1kb
yellow open comments_20220218152158611        1KQU7VJjS-C0y1S9u0tWyA 1 1 20 0 13.1kb 13.1kb
yellow open comment_threads_20220218152154501 15jcaVpzTFCmBpqsSGWXqA 1 1  0 0   208b   208b

And noticed that there is no courseware_content index.

I can confirm that ENABLE_COURSEWARE_INDEX is enabled:

Python 3.8.6 (default, Feb 18 2022, 15:34:04) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.conf import settings 
>>> settings.FEATURES.get('ENABLE_COURSEWARE_INDEX')
True

Does anyone have any idea how I can build this missing index? Or am I missing something else?

What exactly is the action that they are performing? Can you share screenshots/urls?

What are the logs from the elasticsearch container right after the error was triggered?

You mention that you are running Lilac. Does this error also occur on the latest Maple release?

The students are using the search box on the course page.

There is no error in the elasticsearch container. The error is only thrown in the LMS.

lms_1            | Traceback (most recent call last):
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/search/views.py", line 99, in do_search
lms_1            |     results = perform_search(
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/search/api.py", line 60, in perform_search
lms_1            |     searcher = SearchEngine.get_search_engine(
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/search/search_engine_base.py", line 62, in get_search_engine
lms_1            |     return search_engine_class(index=index) if search_engine_class else None
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/search/elastic.py", line 332, in __init__
lms_1            |     if not self._es.indices.exists(index=self.index_name):
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 153, in _wrapped
lms_1            |     return func(*args, params=params, headers=headers, **kwargs)
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/indices.py", line 332, in exists
lms_1            |     return self.transport.perform_request(
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 413, in perform_request
lms_1            |     raise e
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 381, in perform_request
lms_1            |     status, headers_response, data = connection.perform_request(
lms_1            |   File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 264, in perform_request
lms_1            |     raise ConnectionError("N/A", str(e), e)
lms_1            | elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f92360586d0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f92360586d0>: Failed to establish a new connection: [Errno 111] Connection refused)

I am not at a point where I can upgrade to Maple just yet. I’m hoping to be able to start testing the upgrade over the next couple of weeks. This will be one of the first items that I test. I’ll post here if it is still an issue.

I did find this post over on the openedx.org discussion that seems to have the same issue:

I have not had time to dig into it further.