Elasticsearch issue in maple.2?

[sambapete]

I was trying to reindex a course when I stumbled into the following error

2022-02-26 18:56:49,977 ERROR 18 [edx.modulestore] [user 30] [ip 64.229.109.112] courseware_index.py:263 - Indexing error encountered, courseware index may be out of date course-v1:PolyMtl+CHE101.2+P2016 - RequestError(400, 'search_phase_execution_exception', {'error': {'root_cause': [{'type': 'query_shard_exception', 'reason': 'failed to create query: maxClauseCount is set to 1024', 'index_uuid': 'sJGwR4MNROe4nctpafJK-Q', 'index': 'courseware_content'}], 'type': 'search_phase_execution_exception', 'reason': 'all shards failed', 'phase': 'query', 'grouped': True, 'failed_shards': [{'shard': 0, 'index': 'courseware_content', 'node': 'dGLSEa9uRfWeSUO_ton8yw', 'reason': {'type': 'query_shard_exception', 'reason': 'failed to create query: maxClauseCount is set to 1024', 'index_uuid': 'sJGwR4MNROe4nctpafJK-Q', 'index': 'courseware_content', 'caused_by': {'type': 'too_many_clauses', 'reason': 'maxClauseCount is set to 1024'}}}]}, 'status': 400})
Traceback (most recent call last):
  File "/openedx/edx-platform/./cms/djangoapps/contentstore/courseware_index.py", line 260, in index
    cls.remove_deleted_items(searcher, structure_key, indexed_items)
  File "/openedx/edx-platform/./cms/djangoapps/contentstore/courseware_index.py", line 109, in remove_deleted_items
    response = searcher.search(
  File "/openedx/venv/lib/python3.8/site-packages/search/elastic.py", line 645, in search
    es_response = self._es.search(index=self.index_name, body=body, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/client/__init__.py", line 1670, in search
    return self.transport.perform_request(
  File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 415, in perform_request
    raise e
  File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/transport.py", line 381, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
    self._raise_error(response.status, raw_data)
  File "/openedx/venv/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 330, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to create query: maxClauseCount is set to 1024')
2022-02-26 18:56:49,990 ERROR 18 [django.request] [user None] [ip None] log.py:224 - Internal Server Error: /course/course-v1:PolyMtl+CHE101.2+P2016/search_reindex

According to what I found in elasticsearch - Not able to change "maxClauseCount" value in elastic search - Stack Overflow indices.query.bool.max_clause_count is a static setting so you should set it in elasticsearch.yml file in each and every node of elasticsearch cluster. You must restart the cluster.

Is this an issue with the elasticsearch configuration under Tutor?

curl --silent "http://127.0.0.1:9200/_cluster/settings?include_defaults=true&pretty=true"

does return a default value of "max_clause_count" : "1024" in the elasticsearch container.

How should I update it?

[sambapete]

I know I should include something in docker-compose-override.yml for example, but then how do I simply rebuild the elasticsearch container or image?

Do I have to go through a full rebuild of openedx through either “tutor local quickstart” or through “tutor images build openedx” ?

Inquiring mind wants to know what is the quickest way to apply my modifications for

• indices.query.bool.max_clause_count=4096

[sambapete]

It was simpler than I thought since there was an editor already available in the elasticsearch container.

I accessed it through
tutor local exec elasticsearch bash

I then edited the config/elasticsearch.yml file in order to add
indices.query.bool.max_clause_count: 4096

Save the file.

Exit the container.

And then restart the container with
tutor local restart elasticsearch

[regis]

Hey Pierre,
I seriously doubt that your solution is going to work. The elasticsearch.yml file in the container is not stored in a volume, as far as I understand. This means that your changes will not be persisted from one run to the next.

The “right” solution would be to add the following contents to the docker-compose.override.yml file:

elasticsearch:
    environment:
      - indices.query.bool.max_clause_count=4096

Then, to take your changes into account, you need to do a full stop/start cycle (yes, painful, I know, but that’s because docker-compose would ignore your changes otherwise) :

tutor local start && tutor local stop # alternatively, you can run 'tutor local reboot'

[sambapete]

I agree. That’s what I was afraid of and what I finally ended up with. Thanks @regis

Slowly getting more and more familiar with Tutor. One step at a time.