ElasticSearch hogging CPU?

Hi,

After a period of time with production tutor edx running, the system suddenly slows to an unusable speed.

Running docker stats shows elasticsearch1 using over 150% CPU at times! No clue why this is happening, especially with no reported use of the search feature.

Any particular ideas as to why this might be happening? Something sets off elasticsearch into a frenzy but no way to trigger it other than waiting >1 week so far.

Thanks

Also worth mentioning the problem does eventually resolve itself and then return again later - seems to be unpredictable.

Only relevant thing in logs I could find is the following, but not entirely sure what it’s trying to do. Granted the system is on 4GB of ram but there are very few users (~5) and only a few small courses.

How can I increase vm.max_map_count - not sure what unit ‘65530’ is but it wants me to increase the allocated memory approx 4x by the looks of it?

I have run

sysctl -w vm.max_map_count=262144

per Virtual memory | Elasticsearch Guide [8.1] | Elastic and I will update if the issue returns. Let me know if you have any other suggestions or ideas.

Thanks

Let us know if the problem occurs again. If it does, please paste the logs from the elasticsearch container:

tutor local logs --tail=100

(please don’t post screenshots. Pasted logs make it easier both for us, to investigate the issue, and for other users who face the same problem to find a solution.)

Hi,

Many thanks for your response. The problem has since occurred again, seems to be related to when saving/publishing courses - I suspect it is something going on when reindexing the search as it occurred after I published a small change to a course.

The system has 4GB of memory but only one course actually in use and only approximately 5 users.

Logs attached below:

elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:24,940Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-sql]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:24,940Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-stack]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:24,940Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:24,941Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:24,941Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "no plugins loaded" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:25,027Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [62.6gb], net total_space [77.3gb], types [ext4]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:25,028Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:25,212Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "node name [a5e2f9b5879a], node ID [XJnMmpINR5K-Vbc_fVyzrQ], cluster name [openedx], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:30,226Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "[controller/198] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:30,935Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:32,618Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:32,768Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:33,758Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:34,383Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "initialized" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:34,384Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "starting ..." }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:34,533Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:37,946Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "cluster UUID [qiIPWc0MSfywo2Ad4IVd6w]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:38,693Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "elected-as-master ([1] nodes joined)[{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{kYNmqresTDSQlJR7xmHodw}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 319, version: 1189, delta: master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{kYNmqresTDSQlJR7xmHodw}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:39,508Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{kYNmqresTDSQlJR7xmHodw}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 319, version: 1189, reason: Publication{term=319, version=1189}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:40,944Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:40,964Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "started", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:44,716Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "license [cad6af7c-6866-432c-8806-31d76cd60b38] mode [basic] - valid", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:44,733Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:06:44,790Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "recovered [2] indices into cluster_state", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:00,511Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[courseware_content][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:38,174Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "version[7.10.1], pid[7], build[default/docker/1c34507e66d7db1211f66f3513706fdf548736aa/2020-12-05T01:00:33.671820Z], OS[Linux/5.4.0-100-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:38,181Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:38,183Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-14137560325327189926, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,334Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [aggs-matrix-stats]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,334Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [analysis-common]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,335Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [constant-keyword]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,335Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [flattened]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,335Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [frozen-indices]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,336Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-common]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,336Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-geoip]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,337Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-user-agent]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,340Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [kibana]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,341Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-expression]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,341Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-mustache]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,342Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-painless]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,342Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [mapper-extras]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,342Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [mapper-version]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,342Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [parent-join]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,343Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [percolator]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,343Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [rank-eval]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,344Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [reindex]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,344Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [repositories-metering-api]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,344Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [repository-url]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,344Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [search-business-rules]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,345Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [searchable-snapshots]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,345Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [spatial]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,346Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [transform]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,347Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [transport-netty4]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,347Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [unsigned-long]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,347Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [vectors]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,347Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [wildcard]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,348Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-analytics]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,348Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-async]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,348Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-async-search]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,349Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-autoscaling]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,349Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ccr]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,349Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-core]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,349Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-data-streams]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,350Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-deprecation]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,350Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-enrich]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,350Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-eql]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,350Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-graph]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,351Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-identity-provider]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,351Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ilm]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,352Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-logstash]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,352Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ml]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,353Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-monitoring]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,353Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ql]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,353Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-rollup]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,354Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-security]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,354Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-sql]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,354Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-stack]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,354Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,355Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,356Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "no plugins loaded" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,424Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [62.6gb], net total_space [77.3gb], types [ext4]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,425Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:41,603Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "node name [a5e2f9b5879a], node ID [XJnMmpINR5K-Vbc_fVyzrQ], cluster name [openedx], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:46,667Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "[controller/198] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:47,645Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:49,235Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:49,337Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:50,049Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:50,900Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "initialized" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:50,901Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "starting ..." }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:51,088Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:51,597Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "cluster UUID [qiIPWc0MSfywo2Ad4IVd6w]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:51,773Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "elected-as-master ([1] nodes joined)[{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{EK-1cqhdTL2zjDWLg1wlZA}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 320, version: 1195, delta: master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{EK-1cqhdTL2zjDWLg1wlZA}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:51,881Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{EK-1cqhdTL2zjDWLg1wlZA}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 320, version: 1195, reason: Publication{term=320, version=1195}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:51,946Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:51,947Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "started", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:54,201Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "license [cad6af7c-6866-432c-8806-31d76cd60b38] mode [basic] - valid", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:54,216Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:54,275Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "recovered [2] indices into cluster_state", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:08:05,950Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[courseware_content][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }

Many thanks :slight_smile:

Hi,

Occurred again today, logs attached.


elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:44,750Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-sql]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:44,750Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-stack]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:44,750Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:44,750Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:44,751Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "no plugins loaded" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:44,815Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [62.3gb], net total_space [77.3gb], types [ext4]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:44,816Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:44,954Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "node name [a5e2f9b5879a], node ID [XJnMmpINR5K-Vbc_fVyzrQ], cluster name [openedx], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:50,096Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "[controller/198] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:51,223Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:52,703Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:52,848Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:53,677Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:54,434Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "initialized" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:54,437Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "starting ..." }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:54,615Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.7:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:55,740Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "cluster UUID [qiIPWc0MSfywo2Ad4IVd6w]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:56,045Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "elected-as-master ([1] nodes joined)[{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{Em8vQaahQCiiF1Rlm_Vu_w}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 329, version: 1249, delta: master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{Em8vQaahQCiiF1Rlm_Vu_w}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:56,751Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{Em8vQaahQCiiF1Rlm_Vu_w}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 329, version: 1249, reason: Publication{term=329, version=1249}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:56,892Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.7:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:56,893Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "started", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:58,105Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "license [cad6af7c-6866-432c-8806-31d76cd60b38] mode [basic] - valid", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:58,109Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:46:58,130Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "recovered [2] indices into cluster_state", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:47:02,476Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[course_info][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:28,261Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "version[7.10.1], pid[9], build[default/docker/1c34507e66d7db1211f66f3513706fdf548736aa/2020-12-05T01:00:33.671820Z], OS[Linux/5.4.0-100-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:28,270Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:28,271Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-6297220545591555821, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,460Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [aggs-matrix-stats]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,461Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [analysis-common]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,461Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [constant-keyword]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,461Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [flattened]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,462Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [frozen-indices]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,462Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-common]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,463Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-geoip]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,463Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-user-agent]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,467Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [kibana]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,468Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-expression]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,469Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-mustache]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,469Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-painless]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,470Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [mapper-extras]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,470Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [mapper-version]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,470Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [parent-join]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,471Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [percolator]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,471Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [rank-eval]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,472Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [reindex]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,472Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [repositories-metering-api]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,472Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [repository-url]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,472Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [search-business-rules]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,473Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [searchable-snapshots]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,473Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [spatial]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,474Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [transform]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,475Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [transport-netty4]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,475Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [unsigned-long]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,475Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [vectors]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [wildcard]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-analytics]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-async]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-async-search]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-autoscaling]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ccr]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-core]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-data-streams]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-deprecation]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-enrich]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-eql]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-graph]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-identity-provider]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ilm]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-logstash]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ml]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-monitoring]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ql]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-rollup]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-security]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-sql]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-stack]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "no plugins loaded" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,558Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [62.3gb], net total_space [77.3gb], types [ext4]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,559Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:31,736Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "node name [a5e2f9b5879a], node ID [XJnMmpINR5K-Vbc_fVyzrQ], cluster name [openedx], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:37,193Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "[controller/200] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:38,089Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:39,680Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:39,822Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:40,767Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:41,698Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "initialized" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:41,700Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "starting ..." }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:42,378Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.7:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:43,904Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "cluster UUID [qiIPWc0MSfywo2Ad4IVd6w]" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:44,175Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "elected-as-master ([1] nodes joined)[{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{nL_gu2vgTHuG9NrWo7MAsA}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 330, version: 1255, delta: master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{nL_gu2vgTHuG9NrWo7MAsA}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:44,530Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{nL_gu2vgTHuG9NrWo7MAsA}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 330, version: 1255, reason: Publication{term=330, version=1255}" }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:44,928Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.7:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:44,934Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "started", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:47,448Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "license [cad6af7c-6866-432c-8806-31d76cd60b38] mode [basic] - valid", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:47,466Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:47,549Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "recovered [2] indices into cluster_state", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }
elasticsearch_1              | {"type": "server", "timestamp": "2022-03-17T11:49:50,941Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[course_info][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }

I see the following line in your logs:

elasticsearch_1              | {"type": "server", "timestamp": "2022-03-15T18:07:00,511Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[courseware_content][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ"  }

This implies that the cluster was in a RED state, though I don’t know what would be the exact reason. Quite often, it’s because there is less that a certain threshold of free space on disk. I encourage you to explore your Elasticsearch log further to see at what point the cluster switched to RED state.