Hi,
After a period of time with production tutor edx running, the system suddenly slows to an unusable speed.
Running docker stats shows elasticsearch1 using over 150% CPU at times! No clue why this is happening, especially with no reported use of the search feature.
Any particular ideas as to why this might be happening? Something sets off elasticsearch into a frenzy but no way to trigger it other than waiting >1 week so far.
Thanks
Also worth mentioning the problem does eventually resolve itself and then return again later - seems to be unpredictable.
Only relevant thing in logs I could find is the following, but not entirely sure what it’s trying to do. Granted the system is on 4GB of ram but there are very few users (~5) and only a few small courses.
How can I increase vm.max_map_count - not sure what unit ‘65530’ is but it wants me to increase the allocated memory approx 4x by the looks of it?
I have run
sysctl -w vm.max_map_count=262144
per Virtual memory | Elasticsearch Guide [8.1] | Elastic and I will update if the issue returns. Let me know if you have any other suggestions or ideas.
Thanks
Let us know if the problem occurs again. If it does, please paste the logs from the elasticsearch container:
tutor local logs --tail=100
(please don’t post screenshots. Pasted logs make it easier both for us, to investigate the issue, and for other users who face the same problem to find a solution.)
Hi,
Many thanks for your response. The problem has since occurred again, seems to be related to when saving/publishing courses - I suspect it is something going on when reindexing the search as it occurred after I published a small change to a course.
The system has 4GB of memory but only one course actually in use and only approximately 5 users.
Logs attached below:
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:24,940Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-sql]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:24,940Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-stack]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:24,940Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:24,941Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:24,941Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "no plugins loaded" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:25,027Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [62.6gb], net total_space [77.3gb], types [ext4]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:25,028Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:25,212Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "node name [a5e2f9b5879a], node ID [XJnMmpINR5K-Vbc_fVyzrQ], cluster name [openedx], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:30,226Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "[controller/198] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:30,935Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:32,618Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:32,768Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:33,758Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:34,383Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "initialized" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:34,384Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "starting ..." }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:34,533Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:37,946Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "cluster UUID [qiIPWc0MSfywo2Ad4IVd6w]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:38,693Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "elected-as-master ([1] nodes joined)[{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{kYNmqresTDSQlJR7xmHodw}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 319, version: 1189, delta: master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{kYNmqresTDSQlJR7xmHodw}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:39,508Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{kYNmqresTDSQlJR7xmHodw}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 319, version: 1189, reason: Publication{term=319, version=1189}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:40,944Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:40,964Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "started", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:44,716Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "license [cad6af7c-6866-432c-8806-31d76cd60b38] mode [basic] - valid", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:44,733Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:06:44,790Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "recovered [2] indices into cluster_state", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:00,511Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[courseware_content][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:38,174Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "version[7.10.1], pid[7], build[default/docker/1c34507e66d7db1211f66f3513706fdf548736aa/2020-12-05T01:00:33.671820Z], OS[Linux/5.4.0-100-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:38,181Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:38,183Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-14137560325327189926, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,334Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [aggs-matrix-stats]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,334Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [analysis-common]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,335Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [constant-keyword]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,335Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [flattened]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,335Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [frozen-indices]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,336Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-common]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,336Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-geoip]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,337Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-user-agent]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,340Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [kibana]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,341Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-expression]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,341Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-mustache]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,342Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-painless]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,342Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [mapper-extras]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,342Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [mapper-version]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,342Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [parent-join]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,343Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [percolator]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,343Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [rank-eval]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,344Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [reindex]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,344Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [repositories-metering-api]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,344Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [repository-url]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,344Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [search-business-rules]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,345Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [searchable-snapshots]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,345Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [spatial]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,346Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [transform]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,347Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [transport-netty4]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,347Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [unsigned-long]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,347Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [vectors]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,347Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [wildcard]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,348Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-analytics]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,348Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-async]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,348Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-async-search]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,349Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-autoscaling]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,349Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ccr]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,349Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-core]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,349Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-data-streams]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,350Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-deprecation]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,350Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-enrich]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,350Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-eql]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,350Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-graph]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,351Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-identity-provider]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,351Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ilm]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,352Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-logstash]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,352Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ml]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,353Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-monitoring]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,353Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ql]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,353Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-rollup]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,354Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-security]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,354Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-sql]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,354Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-stack]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,354Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,355Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,356Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "no plugins loaded" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,424Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [62.6gb], net total_space [77.3gb], types [ext4]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,425Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:41,603Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "node name [a5e2f9b5879a], node ID [XJnMmpINR5K-Vbc_fVyzrQ], cluster name [openedx], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:46,667Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "[controller/198] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:47,645Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:49,235Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:49,337Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:50,049Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:50,900Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "initialized" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:50,901Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "starting ..." }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:51,088Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:51,597Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "cluster UUID [qiIPWc0MSfywo2Ad4IVd6w]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:51,773Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "elected-as-master ([1] nodes joined)[{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{EK-1cqhdTL2zjDWLg1wlZA}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 320, version: 1195, delta: master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{EK-1cqhdTL2zjDWLg1wlZA}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:51,881Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{EK-1cqhdTL2zjDWLg1wlZA}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 320, version: 1195, reason: Publication{term=320, version=1195}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:51,946Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:51,947Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "started", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:54,201Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "license [cad6af7c-6866-432c-8806-31d76cd60b38] mode [basic] - valid", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:54,216Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:54,275Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "recovered [2] indices into cluster_state", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:08:05,950Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[courseware_content][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
Many thanks 
Hi,
Occurred again today, logs attached.
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:44,750Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-sql]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:44,750Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-stack]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:44,750Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:44,750Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:44,751Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "no plugins loaded" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:44,815Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [62.3gb], net total_space [77.3gb], types [ext4]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:44,816Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:44,954Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "node name [a5e2f9b5879a], node ID [XJnMmpINR5K-Vbc_fVyzrQ], cluster name [openedx], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:50,096Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "[controller/198] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:51,223Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:52,703Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:52,848Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:53,677Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:54,434Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "initialized" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:54,437Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "starting ..." }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:54,615Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.7:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:55,740Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "cluster UUID [qiIPWc0MSfywo2Ad4IVd6w]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:56,045Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "elected-as-master ([1] nodes joined)[{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{Em8vQaahQCiiF1Rlm_Vu_w}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 329, version: 1249, delta: master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{Em8vQaahQCiiF1Rlm_Vu_w}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:56,751Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{Em8vQaahQCiiF1Rlm_Vu_w}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 329, version: 1249, reason: Publication{term=329, version=1249}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:56,892Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.7:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:56,893Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "started", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:58,105Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "license [cad6af7c-6866-432c-8806-31d76cd60b38] mode [basic] - valid", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:58,109Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:46:58,130Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "recovered [2] indices into cluster_state", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:47:02,476Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[course_info][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:28,261Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "version[7.10.1], pid[9], build[default/docker/1c34507e66d7db1211f66f3513706fdf548736aa/2020-12-05T01:00:33.671820Z], OS[Linux/5.4.0-100-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:28,270Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:28,271Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-6297220545591555821, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1g, -Xmx1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,460Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [aggs-matrix-stats]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,461Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [analysis-common]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,461Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [constant-keyword]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,461Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [flattened]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,462Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [frozen-indices]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,462Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-common]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,463Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-geoip]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,463Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [ingest-user-agent]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,467Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [kibana]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,468Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-expression]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,469Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-mustache]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,469Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [lang-painless]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,470Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [mapper-extras]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,470Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [mapper-version]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,470Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [parent-join]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,471Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [percolator]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,471Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [rank-eval]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,472Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [reindex]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,472Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [repositories-metering-api]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,472Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [repository-url]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,472Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [search-business-rules]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,473Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [searchable-snapshots]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,473Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [spatial]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,474Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [transform]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,475Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [transport-netty4]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,475Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [unsigned-long]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,475Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [vectors]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [wildcard]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-analytics]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-async]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,476Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-async-search]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-autoscaling]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ccr]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-core]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,477Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-data-streams]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-deprecation]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-enrich]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-eql]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,478Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-graph]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-identity-provider]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,479Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ilm]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-logstash]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,480Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ml]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-monitoring]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-ql]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-rollup]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-security]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,481Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-sql]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-stack]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-voting-only-node]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,482Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "loaded module [x-pack-watcher]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,483Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "no plugins loaded" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,558Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [62.3gb], net total_space [77.3gb], types [ext4]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,559Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:31,736Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "node name [a5e2f9b5879a], node ID [XJnMmpINR5K-Vbc_fVyzrQ], cluster name [openedx], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:37,193Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "[controller/200] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:38,089Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:39,680Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:39,822Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:40,767Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:41,698Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "initialized" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:41,700Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "starting ..." }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:42,378Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.7:9300}, bound_addresses {0.0.0.0:9300}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:43,904Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "cluster UUID [qiIPWc0MSfywo2Ad4IVd6w]" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:44,175Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "elected-as-master ([1] nodes joined)[{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{nL_gu2vgTHuG9NrWo7MAsA}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 330, version: 1255, delta: master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{nL_gu2vgTHuG9NrWo7MAsA}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:44,530Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "master node changed {previous [], current [{a5e2f9b5879a}{XJnMmpINR5K-Vbc_fVyzrQ}{nL_gu2vgTHuG9NrWo7MAsA}{172.18.0.7}{172.18.0.7:9300}{cdhilmrstw}{ml.machine_memory=4122787840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 330, version: 1255, reason: Publication{term=330, version=1255}" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:44,928Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "publish_address {172.18.0.7:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:44,934Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "started", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:47,448Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "license [cad6af7c-6866-432c-8806-31d76cd60b38] mode [basic] - valid", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:47,466Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:47,549Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "recovered [2] indices into cluster_state", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-17T11:49:50,941Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[course_info][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
I see the following line in your logs:
elasticsearch_1 | {"type": "server", "timestamp": "2022-03-15T18:07:00,511Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "openedx", "node.name": "a5e2f9b5879a", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[courseware_content][0]]]).", "cluster.uuid": "qiIPWc0MSfywo2Ad4IVd6w", "node.id": "XJnMmpINR5K-Vbc_fVyzrQ" }
This implies that the cluster was in a RED state, though I don’t know what would be the exact reason. Quite often, it’s because there is less that a certain threshold of free space on disk. I encourage you to explore your Elasticsearch log further to see at what point the cluster switched to RED state.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.