Failure to initialise forum - No such host

In Tutor (tutor, version 10.0.6) I keep getting initialising failures when running tutor local quickstart. These errors were not present before 10.0.4. I also have these plugins running:
ecommerce==10.0.2
discovery==10.0.1

The output for forum init failure:

Initialising forum...

docker-compose -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.yml -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.jobs.yml run --rm forum-job sh -e -c export MONGOHQ_URL="mongodb://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service"

bundle exec rake search:initialize

bundle exec rake search:rebuild_index

Waiting for mongodb/elasticsearch...

2020/06/22 13:20:38 Waiting for: tcp://mongodb:27017

2020/06/22 13:20:38 Waiting for: http://elasticsearch:9200

2020/06/22 13:20:38 Problem with dial: dial tcp: lookup mongodb on 127.0.0.11:53: no such host. Sleeping 5s

2020/06/22 13:20:38 Received 200 from http://elasticsearch:9200

[REMOVED MANY LINES OF … no such host]

2020/06/22 13:30:30 Problem with dial: dial tcp: lookup mongodb on 127.0.0.11:53: no such host. Sleeping 5s

2020/06/22 13:30:36 Problem with dial: dial tcp: lookup mongodb on 127.0.0.11:53: no such host. Sleeping 5s

2020/06/22 13:30:37 Timeout after 10m0s waiting on dependencies to become available: [tcp://mongodb:27017 http://elasticsearch:9200]

Error: Command failed with status 1: docker-compose -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.yml -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.jobs.yml run --rm forum-job sh -e -c export MONGOHQ_URL="mongodb://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service"

bundle exec rake search:initialize

bundle exec rake search:rebuild_index

The mongo container keeps restarting (mongo:3.6.18 / tutor_local_mongodb_1) and logs show:

2020-06-22T14:00:49.089+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=7f50a4a85f48
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten] db version v3.6.18
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten] git version: 2005f25eed7ed88fa698d9b800fe536bb0410ba4
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten] modules: none
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten] build environment:
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-06-22T14:00:49.090+0000 I CONTROL  [initandlisten] options: { net: { bindIpAll: true }, storage: { engine: "wiredTiger", journal: { enabled: false }, mmapv1: { smallFiles: true } } }
2020-06-22T14:00:49.092+0000 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2020-06-22T14:00:49.097+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2020-06-22T14:00:49.097+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7497M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),,log=(enabled=false),
2020-06-22T14:00:49.573+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1592834449:573520][1:0x7fce40b14a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1592834449:573520][1:0x7fce40b14a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2020-06-22T14:00:49.573+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1592834449:573590][1:0x7fce40b14a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata Raw: [1592834449:573590][1:0x7fce40b14a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata
2020-06-22T14:00:49.573+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1592834449:573613][1:0x7fce40b14a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1592834449:573613][1:0x7fce40b14a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2020-06-22T14:00:49.573+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1592834449:573632][1:0x7fce40b14a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1592834449:573632][1:0x7fce40b14a40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options
2020-06-22T14:00:49.575+0000 F STORAGE  [initandlisten] WiredTiger metadata corruption detected
2020-06-22T14:00:49.575+0000 F STORAGE  [initandlisten] This version of MongoDB is unable to repair this kind of corruption, but version 4.0.3+ may be able to repair it. See http://dochub.mongodb.org/core/repair for more information.
2020-06-22T14:00:49.575+0000 F -        [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 71
2020-06-22T14:00:49.575+0000 F -        [initandlisten] 

***aborting after fassert() failure

All and any tips greatly appreciated!

@ak00001 what are the logs from the mongodb container? If the mongodb database has not been properly upgraded, the Mongodb container will fail to boot. You probably just have to run tutor local upgrade --from=ironwood.

The log from Mongodb is posted in the first entry.

Looks like a corrupted data problem. I might just go back to my backup and start upgrade all over.

Right! Sorry about that.

Before that, can you please try to run tutor local run mongodb --repair? This might fix your data.
Anyway, I’d like to know if you can cleanly upgrade your ironwood data without resorting to --repair. If not, we will have to modify the upgrade process.

–repair does not seem to make a difference:

(tutor) admin@As-MacBook-Pro .virtualenvs % tutor local run mongodb --repair
docker-compose -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.yml -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm mongodb --repair
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=69ca8a895863
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten] db version v3.6.18
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten] git version: 2005f25eed7ed88fa698d9b800fe536bb0410ba4
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten] modules: none
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten] build environment:
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten]     distmod: ubuntu1604
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-06-25T09:00:16.396+0000 I CONTROL  [initandlisten] options: { net: { bindIpAll: true }, repair: true }
2020-06-25T09:00:16.399+0000 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2020-06-25T09:00:16.402+0000 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-06-25T09:00:16.403+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2020-06-25T09:00:16.404+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7497M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),,log=(enabled=false),
2020-06-25T09:00:16.833+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1593075616:833784][1:0x7f3ba11cca40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1593075616:833784][1:0x7f3ba11cca40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 604: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2020-06-25T09:00:16.833+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1593075616:833875][1:0x7f3ba11cca40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata Raw: [1593075616:833875][1:0x7f3ba11cca40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 611: WiredTiger has failed to open its metadata
2020-06-25T09:00:16.833+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1593075616:833886][1:0x7f3ba11cca40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1593075616:833886][1:0x7f3ba11cca40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 614: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2020-06-25T09:00:16.833+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1593075616:833910][1:0x7f3ba11cca40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1593075616:833910][1:0x7f3ba11cca40], file:WiredTiger.wt, connection: __wt_btree_tree_open, 617: You should confirm that you have opened the database with the correct options including all encryption and compression options
2020-06-25T09:00:16.835+0000 F STORAGE  [initandlisten] WiredTiger metadata corruption detected
2020-06-25T09:00:16.835+0000 F STORAGE  [initandlisten] This version of MongoDB is unable to repair this kind of corruption, but version 4.0.3+ may be able to repair it. See http://dochub.mongodb.org/core/repair for more information.
2020-06-25T09:00:16.835+0000 F -        [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 71
2020-06-25T09:00:16.835+0000 F -        [initandlisten] 

***aborting after fassert() failure


Error: Command failed with status 14: docker-compose -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.yml -f /Users/admin/Library/Application Support/tutor/env/local/docker-compose.prod.yml --project-name tutor_local run --rm mongodb --repair

Ok, please try to upgrade again from your backup.