MongoDB Failover / Replica Set Support

Hey there!
Happy Wednesday night!
Hope all has been well!

I was curious about connecting to high availability replicasets with tutor & mongodb if configuration for that was possible?
I’m thinking it would probably be some custom editing to files?

I was looking at this:

and the ‘replicaSet’ item in the dictionary peeked my curiosity.
OpenEdx, (I think) uses pymongo 2.9.1, there docs here covering highly available mongo clusters and what not:
>>> MongoClient("mongodb://morton.local:27017,morton.local:27018,morton.local:27019")
MongoClient(['morton.local:27019', 'morton.local:27017', 'morton.local:27018'])

made me wonder, if it’s just that easy for the library to handle potentially different mongodb nodes like:
MongoClient(“172.XX.YY.BBB:27017”, “172.XX.YY.CCC:27017”, “172.XX.YY.HHH:27017”) …and if that happened to work successfully (of which I’ll probably try tomorrow … or latter tonight when I can’t sleep) …but if that wasn’t too difficult to do from the PyMongo library perspective … how might that translate to configuration and or tutor edits?

Thanks for any input, hazings, thoughts, feedback, and such! :slight_smile:

So I was curious, inside the “auth.json” file that has the templating, would it work to say, anywhere the “DOC_STORE_CONFIG” is listed modify the “host” to be changed to “replicaSet”, and ensuring that the MONGODB_HOST is actually like: AAA.BBB.CCC.GGG, AAA.BBB.CCC.EEE, AAA.BBB.CCC.DDD - given that we use standard port of 27017?
I’m starting to test this to try to see what impact the edits have.

 10     "ENGINE": "xmodule.contentstore.mongo.MongoContentStore",
 11     "DOC_STORE_CONFIG": {
 12       "host": "{{ MONGODB_HOST }}",
 13       "port": {{ MONGODB_PORT }},
 15       "user": "{{ MONGODB_USERNAME }}",
 16       "password": "{{ MONGODB_PASSWORD }}",
 17       {% endif %}
 18       "db": "{{ MONGODB_DATABASE }}"
 19     }
 20   },
 22     "replicaSet": "{{ MONGODB_HOST }}",
 24     "user": "{{ MONGODB_USERNAME }}",
 25     "password": "{{ MONGODB_PASSWORD }}",
 26     {% else %}
 27     "user": null,
 28     "password": null,
 29     {% endif %}
 30     "db": "{{ MONGODB_DATABASE }}"

However, I have noted that there is:

Initialising forum...

docker-compose -f /home/ec2-user/.local/share/tutor/env/local/docker-compose.yml -f /home/ec2-user/.local/share/tutor/env/local/ --project-name tutor_local -f /home/ec2-user/.local/share/tutor/env/local/ run --rm forum-job sh -e -c export MONGOHQ_URL="mongodb://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service"

bundle exec rake search:initialize

bundle exec rake search:rebuild_index

thinking that the: “mongodb://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service” would need custom edits, since the mogodb connection url will be different for using a replica set

Does any of y’all think this may be the right path to head down?

Are you guys working together? This PR addressing replica sets was opened just this morning.

Hahaha wow! That’s so uncanny! Super coincidental! :smiley::laughing:

But no, haha, we’re not working together hahah

@irishgordo, hi!

With latest tutor(from source code) you are able to set up MongoDB connection you want. I created simple plugin like this:

name: mongodb_replicaset
version: 0.1.0
  openedx-common-settings: |
    mongodb_parameters = {
        "read_preference": "PRIMARY",
        "replicaSet": "{{ MONGODB_REPLICASET }}",
        "host": [{% for host in MONGODB_HOST %}"{{ host }}"{{ "," if not loop.last }}{% endfor %}],
        "port": {{ MONGODB_PORT }},
        "user": "{{ MONGODB_USERNAME }}",
        "password": "{{ MONGODB_PASSWORD }}",
        "db": "{{ MONGODB_DATABASE }}"
    DOC_STORE_CONFIG = mongodb_parameters
        "ENGINE": "xmodule.contentstore.mongo.MongoContentStore",

    # Load module store settings from config files
    update_module_store_settings(MODULESTORE, doc_store_settings=DOC_STORE_CONFIG)
1 Like

Hey @odemakov that pull request was great! and perfect timing too!
I manually implemented those changes to our auth.json file and config.yml file and it’s working like a dream on the server!
And thanks to the work you did I am gaining a deeper understanding of how the templating with
is kinda taking place

I haven’t read up on plugins with tutor and should :slight_smile:

Hi @irishgordo, you are welcome. Feel free to share your work with community. :slight_smile:

I was wondering however, I keep seeing this error, just leveraging the plugin, even though I have set ACTIVATE_FORUM: false in the config.yml:

2020/10/26 19:37:16 bad hostname provided: tcp://['',. parse tcp://['',: missing ']' in host

Error: Command failed with status 1: docker-compose -f /Users/usernamel/Library/Application Support/tutor/env/local/docker-compose.yml -f /Users/mike.russell/Library/Application Support/tutor/env/local/ --project-name tutor_local -f /Users/username/Library/Application Support/tutor/env/local/ run --rm forum-job sh -e -c export MONGOHQ_URL="mongodb://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service"

bundle exec rake search:initialize

bundle exec rake search:rebuild_index

I’m thinking there might be an issue with it being an IPv6 address:

ValueError: an IPv6 address literal must be enclosed in '[' and ']' according to RFC 2732.

Forum mongo DB connection is in our todo list, we didn’t check it out yet.

I guess for mongo connection string there should be something like this:

@irishgordo the fact that the forum init job is running despite the fact that the forum is disabled is a bug. Can you please open a Github issue?

1 Like

Can anyone please let me know, how we can configure a remote mongo server for cs_comment_service? I have configured remote mongodb for lms and cms via plugin by changing mongodb parameters but when I try to modify MONGOHQ_URL for cs_comment_service, the forum container keeps on restarting.


  1. Accessed the forum container

  2. bin/
    Changed the MONGOHQ_URL here but it was not working.

  3. On tutor env, here is file path:
    We can find forum configurations here. Do we need to modify this file?