MongoDB Failover / Replica Set Support

Hey there!
Happy Wednesday night!
Hope all has been well!

I was curious about connecting to high availability replicasets with tutor & mongodb if configuration for that was possible?
I’m thinking it would probably be some custom editing to common.py files?

I was looking at this:


and the ‘replicaSet’ item in the dictionary peeked my curiosity.
OpenEdx, (I think) uses pymongo 2.9.1, there docs here covering highly available mongo clusters and what not:
>>> MongoClient("mongodb://morton.local:27017,morton.local:27018,morton.local:27019")
MongoClient(['morton.local:27019', 'morton.local:27017', 'morton.local:27018'])

made me wonder, if it’s just that easy for the library to handle potentially different mongodb nodes like:
MongoClient(“172.XX.YY.BBB:27017”, “172.XX.YY.CCC:27017”, “172.XX.YY.HHH:27017”) …and if that happened to work successfully (of which I’ll probably try tomorrow … or latter tonight when I can’t sleep) …but if that wasn’t too difficult to do from the PyMongo library perspective … how might that translate to configuration and or tutor edits?

Thanks for any input, hazings, thoughts, feedback, and such! :slight_smile:

So I was curious, inside the “auth.json” file that has the templating, would it work to say, anywhere the “DOC_STORE_CONFIG” is listed modify the “host” to be changed to “replicaSet”, and ensuring that the MONGODB_HOST is actually like: AAA.BBB.CCC.GGG, AAA.BBB.CCC.EEE, AAA.BBB.CCC.DDD - given that we use standard port of 27017?
I’m starting to test this to try to see what impact the edits have.

  9   "CONTENTSTORE": {
 10     "ENGINE": "xmodule.contentstore.mongo.MongoContentStore",
 11     "DOC_STORE_CONFIG": {
 12       "host": "{{ MONGODB_HOST }}",
 13       "port": {{ MONGODB_PORT }},
 14       {% if MONGODB_USERNAME and MONGODB_PASSWORD %}
 15       "user": "{{ MONGODB_USERNAME }}",
 16       "password": "{{ MONGODB_PASSWORD }}",
 17       {% endif %}
 18       "db": "{{ MONGODB_DATABASE }}"
 19     }
 20   },
 21   "DOC_STORE_CONFIG": {
 22     "replicaSet": "{{ MONGODB_HOST }}",
 23     {% if MONGODB_USERNAME and MONGODB_PASSWORD %}
 24     "user": "{{ MONGODB_USERNAME }}",
 25     "password": "{{ MONGODB_PASSWORD }}",
 26     {% else %}
 27     "user": null,
 28     "password": null,
 29     {% endif %}
 30     "db": "{{ MONGODB_DATABASE }}"

However, I have noted that there is:

Initialising forum...

docker-compose -f /home/ec2-user/.local/share/tutor/env/local/docker-compose.yml -f /home/ec2-user/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /home/ec2-user/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm forum-job sh -e -c export MONGOHQ_URL="mongodb://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service"

bundle exec rake search:initialize

bundle exec rake search:rebuild_index

thinking that the: “mongodb://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service” would need custom edits, since the mogodb connection url will be different for using a replica set

Does any of y’all think this may be the right path to head down?

Are you guys working together? https://github.com/overhangio/tutor/pull/372 This PR addressing replica sets was opened just this morning.

Hahaha wow! That’s so uncanny! Super coincidental! :smiley::laughing:

But no, haha, we’re not working together hahah

@irishgordo, hi!

With latest tutor(from source code) you are able to set up MongoDB connection you want. I created simple plugin like this:

name: mongodb_replicaset
version: 0.1.0
patches:
  openedx-common-settings: |
    mongodb_parameters = {
        "read_preference": "PRIMARY",
        "replicaSet": "{{ MONGODB_REPLICASET }}",
        "host": [{% for host in MONGODB_HOST %}"{{ host }}"{{ "," if not loop.last }}{% endfor %}],
        "port": {{ MONGODB_PORT }},
        "user": "{{ MONGODB_USERNAME }}",
        "password": "{{ MONGODB_PASSWORD }}",
        "db": "{{ MONGODB_DATABASE }}"
    }
    DOC_STORE_CONFIG = mongodb_parameters
    CONTENTSTORE = {
        "ENGINE": "xmodule.contentstore.mongo.MongoContentStore",
        "ADDITIONAL_OPTIONS": {},
        "DOC_STORE_CONFIG": DOC_STORE_CONFIG
    }

    # Load module store settings from config files
    update_module_store_settings(MODULESTORE, doc_store_settings=DOC_STORE_CONFIG)
1 Like

Hey @odemakov that pull request was great! and perfect timing too!
I manually implemented those changes to our auth.json file and config.yml file and it’s working like a dream on the server!
And thanks to the work you did I am gaining a deeper understanding of how the templating with https://jinja.palletsprojects.com/en/2.11.x/
is kinda taking place

I haven’t read up on plugins with tutor and should :slight_smile: