Connecting to a MongoDB cluster (protocol mongodb+srv)

Tutor
1

When using a MongoDB cluster, the mongo connection string has a protocol or scheme of mongodb+srv, for example:

mongodb+srv://openedx:password@dev.cluster1.mongodb.net:27017/openedx

This is the case, for example, when using Mongo Atlas to host MongoDB.

Has anyone managed to use Mongo Atlas clusters with Tutor?

When running forums, the constructed URL in tutor/templates/build/forum/bin/docker-entrypoint.sh would need to change from mongodb (hardcoded) to an environment variable that could be set to mongodb+srv?

For the LMS and CMS, it is less clear to me how to configure the mongo connection and I can’t find any mention of doing this in native OpenEdx documentation / forums.

Has anyone managed to use Mongo Atlas clusters with Tutor?

2

Hi @james_imsimbi! Tutor users must be able to connect to Atlas clusters, and the fact that the “mongodb+srv” connection string is not supported is considered as a bug – either in Tutor, Open edX or both.

I was not aware of this syntax, which is documented here: https://developer.mongodb.com/article/srv-connection-strings If I understand correctly, with “mongodb+srv” the server addresses are seeded from SRV DNS records.

We could easily modify the mongodb connection strings for the forums, but this change would not affect the connection string used in the LMS and CMS (in edx-platform). These apps use the pymongo library, for which the “mongodb+srv” syntax is supported: mongo_client – Tools for connecting to MongoDB — PyMongo 3.11.2 documentation

However, this new syntax requires that the MongoClient is instantiated by a single URI string. This is not the case in edx-platform: edx-platform/mongo_utils.py at 3c3e35e0f8f6e2d4093bd0175b318c296980a967 · edx/edx-platform · GitHub

It might be possible to redefine the “host” constructor argument in Tutor: tutor/common_all.py at 0f1dbf0ba59e6cd3d77cfb0ba86db6866bf2624e · overhangio/tutor · GitHub
If this does not work, we would have to make a change upstream, in edx-platform.

Unfortunately I do not have access to an Atlas cluster and I do not have the time to implement this change myself. Would you like to attempt making the change yourself? If not, this is typically the kind of change that would be the subject of a consulting contract (for which I would be available).

3

Hi @regis, thank you so much for your detailed response! I am happy to attempt the change and will respond here and make a PR with the results.

4

The way I could get a Mongo Atlas instance to work with tutor CMS/LMS is to use mongo config like this:

mongodb_parameters = {
    "host": "xxxxx-shard-00-00.xxxx.mongodb.net:27017xxxx-shard-00-01.xxxx.mongodb.net:27017,xxxx-shard-00-02.xxxx.mongodb.net:27017",
    "port": 27017,
    "replicaSet": "atlas-xxxx-shard-0",
    "user": "userx",
    "password": "pwdx",

    "db": "openedx",
    "ssl": True,
    "authsource": "admin",
}

All of these parameters can be seen from the Mongo Atlas connection string if you look at a connection string for mongo 3.4

In particular, rather than use mongo+srv to get the server addresses, all addresses are all listed in the host parameter

5

Getting forum to work with a mongo cluster (e.g. Mongo Atlas) required two new environment variables to env/build/forum/Dockerfile:

ENV MONGODB_PROTOCOL "mongodb"
ENV MONGODB_PARAMS ""

MONGODB_PROTOCOL can be set to mongodb+srv for a cluster
MONGODB_PARAMS is optional for adding query parameters to the URI (it must start with ?)

This change is propsoed to env/build/forum/bin/docker-entrypoint.sh:

#!/bin/sh -e


if test "$MONGODB_PROTOCOL" = 'mongodb+srv'; then
  export MONGOHQ_URL="$MONGODB_PROTOCOL://$MONGODB_AUTH$MONGODB_HOST/cs_comments_service$MONGODB_PARAMS"
  export MONGO_WAIT=""
else
  export MONGOHQ_URL="$MONGODB_PROTOCOL://$MONGODB_AUTH$MONGODB_HOST:$MONGODB_PORT/cs_comments_service$MONGODB_PARAMS"
  export MONGO_WAIT="-wait tcp://$MONGODB_HOST:$MONGODB_PORT"
  echo "Waiting for mongodb..."
fi

echo "Waiting for elasticsearch..."
dockerize $MONGO_WAIT -wait $SEARCH_SERVER -wait-retry-interval 5s -timeout 600s

exec "$@"

If the MONGODB_PROTOCOL is set to mongodb+srv then there are two key changes:

  1. The port must not be included in the URL
  2. The host cannot be used as a true server name for “waiting” because it is a DNS entry with only SRV records. I have elected not to wait at all for a cluster because it is unlikely that this cluster is starting up at the same time as the forum, because it is typically provisioned beforehand.
6

@regis , I attempted to commit a branch with a view to created a PR, but it seems there is no general access to do this. What is the process for creating a PR?

7

Your change looks good, you are welcome to open a PR.

Just fork the tutor repo, create a branch, and create a PR from your branch: Creating a pull request from a fork - GitHub Docs

8

Thanks @regis ! I don’t have permission to create a branch - can you create one for me please?

9

No: you need to create a branch in a fork of the repository that you create yourself, from your Github account. Can you do that?

10

Ok thanks, I have made the PR Added configuration changes for mongo atlas style clusters by zakum1 · Pull Request #415 · overhangio/tutor · GitHub

11

@regis sorry that I did not get this finished for the lilac release. Before submitting a new PR, I want to discuss some of what I have found:

While the URI works as the host parameter in pymongo (after adding the dnspython module) we still need to set parameters for ssl, replica_set and the auth_source database - I have not had time to investigate why this is - I am guessing it has to do with the version of pymongo, because, as mentioned previously, when I test the URI outside of edx it worked fine without these parameters

I have managed to get tutor 12.0 to work with a mongo atlas cluster (by adding the above parameters), largely the same as what I did before, except I have removed the MONGODB_PROTOCOL parameter.

I can spend a bit more time on it to find an elegant solution.

12

That would be great!