137 Docker Error With Remote MySQL DB

Tutor
1

Good Evening.

I recently created a new MySQL 5.6 database on Google Cloud. I am able to connect to the database from my local machine as well as the cloud server running Tutor-Open EDX.

I edited the config.yml file to add the MySQL connection properties for this remote database, saved and ran tutor local quickstart and got the following error.

databaseError
databaseError1585×617 31.3 KB

I checked the posts here and couldn’t find anyone with this exact issue. I tried deleting the old /data/mysql directory encase it was causing issues on the tutor-open-edx server and I got the same error.

I checked the mysql server logs and didn’t see any errors. I did see it created the ‘openedx’ schema so it seems it can connect to the database as well.

While it did create the ‘openedx’ schema. None of the tables in the database were created.

I also ran a ‘top’ command in another shell and the server I was running tutor on never seemed to run out of memory (tutor server has 4GB, mysql has 4GB and mongo has 2GB).

Looking for suggestions on what could be the issue causing this failure.

Does tutor create logs anywhere during bootup that might give more details about where it was when it failed?

2

Hi @sbrewer! Please check the platform logs with: tutor local logs or tutor local logs --tail=100.

3

The logs were missing when I checked last night. Gave it another go this morning and was able to see some logs. Nothing was obvious but in the forum logs I saw some warnings about certain actions being unsupported with that version of Mongo. I removed the new mongo db from the config and it booted up. So now I have to try and figure out what it doesn’t like about the new mongo database. Couldn’t find a specific version supported anywhere so if you know the requirements for a mongo database that would be helpful.

4

@regis I saw the change log notes that 3.8 fixed an issue with external databases and forums. I thought the issue might be related so I updated the platform to 3.8 and rebooted.

Saw the same error in the forum logs. I thought it might be an issue using the root account for database as the error mentions not authorized. I created a new user in the mongo database and gave it the ‘dbOwner’ role which gives it permission to do anything. Update config.yml with new user name and did a quick start. Same error. The logs are below for the forum section as it appears to be only area with error. I have also included a few logs from LMS section which mentions a worker failed to start. I am guessing the forum failing to work is causing the LMS section to fail?

lms_1               | 2019-12-01 20:36:38,662 WARNING 11 [enterprise.utils] utils.py:50 - Could not import Registry from third_party_auth.provider
lms_1               | 2019-12-01 20:36:38,663 WARNING 11 [enterprise.utils] utils.py:51 - cannot import name _LTI_BACKENDS
lms_1               | 2019-12-01 20:36:38,677 WARNING 13 [enterprise.utils] utils.py:50 - Could not import Registry from third_party_auth.provider
lms_1               | 2019-12-01 20:36:38,677 WARNING 13 [enterprise.utils] utils.py:51 - cannot import name _LTI_BACKENDS
lms_1               | Traceback (most recent call last):
lms_1               |   File "/openedx/venv/bin/gunicorn", line 11, in <module>
lms_1               |     sys.exit(run())
lms_1               |   File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
lms_1               |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
lms_1               |   File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 166, in run
lms_1               |     super(Application, self).run()
lms_1               |   File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 71, in run
lms_1               |     Arbiter(self).run()
lms_1               |   File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 196, in run
lms_1               |     self.halt(reason=inst.reason, exit_status=inst.exit_status)
lms_1               |   File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 292, in halt
lms_1               |     self.stop()
lms_1               |   File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 343, in stop
lms_1               |     time.sleep(0.1)
lms_1               |   File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 209, in handle_chld
lms_1               |     self.reap_workers()
lms_1               |   File "/openedx/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 459, in reap_workers
lms_1               |     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
lms_1               | gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

forum_1             | W, [2019-12-01T20:30:21.846235 #15]  WARN -- : Overwriting existing field _id in class User.
forum_1             | W, [2019-12-01T20:30:21.902140 #15]  WARN -- : MONGODB | Unsupported client option 'max_retries'. It will be ignored.
forum_1             | W, [2019-12-01T20:30:21.902337 #15]  WARN -- : MONGODB | Unsupported client option 'retry_interval'. It will be ignored.
forum_1             | W, [2019-12-01T20:30:21.902447 #15]  WARN -- : MONGODB | Unsupported client option 'timeout'. It will be ignored.
forum_1             | /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/result.rb:226:in `validate!': not authorized for query on cs_comments_service.blocked_hash (13) (Mongo::Error::OperationFailure)
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/executable.rb:36:in `block in execute'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/server/connection_pool.rb:111:in `with_connection'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/server/context.rb:63:in `with_connection'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/executable.rb:34:in `execute'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view.rb:164:in `send_initial_query'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view/iterable.rb:41:in `block in each'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/retryable.rb:46:in `read_with_retry'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view/iterable.rb:39:in `each'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/query_cache.rb:207:in `each'
forum_1             |   from /openedx/cs_comments_service/app.rb:172:in `map'
forum_1             |   from /openedx/cs_comments_service/app.rb:172:in `<top (required)>'
forum_1             |   from config/unicorn_tcp.rb:4:in `require'
forum_1             |   from config/unicorn_tcp.rb:4:in `reload'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:84:in `instance_eval'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:84:in `reload'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:77:in `initialize'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:77:in `new'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:77:in `initialize'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/bin/unicorn:126:in `new'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/bin/unicorn:126:in `<top (required)>'
forum_1             |   from ./bin/unicorn:16:in `load'
forum_1             |   from ./bin/unicorn:16:in `<main>'
forum_1             | Waiting for mongodb/elasticsearch...
forum_1             | 2019/12/01 20:30:23 Waiting for: tcp://REPLACED.206:27017
forum_1             | 2019/12/01 20:30:23 Waiting for: http://elasticsearch:9200/content
forum_1             | 2019/12/01 20:30:23 Received 200 from http://elasticsearch:9200/content
forum_1             | 2019/12/01 20:30:23 Connected to tcp://REPLACED.206:27017
forum_1             | /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/elasticsearch-transport-1.1.2/lib/elasticsearch/transport/transport/base.rb:52: warning: constant ::Fixnum is deprecated
forum_1             | /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/elasticsearch-transport-1.1.2/lib/elasticsearch/transport/transport/base.rb:54: warning: constant ::Fixnum is deprecated
forum_1             | W, [2019-12-01T20:30:25.911456 #15]  WARN -- : Overwriting existing field _id in class User.
forum_1             | W, [2019-12-01T20:30:25.968762 #15]  WARN -- : MONGODB | Unsupported client option 'max_retries'. It will be ignored.
forum_1             | W, [2019-12-01T20:30:25.968850 #15]  WARN -- : MONGODB | Unsupported client option 'retry_interval'. It will be ignored.
forum_1             | W, [2019-12-01T20:30:25.968867 #15]  WARN -- : MONGODB | Unsupported client option 'timeout'. It will be ignored.
forum_1             | /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/result.rb:226:in `validate!': not authorized for query on cs_comments_service.blocked_hash (13) (Mongo::Error::OperationFailure)
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/executable.rb:36:in `block in execute'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/server/connection_pool.rb:111:in `with_connection'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/server/context.rb:63:in `with_connection'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/executable.rb:34:in `execute'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view.rb:164:in `send_initial_query'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view/iterable.rb:41:in `block in each'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/retryable.rb:46:in `read_with_retry'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view/iterable.rb:39:in `each'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/query_cache.rb:207:in `each'
forum_1             |   from /openedx/cs_comments_service/app.rb:172:in `map'
forum_1             |   from /openedx/cs_comments_service/app.rb:172:in `<top (required)>'
forum_1             |   from config/unicorn_tcp.rb:4:in `require'
forum_1             |   from config/unicorn_tcp.rb:4:in `reload'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:84:in `instance_eval'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:84:in `reload'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:77:in `initialize'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:77:in `new'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:77:in `initialize'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/bin/unicorn:126:in `new'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/bin/unicorn:126:in `<top (required)>'
forum_1             |   from ./bin/unicorn:16:in `load'
forum_1             |   from ./bin/unicorn:16:in `<main>'
forum_1             | Waiting for mongodb/elasticsearch...
forum_1             | 2019/12/01 20:30:28 Waiting for: tcp://REPLACED.206:27017
forum_1             | 2019/12/01 20:30:28 Waiting for: http://elasticsearch:9200/content
forum_1             | 2019/12/01 20:30:28 Received 200 from http://elasticsearch:9200/content
forum_1             | 2019/12/01 20:30:28 Connected to tcp://REPLACED.206:27017
forum_1             | /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/elasticsearch-transport-1.1.2/lib/elasticsearch/transport/transport/base.rb:52: warning: constant ::Fixnum is deprecated
forum_1             | /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/elasticsearch-transport-1.1.2/lib/elasticsearch/transport/transport/base.rb:54: warning: constant ::Fixnum is deprecated
forum_1             | W, [2019-12-01T20:30:30.935245 #13]  WARN -- : Overwriting existing field _id in class User.
forum_1             | W, [2019-12-01T20:30:30.994704 #13]  WARN -- : MONGODB | Unsupported client option 'max_retries'. It will be ignored.
forum_1             | W, [2019-12-01T20:30:30.994814 #13]  WARN -- : MONGODB | Unsupported client option 'retry_interval'. It will be ignored.
forum_1             | W, [2019-12-01T20:30:30.994852 #13]  WARN -- : MONGODB | Unsupported client option 'timeout'. It will be ignored.
forum_1             | /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/result.rb:226:in `validate!': not authorized for query on cs_comments_service.blocked_hash (13) (Mongo::Error::OperationFailure)
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/executable.rb:36:in `block in execute'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/server/connection_pool.rb:111:in `with_connection'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/server/context.rb:63:in `with_connection'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/operation/executable.rb:34:in `execute'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view.rb:164:in `send_initial_query'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view/iterable.rb:41:in `block in each'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/retryable.rb:46:in `read_with_retry'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongo-2.1.2/lib/mongo/collection/view/iterable.rb:39:in `each'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/mongoid-5.0.0/lib/mongoid/query_cache.rb:207:in `each'
forum_1             |   from /openedx/cs_comments_service/app.rb:172:in `map'
forum_1             |   from /openedx/cs_comments_service/app.rb:172:in `<top (required)>'
forum_1             |   from config/unicorn_tcp.rb:4:in `require'
forum_1             |   from config/unicorn_tcp.rb:4:in `reload'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:84:in `instance_eval'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:84:in `reload'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/configurator.rb:77:in `initialize'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:77:in `new'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/lib/unicorn/http_server.rb:77:in `initialize'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/bin/unicorn:126:in `new'
forum_1             |   from /openedx/cs_comments_service/vendor/bundle/ruby/2.4.0/gems/unicorn-5.3.1/bin/unicorn:126:in `<top (required)>'
forum_1             |   from ./bin/unicorn:16:in `load'
forum_1             |   from ./bin/unicorn:16:in `<main>'
forum_1             | Waiting for mongodb/elasticsearch...
forum_1             | 2019/12/01 20:30:34 Waiting for: tcp://REPLACED.206:27017
forum_1             | 2019/12/01 20:30:34 Waiting for: http://elasticsearch:9200/content
forum_1             | 2019/12/01 20:30:34 Received 200 from http://elasticsearch:9200/content
forum_1             | 2019/12/01 20:30:34 Connected to tcp://REPLACED.206:27017
5

@sbrewer I don’t quite understand. Are you trying to connect to an external MySQL or MongoDb database? Or both?

The required MogoDb version is 3.2.16 https://github.com/overhangio/tutor/blob/master/tutor/templates/config.yml#L35 (this should probably be in the docs)

The forum logs are not going to tell you much. You need to find the actual error that is causing the LMS container to fail. To do so, run:

tutor local logs --tail=100 -f lms

Let it run for some time and paste the output here.

6

Just wanted to circle back for futures searching forums. I never found any useful logs that could pinpoint why the LMS wasn’t starting up with the errors I posted here.

When I downgraded from MongoDB 4.X to 3.2.16 I was able to get the system to boot up as expected.

Is there a blog site for tutor where we can create supplemental documentation for specific scenarios? I thought others might benefit from my findings when migrating MySQL and Mongo to external systems.

1 Like
7

I guess this should be part of the docs…

It’s great that you are willing to share your knowledge on this, I appreciate it. May I suggest you write a dedicated HOWTO post on these forums? I wrote a couple myself: Search results for 'howto' - Overhang.IO
In the future, as I mentioned elsewhere, we should have dedicated HOWTO articles in the docs.

8

@sbrewer have you written about your findings how to migrating MySQL and Mongo to external systems
if you have can you share the link here

9

I’m using AWS aurora instead of MySQL and I changed the next config variables in $(tutor config printroot)/config.yml
MYSQL_HOST:
MYSQL_ROOT_PASSWORD:

If you already have have data stored in the database you should make backups https://docs.tutor.overhang.io/local.html?highlight=mysql#making-database-dumps