Migrating from Native Install to Tutor (juniper -> tutor 10.x)

misilot · May 5, 2021, 9:43pm

See the solution for the steps I used, you can view the edit history for my first not so great attempt

Hello,

I am working on trying to convert from a native install of Juniper to a Tutor (v3.12.6) based install.

The steps I did were:

So not sure where else to go from here, as I can’t login with an existing user, and I can’t seem to trigger migrations from Juniper to Koa? Is there a way to install a Juniper version of Koa and try doing the upgrade that way?

misilot · May 6, 2021, 2:14am

Ok, I got a little closer. Realized I was restoring to edxapp and not openedx. So I switched that and now getting the following. More or less migrations being applied out of order?? Any ideas on either a way to force the migrations or ignore the error?

django.db.migrations.exceptions.InconsistentMigrationHistory: Migration third_party_auth.0001_initial is applied before its dependency oauth2.0001_initial on database 'default'.

Full output below:

2021/05/06 02:09:08 Waiting for: tcp://mysql:3306
2021/05/06 02:09:08 Connected to tcp://mysql:3306
Loading settings lms.envs.tutor.production
WARNING:py.warnings:/openedx/edx-platform/lms/djangoapps/courseware/__init__.py:5: DeprecationWarning: Importing 'lms.djangoapps.courseware' as 'courseware' is no longer supported
  warnings.warn("Importing 'lms.djangoapps.courseware' as 'courseware' is no longer supported", DeprecationWarning)

2021-05-06 02:09:33,664 WARNING 13 [enterprise.utils] utils.py:50 - Could not import Registry from third_party_auth.provider
2021-05-06 02:09:33,674 WARNING 13 [enterprise.utils] utils.py:51 - cannot import name _LTI_BACKENDS
Traceback (most recent call last):
  File "./manage.py", line 123, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 86, in handle
    executor.loader.check_consistent_history(connection)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 299, in check_consistent_history
    connection.alias,
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration third_party_auth.0001_initial is applied before its dependency oauth2.0001_initial on database 'default'.
Error: Command failed with status 1: docker-compose -f /klib/data/nla-openedx/env/local/docker-compose.yml --project-name tutor_local -f /klib/data/nla-openedx/env/local/docker-compose.jobs.yml run --rm lms-job sh -e -c dockerize -wait tcp://mysql:3306 -timeout 20s

export DJANGO_SETTINGS_MODULE=$SERVICE_VARIANT.envs.$SETTINGS
echo "Loading settings $DJANGO_SETTINGS_MODULE"

./manage.py lms migrate

./manage.py lms create_oauth2_client \
    "http://androidapp.com" "http://androidapp.com/redirect" public \
    --client_id android --client_secret ypCaz7VXZxX9L6RaOvWEUZcy \
    --trusted

# Fix incorrect uploaded file path
if [ -d /openedx/data/uploads/ ]; then
  if [ -n "$(ls -A /openedx/data/uploads/)" ]; then
    echo "Migrating LMS uploaded files to shared directory"
    mv /openedx/data/uploads/* /openedx/media/
    rm -rf /openedx/data/uploads/
  fi
fi

Tommy · May 6, 2021, 4:53am

I think you need install Tutor lates version
And then you migrate content manully

regis · May 6, 2021, 7:48am

@misilot I suggest you do not attempt to both switch to Tutor and upgrade to Koa in one go. Instead, first switch to tutor v10 (juniper). Then, if everything is working fine, upgrade to Koa.

misilot · May 6, 2021, 11:35am

Thanks @regis, I will try that. I think what happened is I tried adding tutor-figures which downgraded tutor to version 3.12.6. Removing that allows me to install v10 and v11.

misilot · May 6, 2021, 12:14pm

@regis, is mongodb used in v10 of tutor? I was looking to copy over my mongodb database, but when I look at mongo I only see

show dbs
admin   0.000GB
config  0.000GB
local   0.000GB

and no edxapp or openedx database? Just trying to figure out if I need to copy my mongodb data to tutor, or not and what the database should be called. Thanks!

regis · May 6, 2021, 12:31pm

Yes, as you can see in the generated docker-compose.yml file:

$ grep -i mongodb $(tutor config printroot)/env/local/docker-compose.yml 
  mongodb:
      - ../../data/mongodb:/data/db
      MONGODB_AUTH: ""
      MONGODB_HOST: "mongodb"
      MONGODB_PORT: "27017"
    depends_on: ["elasticsearch", "mongodb"]
      - mongodb
      - mongodb

The Mongodb database name is given by the MONGODB_DATABASE configuration entry: Configuration and customisation — Tutor documentation

misilot · May 6, 2021, 12:59pm

I knew it was installed, but didn’t see any databases (other than the 3 listed). I will copy the data into the database openedx (the default value listed).

Thanks!

misilot · May 12, 2021, 2:27pm

Updated steps (new solution, missed a couple of steps)

Here are the steps I used to migrate from NativeInstall (Juniper) to Tutor

Install tutor via pip in a venv.
1.1 pip install tutor-openedx==v10.5.3
Run tutor local quickstart
wait for it to finish
Have export of mongodb and mysql database ready from existing install ready
4.1 Edit your mysql dump file:
4.1.1 Add DROP DATABASE openedx; to the sql file above the CREATE DATABASE line
4.1.2 Change CREATE DATABASE /*!32312 IF NOT EXISTS*/ edxapp /*!40100 DEFAULT CHARACTER SET utf8 */; to CREATE DATABASE /*!32312 IF NOT EXISTS*/ openedx /*!40100 DEFAULT CHARACTER SET utf8 */;
4.1.3 Change USE edxapp; to USE openedx;
Restored the mysql database
5.1 docker exec -i tutor_local_mysql_1 sh -c 'exec mysql -uroot -p"PASSWORD"' < /home/user/mysql-data-20210504T110001.sql
restore the mongodb (I didn’t have openedx mongodb after starting so just putting my edxapp db in place there)
6.1 Copy mongodb backup to $TUTOR_ROOT/data/mongodb
6.2 Run tutor local exec mongodb bash
6.3 Run mongorestore -d openedx /data/db/edxapp/
6.4 rm -r /data/db/edxapp
6.5 Exit mongodb container
Add the following here to successfully build the Juniper image.
tutor images build openedx
Copy Media from native install to tutor
9.1 copy /edx/var/edxapp/media to $TUTOR_ROOT/data/openedx-media
tutor local quickstart
Wait for it to come up and login with your existing credentials
If your site url changes login to the admin and update the Site URL so settings will match / SAML settings as well if applicable

Upgrading to Latest Release of Tutor

pip install --upgrade tutor-openedx
tutor images build openedx
tutor local upgrade (needed to run for mysql upgrade)
tutor local quickstart
Wait a while (got a 500 error while waiting for services to eventually come up)

Plugins I had to add

Enabling LTI

name: enable_lti
version: 0.1.0
patches:
 common-env-features: |
    "ENABLE_LTI_PROVIDER": true

SAML (Juniper)

name: saml
version: 0.1.0
patches:
 common-env-features: |
    "ENABLE_THIRD_PARTY_AUTH" : true

 openedx-lms-common-settings: |
    # saml special settings
    THIRD_PARTY_AUTH_BACKENDS = ["third_party_auth.saml.SAMLAuthBackend"]

 openedx-auth: |
    "SOCIAL_AUTH_SAML_SP_PRIVATE_KEY" : "{{ SAML_PRIVATE_KEY }}",
    "SOCIAL_AUTH_SAML_SP_PUBLIC_CERT" : "{{ SAML_PUBLIC_CERT }}"

SAML (Koa)

name: saml
version: 0.2.0
patches:
 common-env-features: |
    "ENABLE_THIRD_PARTY_AUTH" : true

 openedx-lms-common-settings: |
    # saml special settings
    THIRD_PARTY_AUTH_BACKENDS = ["common.djangoapps.third_party_auth.saml.SAMLAuthBackend"]


 openedx-auth: |
    "SOCIAL_AUTH_SAML_SP_PRIVATE_KEY" : "{{ SAML_PRIVATE_KEY }}",
    "SOCIAL_AUTH_SAML_SP_PUBLIC_CERT" : "{{ SAML_PUBLIC_CERT }}"

config.yml

I added SAML_PRIVATE_KEY and SAML_PUBLIC_CERT

regis · May 21, 2021, 1:54pm

Hey @misilot! This is tremendously useful. It’s great to see a detailed description to migrate from the native installation to Tutor. It’s highly likely that we copy your post and add it to the docs, somewhere, at some point.

I believe you could not edit your own post because you did not have the prerequisite trust level. I adjusted it and you should have the right permissions. Please be free to edit and add as much information as needed. The more the better! this will save us a lot of work down the road.

misilot · June 9, 2021, 4:26pm

Thanks, I will edit and clean up my posts!

Found another area we needed to copy content from (assets uploaded, such as scorm content)

@regis, it looks like I can only the last post, not the older ones. I don’t think it will be a big deal, I will edit the one marked as solution while I can, and make new posts and mark that as the solution as needed.

taufanbudiman · July 23, 2021, 4:46am

Hello, @misilot
I followed this step to migrate, but in step 9.1 I didn’t find openedx-media folder. do i have to create a new folder?

I use ironwood not juniper

misilot · September 22, 2021, 2:35pm

Hi, sorry for the late reply. It is possible that the folder doesn’t exist? It did for us since I think part of that might be tied to xblocks.

system · December 21, 2021, 2:35pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.