I would like to announce the release of our backup and restore plugin, called tutor-contrib-backup. It provides backup and restore functionality for MySQL, MongoDB, and Caddy services in both local and Kubernetes Tutor deployments.
In a local Tutor deployment, you can run the backup from the command line with the tutor local backup command. This will create a dump of MySQL and MongoDB, and create a copy of the Caddy data directory. The backups are stored as a single tar file in $(tutor config printroot)/env/backup/. You can then copy the Tutor config root folder to a new host and restore your Open edX environment with the restore command: tutor local restore.
In Kubernetes, the plugin runs the backup job as a CronJob by default. You can also run the backup job from the command line. In both cases the backup tar file is stored in an S3 bucket. Then you can use the restore command to restore your Open edX environment. You can even schedule the restore as a CronJob to periodically download the latest backup and restore your environment. This can, for example, be useful if you want to maintain a standby site for disaster recovery purposes.
The plugin is new and experimental, but we are working on it to make it production-ready. Please go ahead and give it a try and let us know what you think
Thanks @thinnguyen for pointing this out. I noticed that the tutor config save step was missing from the docs. It’s added now. @Harryp let us know if running tutor config save before building the image solved the problem.
@foadlind thanks for fixing tutor images build backup. But I got another problem when I restart the server:
lms-worker_1 | storage=get_video_image_storage(),
lms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/edxval/utils.py", line 158, in get_video_image_storage
lms-worker_1 | return get_storage_class(
lms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto.py", line 253, in __init__
lms-worker_1 | check_location(self)
lms-worker_1 | File "/openedx/venv/lib/python3.8/site-packages/storages/utils.py", line 84, in check_location
lms-worker_1 | raise ImproperlyConfigured(
lms-worker_1 | django.core.exceptions.ImproperlyConfigured: S3BotoStorage.location cannot begin with a leading slash. Found '/openedx/media/'. **Use 'openedx/media/' instead.**
Where I can fix this back slash problem. Thank you !
Thank you @fghaas and @foadlind now fixed issues without loading S3 plugin. The problem came from default values aws kyes as @regis mentioned here. I removed keys from OPENEDX_AWS_SECRET_ACCESS_KEY and added them to BACKUP_S3_SECRET_ACCESS_KEY.
One more question, tutor local backup
creates tar file but does not push it into the bucket. Should I install S3 plugin as you mentioned
Currently, the plugin does not upload the backup tar file to S3 when running tutor local. It will only do that when Tutor is running in Kubernetes. In a tutor local deployment, the the tar file is stored in $(tutor config printroot)/env/backup/.
@foadlind thanks for the update. I use AWS-CLI to sync data from instances to s3, and vice-versa. Just required a two-step process. Temporary resolved the issue
⚠️ You are running Tutor as root. This is strongly not recommended. If you are doing this in order to access the Docker daemon, you should instead add your user to the 'docker' group. (see https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user)
docker compose -f /root/.local/share/tutor/env/local/docker-compose.yml -f /root/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /root/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm backup-job sh -e -c 'python restore_services.py --date=2023-12-04 --exclude=caddy'
[+] Creating 4/0
✔ Container tutor_local-caddy-1 Running 0.0s
✔ Container tutor_local-mysql-1 Running 0.0s
✔ Container tutor_local-permissions-1 Created 0.0s
✔ Container tutor_local-mongodb-1 Running 0.0s
[+] Running 1/1
✔ Container tutor_local-permissions-1 Started 0.3s
2023-12-04 07:11:55,005 INFO Extracting archive /data/backup/backup.2023-12-04.tar.xz to /data
2023-12-04 07:11:55,044 INFO Complete. /data is 6545890 bytes.
2023-12-04 07:11:55,044 INFO Restoring MySQL databases on mysql:3306 from /data/mysql_dump.sql
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 3552 (HY000) at line 24: Access to system schema 'mysql' is rejected.
Traceback (most recent call last):
File "restore_services.py", line 258, in <module>
main()
File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/s3/venv/lib/python3.8/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "restore_services.py", line 250, in main
restore_mysql()
File "restore_services.py", line 50, in restore_mysql
check_call(mysql_cmd,
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'mysql --host=mysql --port=3306 --user=root --password=uTGAkVTr' returned non-zero exit status 1.
Error: Command failed with status 1: docker compose -f /root/.local/share/tutor/env/local/docker-compose.yml -f /root/.local/share/tutor/env/local/docker-compose.prod.yml --project-name tutor_local -f /root/.local/share/tutor/env/local/docker-compose.jobs.yml run --rm backup-job sh -e -c python restore_services.py --date=2023-12-04 --exclude=caddy
