Nginx error when restoring backup

Hi,
I am trying to test the backup and restore process following the documentation.

Apologies on the long post, I kept adding to it as I tried to resolve.

When trying to restore a back up nginx keeps producing an error due to a letsencrypt file not being found

nginx_1 | nginx: [emerg] BIO_new_file(“/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem’,‘r’) error:2006D080:BIO routines:BIO_new_file:no such file)
nginx_1 | 2020/10/09 13:51:20 [emerg] 1#1: BIO_new_file(“/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem’,‘r’) error:2006D080:BIO routines:BIO_new_file:no such file)

tutor version 10.2.0.
$TUTOR_ROOT = /usr/local/share/tutor

What am I doing wrong?
What was unnessary?
What should I never do again?

I have the logs for different stages saved to file, if needed

Backed up tutor:

tutor local stop
sudo rsync -avr “$(tutor config printroot)”/ /tmp/tutor-backup/
tutor local start -d

It took about 20mins for the system to get up and running properly, receiving 504 Gateway timeout within that time. Is that normal?

I then created a new User and new Course, with the intention of testing if they were removed when restore the created backup

I then tried to restore the backup

tutor local stop
sudo rsync -avr /tmp/tutor-backup/ “$(tutor config printroot)”/
tutor local start -d

But even after 20min, was not able to connect to server with error in logs

nginx_1 | 2020/10/09 11:32:41 [emerg] 1#1: BIO_new_file(“/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem’,‘r’) error:2006D080:BIO routines:BIO_new_file:no such file)

tried renewing the certificate

tutor local stop nginx
tutor local https renew
tutr local start -d

But no change

So tried building nginx image

tutor images build nginx

…but it didn’t give an output so I presumed it did nothing and built all (which I’d never tried before, but saw it referenced in the forum not the docs)

tutor images build all

but got error

The command ‘/bin/sh -c bundle install --deployment’ returned a non-zero code: 7
Error: Command failed with status 7: docker build -t docker.io/overhangio/openedx-forum:10.2.0 /usr/local/share/tutor/env/build/forum

So just built openedx and quickstart

tutor images build openedx
tutor local quickstart

Same issue with nginx

nginx_1 | 2020/10/09 12:33:29 [emerg] 1#1: BIO_new_file(“/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem’,‘r’) error:2006D080:BIO routines:BIO_new_file:no such file)
nginx_1 | nginx: [emerg] BIO_new_file(“/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem’,‘r’) error:2006D080:BIO routines:BIO_new_file:no such file)

So tried removing all docker containers etc, rebuild image remove all files in $TUTOR_ROOT, transfer in backup and quickstart, while also deleting rabbitmq and letsencrypt data

docker system prune -a
tutor images build openedx
sudo rm -r “$(tutor config printroot)”/
sudo mkdir “$(tutor config printroot)”
sudo rsync -avr /tmp/tutor-backup/ “$(tutor config printroot)”/
sudo chmod -R 777 “$(tutor config printroot)”
sudo rm -r “$(tutor config printroot)”/data/rabbitmq
sudo rm -r “$(tutor config printroot)”/data/letsencrypt

Same nginx issue

tutor local stop
sudo rm -r “$(tutor config printroot)”
sudo mkdir tutor
sudo rsync -avr /tmp/tutor-backup/ “$(tutor config printroot)”/
sudo rm -r “$(tutor config printroot)”/data/letsencrypt
sudo rm -r “$(tutor config printroot)”/data/rabbitmq
sudo chmod -R 777 “$(tutor config printroot)”
tutor local quickstart

Same nginx error

nginx_1 | nginx: [emerg] BIO_new_file(“/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem’,‘r’) error:2006D080:BIO routines:BIO_new_file:no such file)
nginx_1 | 2020/10/09 13:51:16 [emerg] 1#1: BIO_new_file(“/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/etc/letsencrypt/live/icip3-learn.ichec.ie/fullchain.pem’,‘r’) error:2006D080:BIO routines:BIO_new_file:no such file)

But this works fine …

Before all the above I simply copied the the working tutor directory

sudo cp -r tutor tutor_backup

Then after all the above testing and trying to use rsync, I deleted the not workng tutor directory and copyied across the backup and it all worked fine

tutor local stop
sudo rm -r “$(tutor config printroot)”
sudo cp -r tutor_backup tutor
sudo chmod -R 777 “$(tutor config printroot)”
tutor build openedx
tutor local quickstart

Though I did have to delete rabbitmq data and restart the service.

tutor local stop rabbitmq
sudo rm -r “$(tutor config printroot)”/data/rabbitmq
tutor local start rabbitmq -d

Is there a reason why not to back up as done here?

I plan to back up every night by compressing the ‘tutor’ or the copied ‘tutor_backup’ folder as a tar.gz and then transferring to Google Drive using rclone.

Any insights into the errors of my ways would be appreciated.

By default Tutor puts the certs in $(tutor config printroot)/data/letsencrypt rather than /etc/letsencrypt. You can see this in the docker-compose.prod.yml if you have https turned on.
env/local/docker-compose.prod.yml: - ../../data/letsencrypt:/etc/letsencrypt/:ro. So that error means that the volume didn’t mount correctly. This volume line is absent from the docker-compose.prod.yml if you don’t have HTTPS turned on in the config.yml file.

So first I would check to make sure that after you perform your regular backup that the data/ directory has everything that you would expect, and that permissions are as you would expect. Also check your config.yml to make sure https is turned on.

But the root issue I believe is permissions related. There is occasionally an issue with data/rabbitmq/.erlang.cookie where the permissions get messed up. They should be 600, otherwise rabbitmq gets mad.

Thanks @iammcgaber I will try and test your recommendations next.

As it stands my method of backing up to a tar.gz seems to work just fine, but I am keen to understand the underlying issue.