Open edX not starting after disk full event

Hello,

Maybe this helps someone else.

After a disk full situation, “tutor local start -d” did not return any error but trying to access Open edX results in http error code 504 - Gateway Timeout.
I also have noticed high cpu usage on several process, like gunicorn, that did not match the scenario of zero access to Open edX.
To figure out, I did:
tutor local logs --tail=100 -f

there I found this error:

rabbitmq_1          |     application: rabbit
rabbitmq_1          |     exited: {**bad_return**,
rabbitmq_1          |              {{**rabbit,start**,[normal,[]]},
rabbitmq_1          |               {'EXIT',
rabbitmq_1          |                {{badmatch,
rabbitmq_1          |                  {error,
rabbitmq_1          |                   {{{badmatch,
rabbitmq_1          |                      {error,
rabbitmq_1          |                       {**not_a_dets_file**,
rabbitmq_1          |                        "**/var/lib/rabbitmq/mnesia/rabbit@f4a2f1e9af17/recovery.dets**"}}},

tutor was in a loop trying to start rabbitmq, and searching for this error I figured out the solution was to remove the recovery.dets file.

The path is relative to “tutor config printroot”, so I did:
sudo rm .local/share/tutor/data/rabbitmq/mnesia/rabbit@f4a2f1e9af17/recovery.dets

Then tutor could start rabbitmq and everything was back to regular service. :slightly_smiling_face:

[]s,
Guilherme

1 Like