Hi,
I am getting this error when deploying tutor with k8s:
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://minio.lms.example.org/openedx/badges/badges/honor.png"
Below is the quickstart output:
tutor k8s quickstart
==================================================
Interactive platform configuration
==================================================
Are you configuring a production platform? Type 'n' if you are just testing Tutor on your local computer [Y/n] Y
Your website domain name for students (LMS) [] lms.example.org
Your website domain name for teachers (CMS) [] cms.example.org
Your platform name/title [] Test
Your public contact email address [] test@example.org
The default language code for the platform [en]
Activate SSL/TLS certificates for HTTPS access? Important note: this will NOT work in a development environment. [Y/n] Y
Configuration saved to /opt/cloudadm/.local/share/tutor/config.yml
================================================
Updating the current environment
================================================
Environment generated in /opt/cloudadm/.local/share/tutor/env
=====================================
Starting the platform
=====================================
kubectl apply --kustomize /opt/cloudadm/.local/share/tutor/env --wait --selector app.kubernetes.io/component=namespace
namespace/openedx unchanged
kubectl apply --kustomize /opt/cloudadm/.local/share/tutor/env --wait --selector app.kubernetes.io/component=volume
persistentvolumeclaim/caddy unchanged
persistentvolumeclaim/elasticsearch unchanged
persistentvolumeclaim/minio unchanged
persistentvolumeclaim/mongodb unchanged
persistentvolumeclaim/mysql unchanged
persistentvolumeclaim/redis unchanged
kubectl apply --kustomize /opt/cloudadm/.local/share/tutor/env --selector app.kubernetes.io/component!=job
namespace/openedx unchanged
configmap/caddy-config-5m44g89c8h unchanged
configmap/nginx-config-c5kb726hdm unchanged
configmap/openedx-config-cm78mt2gtc unchanged
configmap/openedx-settings-cms-572gc98dh7 unchanged
configmap/openedx-settings-lms-54bk856tkk unchanged
configmap/redis-config-5m6b9th9d6 unchanged
service/caddy unchanged
service/cms unchanged
service/elasticsearch unchanged
service/forum unchanged
service/lms unchanged
service/minio unchanged
service/mongodb unchanged
service/mysql unchanged
service/nginx unchanged
service/redis unchanged
service/smtp unchanged
deployment.apps/caddy unchanged
deployment.apps/cms-worker unchanged
deployment.apps/cms unchanged
deployment.apps/elasticsearch unchanged
deployment.apps/forum configured
deployment.apps/lms-worker unchanged
deployment.apps/lms unchanged
deployment.apps/minio unchanged
deployment.apps/mongodb unchanged
deployment.apps/mysql unchanged
deployment.apps/nginx unchanged
deployment.apps/redis unchanged
deployment.apps/smtp unchanged
persistentvolumeclaim/caddy unchanged
persistentvolumeclaim/elasticsearch unchanged
persistentvolumeclaim/minio unchanged
persistentvolumeclaim/mongodb unchanged
persistentvolumeclaim/mysql unchanged
persistentvolumeclaim/redis unchanged
================================================
Database creation and migrations
================================================
Waiting for a mysql pod to be ready...
pod/mysql-55fcfdcb4b-wnh2m condition met
pod/elasticsearch-5bbdc6f7f-d9dh9 condition met
pod/mongodb-845c7b9bb9-74r6v condition met
job.batch/mysql-job-20210611120611 created
Job mysql-job-20210611120611 successful.
Plugin minio: running pre-init for service minio...
app.kubernetes.io/name=minio-job-20210611120637
job.batch/minio-job-20210611120637 created
Job minio-job-20210611120637 is running.
Job minio-job-20210611120637 successful.
Initialising lms...
job.batch/lms-job-20210611120643 created
Job lms-job-20210611120643 is running.
Waiting for job completion...
Error: Job lms-job-20210611120643 failed. View the job logs to debug this issue.
And the loks for the failed pod:
2021/06/11 12:06:47 Waiting for: tcp://mysql:3306
2021/06/11 12:06:47 Connected to tcp://mysql:3306
Loading settings lms.envs.tutor.production
2021-06-11 12:06:58,521 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (1): minio.lms02.fedcloud-tf.fedcloud.eu
2021-06-11 12:06:59,590 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (2): minio.lms02.fedcloud-tf.fedcloud.eu
2021-06-11 12:07:01,176 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (3): minio.lms02.fedcloud-tf.fedcloud.eu
2021-06-11 12:07:03,103 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (4): minio.lms02.fedcloud-tf.fedcloud.eu
2021-06-11 12:07:08,480 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (5): minio.lms02.fedcloud-tf.fedcloud.eu
Operations to perform:
Apply all migrations: admin, announcements, api_admin, assessment, auth, badges, blackboard, block_structure, bookmarks, branding, bulk_email, bulk_grades, calendar_sync, canvas, catalog, celery_utils, certifi
cates, commerce, completion, consent, content_libraries, content_type_gating, contentserver, contenttypes, cornerstone, cors_csrf, course_action_state, course_date_signals, course_duration_limits, course_goals,
course_groups, course_modes, course_overviews, courseware, crawlers, credentials, credit, dark_lang, database_fixups, degreed, demographics, discounts, django_celery_results, django_comment_common, django_notify
, edx_proctoring, edx_when, edxval, email_marketing, embargo, enterprise, entitlements, experiments, external_user_ids, grades, instructor_task, integrated_channel, learning_sequences, lms_xblock, lti_consumer,
milestones, mobile_api, moodle, oauth2_provider, oauth_dispatch, organizations, program_enrollments, programs, redirects, rss_proxy, sap_success_factors, schedules, self_paced, sessions, shoppingcart, site_confi
guration, sites, social_django, splash, static_replace, status, student, submissions, super_csv, survey, system_wide_roles, teams, theming, third_party_auth, thumbnail, track, user_api, user_authn, util, verifie
d_track_content, verify_student, video_config, video_pipeline, waffle, waffle_utils, wiki, workflow, xapi, xblock_django
Running migrations:
Traceback (most recent call last): [56/1158]
File "./manage.py", line 123, in <module>
execute_from_command_line([sys.argv[0]] + django_args)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/openedx/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
post_migrate_state = executor.migrate(
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
state = migration.apply(state, schema_editor)
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 121, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
self.code(from_state.apps, schema_editor)
File "/openedx/edx-platform/lms/djangoapps/certificates/migrations/0003_data__default_modes.py", line 23, in forwards
conf.icon.save(
File "/openedx/venv/lib/python3.8/site-packages/django/db/models/fields/files.py", line 88, in save
self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "/openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
return self._save(name, content)
File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
self._save_content(obj, content, parameters=parameters)
File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
obj.upload_fileobj(content, ExtraArgs=put_parameters)
File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj
return self.meta.client.upload_fileobj(
File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
return future.result()
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
raise self._exception
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/openedx/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 601, in _make_api_call
http, parsed_response = self._endpoint.make_request(
File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 143, in make_request
return self._send_request(request_dict, operation_model)
File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 171, in _send_request
while self._needs_retry(attempts, operation_model, request_dict,
File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 262, in _needs_retry
responses = self._event_emitter.emit(
File "/openedx/venv/lib/python3.8/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/openedx/venv/lib/python3.8/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 250, in __call__
should_retry = self._should_retry(attempt_number, response,
File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 316, in __call__
checker_response = checker(attempt_number, response,
File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 222, in __call__
return self._check_caught_exception(
File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://minio.lms.example.org/openedx/badges/badges/honor.png"
Applying certificates.0003_data__default_modes...
Any ideas on where to start troubleshooting?
Many thanks,
Sebastian
