botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL

Hi,

I am getting this error when deploying tutor with k8s:

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://minio.lms.example.org/openedx/badges/badges/honor.png"

Below is the quickstart output:

tutor k8s quickstart                                                                                                                                                                        
==================================================                                                                                                                                                                 
        Interactive platform configuration                                                                                                                                                                         
==================================================                                                                                                                                                                 
Are you configuring a production platform? Type 'n' if you are just testing Tutor on your local computer [Y/n] Y                                                                                                   
Your website domain name for students (LMS) [] lms.example.org
Your website domain name for teachers (CMS) [] cms.example.org
Your platform name/title [] Test
Your public contact email address [] test@example.org
The default language code for the platform [en]                                                                                                                                                                    
Activate SSL/TLS certificates for HTTPS access? Important note: this will NOT work in a development environment. [Y/n] Y                                                                                           
Configuration saved to /opt/cloudadm/.local/share/tutor/config.yml                                                                                                                                                 
================================================                                                                                                                                                                   
        Updating the current environment                                                                                                                                                                           
================================================                                                                                                                                                                   
Environment generated in /opt/cloudadm/.local/share/tutor/env                                                                                                                                                      
=====================================                                                                                                                                                                              
        Starting the platform                                                                                                                                                                                      
=====================================                                                                                                                                                                              
kubectl apply --kustomize /opt/cloudadm/.local/share/tutor/env --wait --selector app.kubernetes.io/component=namespace                                                                                             
namespace/openedx unchanged                                                                                                                                                                                        
kubectl apply --kustomize /opt/cloudadm/.local/share/tutor/env --wait --selector app.kubernetes.io/component=volume                                                                                                
persistentvolumeclaim/caddy unchanged                                                                                                                                                                              
persistentvolumeclaim/elasticsearch unchanged                                                                                                                                                                      
persistentvolumeclaim/minio unchanged                                                                                                                                                                              
persistentvolumeclaim/mongodb unchanged                                                                                                                                                                            
persistentvolumeclaim/mysql unchanged                                                                                                                                                                              
persistentvolumeclaim/redis unchanged                                                                                                                                                                              
kubectl apply --kustomize /opt/cloudadm/.local/share/tutor/env --selector app.kubernetes.io/component!=job                                                                                                         
namespace/openedx unchanged                                                                                                                                                                                        
configmap/caddy-config-5m44g89c8h unchanged                                                                                                                                                                        
configmap/nginx-config-c5kb726hdm unchanged                                                                                                                                                                        
configmap/openedx-config-cm78mt2gtc unchanged                                                                                                                                                                      
configmap/openedx-settings-cms-572gc98dh7 unchanged                                                                                                                                                                
configmap/openedx-settings-lms-54bk856tkk unchanged                                                                                                                                                                
configmap/redis-config-5m6b9th9d6 unchanged                                                                                                                                                                        
service/caddy unchanged                                                                                                                                                                                            
service/cms unchanged                                                                                                                                                                                              
service/elasticsearch unchanged                                                                                                                                                                                    
service/forum unchanged                                                                                                                                                                                            
service/lms unchanged                                                                                                                                                                                              
service/minio unchanged                                                                                                                                                                                            
service/mongodb unchanged                                                                                                                                                                                          
service/mysql unchanged                                                                                                                                                                                            
service/nginx unchanged                                                                                                                                                                                            
service/redis unchanged                                
service/smtp unchanged
deployment.apps/caddy unchanged
deployment.apps/cms-worker unchanged
deployment.apps/cms unchanged
deployment.apps/elasticsearch unchanged
deployment.apps/forum configured
deployment.apps/lms-worker unchanged
deployment.apps/lms unchanged
deployment.apps/minio unchanged
deployment.apps/mongodb unchanged
deployment.apps/mysql unchanged
deployment.apps/nginx unchanged
deployment.apps/redis unchanged
deployment.apps/smtp unchanged
persistentvolumeclaim/caddy unchanged
persistentvolumeclaim/elasticsearch unchanged
persistentvolumeclaim/minio unchanged
persistentvolumeclaim/mongodb unchanged
persistentvolumeclaim/mysql unchanged
persistentvolumeclaim/redis unchanged
================================================
        Database creation and migrations
================================================
Waiting for a mysql pod to be ready...
pod/mysql-55fcfdcb4b-wnh2m condition met
pod/elasticsearch-5bbdc6f7f-d9dh9 condition met
pod/mongodb-845c7b9bb9-74r6v condition met
job.batch/mysql-job-20210611120611 created
Job mysql-job-20210611120611 successful.
Plugin minio: running pre-init for service minio...
app.kubernetes.io/name=minio-job-20210611120637
job.batch/minio-job-20210611120637 created
Job minio-job-20210611120637 is running.
Job minio-job-20210611120637 successful.
Initialising lms...
job.batch/lms-job-20210611120643 created
Job lms-job-20210611120643 is running.
Waiting for job completion...
Error: Job lms-job-20210611120643 failed. View the job logs to debug this issue.

And the loks for the failed pod:

2021/06/11 12:06:47 Waiting for: tcp://mysql:3306                                                        
2021/06/11 12:06:47 Connected to tcp://mysql:3306                                                                                                                                                                  
Loading settings lms.envs.tutor.production                                                               
2021-06-11 12:06:58,521 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (1): minio.lms02.fedcloud-tf.fedcloud.eu
2021-06-11 12:06:59,590 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (2): minio.lms02.fedcloud-tf.fedcloud.eu
2021-06-11 12:07:01,176 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (3): minio.lms02.fedcloud-tf.fedcloud.eu
2021-06-11 12:07:03,103 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (4): minio.lms02.fedcloud-tf.fedcloud.eu
2021-06-11 12:07:08,480 INFO 13 [botocore.vendored.requests.packages.urllib3.connectionpool] [user None] [ip None] connectionpool.py:734 - Starting new HTTPS connection (5): minio.lms02.fedcloud-tf.fedcloud.eu
Operations to perform:                                                                                   
  Apply all migrations: admin, announcements, api_admin, assessment, auth, badges, blackboard, block_structure, bookmarks, branding, bulk_email, bulk_grades, calendar_sync, canvas, catalog, celery_utils, certifi
cates, commerce, completion, consent, content_libraries, content_type_gating, contentserver, contenttypes, cornerstone, cors_csrf, course_action_state, course_date_signals, course_duration_limits, course_goals, 
course_groups, course_modes, course_overviews, courseware, crawlers, credentials, credit, dark_lang, database_fixups, degreed, demographics, discounts, django_celery_results, django_comment_common, django_notify
, edx_proctoring, edx_when, edxval, email_marketing, embargo, enterprise, entitlements, experiments, external_user_ids, grades, instructor_task, integrated_channel, learning_sequences, lms_xblock, lti_consumer, 
milestones, mobile_api, moodle, oauth2_provider, oauth_dispatch, organizations, program_enrollments, programs, redirects, rss_proxy, sap_success_factors, schedules, self_paced, sessions, shoppingcart, site_confi
guration, sites, social_django, splash, static_replace, status, student, submissions, super_csv, survey, system_wide_roles, teams, theming, third_party_auth, thumbnail, track, user_api, user_authn, util, verifie
d_track_content, verify_student, video_config, video_pipeline, waffle, waffle_utils, wiki, workflow, xapi, xblock_django
Running migrations:                                                                                      
Traceback (most recent call last):                                                                                                                                                                        [56/1158]
  File "./manage.py", line 123, in <module>                                                              
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
    post_migrate_state = executor.migrate(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate 
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 121, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/openedx/venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/openedx/edx-platform/lms/djangoapps/certificates/migrations/0003_data__default_modes.py", line 23, in forwards
    conf.icon.save(
  File "/openedx/venv/lib/python3.8/site-packages/django/db/models/fields/files.py", line 88, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/openedx/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 495, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/openedx/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 510, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 511, in object_upload_fileobj 
    return self.meta.client.upload_fileobj(
  File "/openedx/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/client.py", line 601, in _make_api_call
    http, parsed_response = self._endpoint.make_request(
  File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 143, in make_request
    return self._send_request(request_dict, operation_model)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 171, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/openedx/venv/lib/python3.8/site-packages/botocore/endpoint.py", line 262, in _needs_retry
    responses = self._event_emitter.emit(
  File "/openedx/venv/lib/python3.8/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/openedx/venv/lib/python3.8/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://minio.lms.example.org/openedx/badges/badges/honor.png"
  Applying certificates.0003_data__default_modes...

Any ideas on where to start troubleshooting?

Many thanks,
Sebastian

Is the minio deployment up and available? You can check by opening the following url in your browser:
https://minio.lms.example.org/

Hi,

It’s not but due to a DNS resolution issue.

Inside the k8s cluster I get:

curl minio.lms.example.org
curl: (6) Could not resolve host: minio.lms.example.org

Externally I have defined two DNS records pointing to the same public IP:

- https://lms.example.org/
- https://cms.example.org/

Is this due to the fact that I haven’t defined the CNAME DNS record in the docs? Install Tutor — Tutor documentation

Or am I just confused?

Many thanks,
Sebastian

Yes, exactly. You need to define a DNS record that matches minio.lms.example.org in order for minio to work properly.

Thanks @regis

At the moment I have only created an entry in /etc/hosts/ to point to minio.lms.example.org.

Now I get

curl http://minio.lms.example.org/openedx/badges/badges/honor.png

<html>                                                                                                   
<head><title>404 Not Found</title></head>
<body>                                                                                                   
<center><h1>404 Not Found</h1></center>                                                                  
<hr><center>nginx/1.19.1</center>                                                                        
</body>                                                                                                  
</html>                                   

Is this expected?

Here is additional output:

kubectl get all -n openedx                                                             

NAME                                 READY   STATUS             RESTARTS   AGE
pod/caddy-7f7f698f88-829tk           1/1     Running            0          24h
pod/cms-85ffbdfdf9-n4kzt             1/1     Running            0          24h
pod/cms-worker-66c664fc58-gj24j      1/1     Running            0          24h
pod/elasticsearch-5bbdc6f7f-xz2n7    1/1     Running            0          24h
pod/forum-854dfbd47c-vxl49           0/1     CrashLoopBackOff   288        24h
pod/lms-5f956f5965-rrtj4             1/1     Running            0          24h
pod/lms-job-20210615074239-c9jgm     0/1     Error              0          24h
pod/lms-job-20210615074239-gm7cn     0/1     Error              0          24h
pod/lms-worker-69bbf7cb79-bxndc      1/1     Running            0          24h
pod/minio-654ddf7bbc-t8g7z           1/1     Running            0          24h
pod/minio-job-20210615074234-qvk2d   0/1     Completed          0          24h
pod/mongodb-845c7b9bb9-4k8mz         1/1     Running            0          24h
pod/mysql-55fcfdcb4b-rvflp           1/1     Running            0          24h
pod/mysql-job-20210615074202-g6k2b   0/1     Completed          0          24h
pod/nginx-7656559ff7-rzxdq           1/1     Running            0          24h
pod/redis-86bf9886bb-n8j5d           1/1     Running            0          24h
pod/smtp-7bf8796b8b-bt97l            1/1     Running            0          24h

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/caddy           LoadBalancer   10.103.226.208   <pending>     80:20125/TCP,443:17622/TCP   24h
service/cms             NodePort       10.103.111.187   <none>        8000:1682/TCP                24h
service/elasticsearch   NodePort       10.105.107.204   <none>        9200:1848/TCP                24h
service/forum           NodePort       10.98.185.59     <none>        4567:6501/TCP                24h
service/lms             NodePort       10.98.114.18     <none>        8000:21320/TCP               24h
service/minio           NodePort       10.106.34.202    <none>        9000:6375/TCP                24h
service/mongodb         NodePort       10.99.16.108     <none>        27017:34536/TCP              24h
service/mysql           NodePort       10.105.56.149    <none>        3306:13633/TCP               24h
service/nginx           NodePort       10.108.5.230     <none>        80:19123/TCP                 24h
service/redis           NodePort       10.101.176.193   <none>        6379:11941/TCP               24h
service/smtp            NodePort       10.103.227.82    <none>        25:27551/TCP                 24h

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/caddy           1/1     1            1           24h
deployment.apps/cms             1/1     1            1           24h
deployment.apps/cms-worker      1/1     1            1           24h
deployment.apps/elasticsearch   1/1     1            1           24h
deployment.apps/forum           0/1     1            0           24h
deployment.apps/lms             1/1     1            1           24h
deployment.apps/lms-worker      1/1     1            1           24h
deployment.apps/minio           1/1     1            1           24h
deployment.apps/mongodb         1/1     1            1           24h
deployment.apps/mysql           1/1     1            1           24h
deployment.apps/nginx           1/1     1            1           24h
deployment.apps/redis           1/1     1            1           24h
deployment.apps/smtp            1/1     1            1           24h

The log for pod/lms-job-20210615074239-gm7cn is the same as shown in my previous post:

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://minio.lms.example.org/openedx/badges/badges/honor.png"

I am deploying tutor on a cloud without Load Balancer. That’s why I get:

service/caddy           LoadBalancer   10.103.226.208   <pending>     80:20125/TCP,443:17622/TCP   24h

Is this the root cause of the issue?

This second deployment attempt with tutor k8s quickstart I answered no to the question:

Activate SSL/TLS certificates for HTTPS access? Important note: this will NOT work in a development environment.

Best regards,
Sebastian

It looks like traffic is not properly redirected traffic to your minio container. If caddy is not running, something, somehow must be responsible for redirecting traffic. Otherwise, there is no way for either curl or the lms to make requests to the minio service.

Thanks @regis

Is the use of the LoadBalancer service type mandatory for caddy?

I don’t have a load balancer available on the cloud infrastructure where testing this deployment, is it possible to deploy Open edX using tutor without load balancer?

Not having load balancers is a common scenario in the academic clouds that I have been playing with so far.

Best regards,
Sebastian

Best way I think is a wildcard * CNAME record in DNS, so you only have to create a Letsencrypt certificate for ‘yourdomain.com’ and ‘*.yourdomain.com’. Besides you’ll avoid a lot of scam bots hitting your other endpoints as ‘studio’, ‘preview’ etc. :grin:

I don’t know what we could use to replace the LoadBalancer.

So what do you use instead? Note that it’s quite possible for you to just disable Caddy entirely (RUN_CADDY=false) and then deploy your own web proxy.

Hi,

Finally the solution for me was the following:

  • Disable caddy
  • Disable minio

I replaced caddy with an nginx ingress controller that I could customize for my deployment. I can share the details if anyone is interested.

I think the reason that caddy wasn’t working properly is what prevented minio from being contacted with the quickstart. Provided that my k8s cluster comes with a default storage class with an NFS server, I also disabled minio.

Best regards,
Sebastian

I need nginx ingress controller

Hi,

After following instructions to install the nginx ingress controller:
https://kubernetes.github.io/ingress-nginx/deploy/

My steps were:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --cleanup-on-fail --install --create-namespace --namespace ingress-nginx ingress-nginx ingress-nginx/ingress-nginx --values ingress.yaml

with

controller:
  tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
  service:
    type: NodePort # I don't have a load balancer on my cloud provider
    externalIPs:
      - n.n.n.n # update with yours
defaultBackend:
  tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master

Finally, kubectl apply -f expose-ingress-tls.yaml with expose-ingress-tls.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-test
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: openedx
spec:
  rules:   
    - host: <lms.fqdn>
      http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80
            path: /
    - host: <studio.fqdn>
      http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80
            path: /
    - host: <preview.fqdn>
      http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80
            path: /

I hope that helps.
Best regards,
Sebastian

1 Like

Hello all,

I experienced similar challenge with bare metal kubernetes deployment using external caddy load balancer. I noticed the challenge is the nodeport used by Minio. It is a randomly generated tcp port which cannot be known in advance to configure on external caddy load balancer. This results in connectivity failure when running tutor k8s quickstart.

My question is can I get help with a patch to manually configure the nodeport for some of the services on tutor k8s such as minio and nginx. This allows me to configure my external caddy load balancer to forward traffic to this nodeports.

Thank you