External minio tutor k8s init boto error

Tutor
1

Hello,

I am trying to use an external minio server, so I managed to remove the minio deploy step and created the buckets as is done in the pre-init script. I also updated the configMaps to use the correct key and secret as well as host but I keep running into this weird error when running the init script.

Running migrations:
  Applying certificates.0003_data__default_modes...Traceback (most recent call last):
  File "./manage.py", line 123, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/openedx/venv/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/openedx/edx-platform/lms/djangoapps/certificates/migrations/0003_data__default_modes.py", line 24, in forwards
    File(open(settings.PROJECT_ROOT / 'static' / 'images' / 'default-badges' / file_name))
  File "/openedx/venv/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 94, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/openedx/venv/local/lib/python2.7/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/openedx/venv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 417, in _save
    self._save_content(key, content, headers=headers)
  File "/openedx/venv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 428, in _save_content
    rewind=True, **kwargs)
  File "/openedx/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/openedx/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/openedx/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
    query_args=query_args
  File "/openedx/venv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request
    retry_handler=retry_handler
  File "/openedx/venv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/openedx/venv/local/lib/python2.7/site-packages/boto/connection.py", line 940, in _mexe
    request.body, request.headers)
  File "/openedx/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 882, in sender
    if not self.should_retry(response, chunked_transfer):
  File "/openedx/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 983, in should_retry
    '%s vs. %s' % (self.etag, self.md5))
boto.exception.S3DataError: BotoClientError: ETag from S3 did not match computed MD5. None vs. c4a65ea6d491230148feff2394581852

Best,
Max

2

Hi @yamashi! How did you install the MinIO server?

3

I followed the tutorial on the official website for distributed platforms. I am able to connect to it via web browser or the mc command and I also was able to execute the pre-init script (of course changing the endpoint from http://minio to the service’s endpoint).

Thanks,
Max

4

I honestly have no idea of what the issue could be. There are other reports that point to KMS encryption as the culprit, but I find it hard to believe.

Did you correctly set all MinIO-related values in the config.yml file?

5

So I have news, after messing around with /tmp/boto.cfg with my host being minio.default.svc:9000 I made a tunnel on localhost:80 to minio.default.svc:9000 and now when using host=localhost it works !

My assumption is that boto.cfg’s host does not support specifying the port, I will keep digging.

6

For reference if anyone encounters this, I ended up adding an additionnal service/deployment to handle port forwarding here is the source:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: port-forward
  labels:
    app.kubernetes.io/name: port-forward
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: port-forward
  template:
    metadata:
      labels:
        app.kubernetes.io/name: port-forward
    spec:
      containers:
        - name: port-forward
          image: marcnuri/port-forward
          ports:
            - containerPort: 80
          env:
            - name: REMOTE_HOST
              value: "minio.default.svc"
            - name: REMOTE_PORT
              value: "9000"
            - name: LOCAL_PORT
              value: "80"
---
apiVersion: v1
kind: Service
metadata:
  name: port-forward
spec:
  type: NodePort
  ports:
    - port: 80
      protocol: TCP
  selector:
    app.kubernetes.io/name: port-forward
7

Indeed, I see by digging inside the boto.connection.AWSAuthConnection that the port can only take values 80 or 443. So setting up a service is your best bet to solve this issue. Congrats on the investigation!