Tutor binary and Python plugins

obscherler · July 14, 2021, 1:46pm

Hi,

I tried creating a Python Tutor plugin* using cookiecutter. I then installed it with pip install -e ./tutor-contrib-myplugin. It succeeded and it appears in pip list. However it does not appear in tutor plugins list.

I’m using the Tutor binary. Is it the reason it doesn’t find my plugin? Is there a way of making it find it, or do I need to install Tutor (and all plugins) via pip?

I searched the docs and the forum and didn’t find any information about this.

* I really wanted to create a YAML plugin with a templates section, but I couldn’t get that to work at all.

qali · July 14, 2021, 3:54pm

Hi @obscherler,

pip install -e ./tutor-contrib-myplugin

please reboot your server, than you can see your custom plugin in the list.

sudo reboot

tutor plugins list

tutor plugins enable myplugin

tutor config save

tutor local quickstart

obscherler · July 15, 2021, 5:42pm

Thank you, but rebooting is not the issue.

qali · July 15, 2021, 8:04pm

I see and understand that. It’s not good answer
but Yaml plugins takes effect after installation. But custom Python plugins needs reboot server, after first installation. After that all custom plugins will appear in the plugins list. Off course it’s my experience

thinnguyen · September 26, 2021, 2:41am

Exactly, I tried and it works as expected!
Thanks @qali

regis · September 26, 2021, 10:11am

This is incorrect. Plugins should be detected immediately. That is, if you are not running the binary installation. See the warning in the Tutor docs: Installing Tutor — Tutor documentation

Note however that you will not be able to use custom plugins with this pre-compiled binary. The only plugins you can use with this approach are those that are already bundled with the binary: see the existing plugins.

The reason for that is that binaries built with pyinstaller load their Python packages from a custom temporary directory created at runtime, and it is not possible or recommended to add extra directories to sys.path. See the pyinstaller FAQ:

If I make an executable with Pyinstaller, is it still possible to import new things that are on the PYTHONPATH?
No and yes. PYTHONPATH is ignored, but you could evaluate it yourself and add the paths to sys.path. This is strongly discouraged, though, as it may lead to conflicts and thus problems which are very hard to track down.

qali · September 26, 2021, 4:58pm

Hi @regis,
I always install tutor via binary, and to the new clean Ubuntu 20.04. I mean after first installation of custom plugin(clean VPS), not apears immediately. As you see example below. But after first restart of server all goes fine. Off course is not god answer as I mantioned before And I belive I am not the only one face this issue. Thanks.

mpo@vmi651616:~$ tutor plugins list
android==12.0.0 (disabled)
discovery==12.0.2 (disabled)
ecommerce==12.0.0 (disabled)
license==12.0.1 (disabled)
mfe==12.0.2 (disabled)
minio==12.0.4 (disabled)
notes==12.0.0 (disabled)
webui==12.0.1 (disabled)
xqueue==12.1.0 (disabled)
mpo@vmi651616:~$ cookiecutter https://github.com/overhangio/cookiecutter-tutor-plugin.git
You've downloaded /home/mpo/.cookiecutters/cookiecutter-tutor-plugin before. Is it okay to delete and re-download it? [yes]:
plugin_name [myplugin]: customplg
package_name [tutor-contrib-customplg]:
module_name [tutorcustomplg]:
git_repo [https://github.com/myusername/tutor-contrib-customplg]:
author [John Doe]:
license [AGPLv3]:
mpo@vmi651616:~$
mpo@vmi651616:~$ tutor plugins list
android==12.0.0 (disabled)
discovery==12.0.2 (disabled)
ecommerce==12.0.0 (disabled)
license==12.0.1 (disabled)
mfe==12.0.2 (disabled)
minio==12.0.4 (disabled)
notes==12.0.0 (disabled)
webui==12.0.1 (disabled)
xqueue==12.1.0 (disabled)
mpo@vmi651616:~$ pip3 install -e tutor-contrib-customplg/
Obtaining file:///home/mpo/tutor-contrib-customplg
Requirement already satisfied: tutor in ./.local/lib/python3.8/site-packages (from tutor-contrib-customplg==0.1.0) (12.1.2)
Requirement already satisfied: appdirs in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-customplg==0.1.0) (1.4.4)
Requirement already satisfied: click>=7.0 in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-customplg==0.1.0) (8.0.1)
Requirement already satisfied: kubernetes in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-customplg==0.1.0) (18.20.0)
Requirement already satisfied: pyyaml>=4.2b1 in /usr/lib/python3/dist-packages (from tutor->tutor-contrib-customplg==0.1.0) (5.3.1)
Requirement already satisfied: mypy in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-customplg==0.1.0) (0.910)
Requirement already satisfied: jinja2 in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-customplg==0.1.0) (3.0.1)
Requirement already satisfied: pycryptodome in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-customplg==0.1.0) (3.10.4)
Requirement already satisfied: urllib3>=1.24.2 in /usr/lib/python3/dist-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (1.25.8)
Requirement already satisfied: google-auth>=1.0.1 in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (2.1.0)
Requirement already satisfied: requests-oauthlib in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (1.3.0)
Requirement already satisfied: requests in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (2.26.0)
Requirement already satisfied: python-dateutil>=2.5.3 in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (2.8.2)
Requirement already satisfied: certifi>=14.05.14 in /usr/lib/python3/dist-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (2019.11.28)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (1.14.0)
Requirement already satisfied: setuptools>=21.0.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (45.2.0)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-customplg==0.1.0) (1.2.1)
Requirement already satisfied: mypy-extensions<0.5.0,>=0.4.3 in ./.local/lib/python3.8/site-packages (from mypy->tutor->tutor-contrib-customplg==0.1.0) (0.4.3)
Requirement already satisfied: typing-extensions>=3.7.4 in ./.local/lib/python3.8/site-packages (from mypy->tutor->tutor-contrib-customplg==0.1.0) (3.10.0.2)
Requirement already satisfied: toml in ./.local/lib/python3.8/site-packages (from mypy->tutor->tutor-contrib-customplg==0.1.0) (0.10.2)
Requirement already satisfied: MarkupSafe>=2.0 in ./.local/lib/python3.8/site-packages (from jinja2->tutor->tutor-contrib-customplg==0.1.0) (2.0.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in ./.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->tutor->tutor-contrib-customplg==0.1.0) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in ./.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->tutor->tutor-contrib-customplg==0.1.0) (4.2.2)
Requirement already satisfied: rsa<5,>=3.1.4 in ./.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->tutor->tutor-contrib-customplg==0.1.0) (4.7.2)
Requirement already satisfied: oauthlib>=3.0.0 in ./.local/lib/python3.8/site-packages (from requests-oauthlib->kubernetes->tutor->tutor-contrib-customplg==0.1.0) (3.1.1)
Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /usr/lib/python3/dist-packages (from requests->kubernetes->tutor->tutor-contrib-customplg==0.1.0) (2.8)
Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in ./.local/lib/python3.8/site-packages (from requests->kubernetes->tutor->tutor-contrib-customplg==0.1.0) (2.0.6)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in ./.local/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes->tutor->tutor-contrib-customplg==0.1.0) (0.4.8)
Installing collected packages: tutor-contrib-customplg
  Running setup.py develop for tutor-contrib-customplg
Successfully installed tutor-contrib-customplg
mpo@vmi651616:~$ tutor plugins list
android==12.0.0 (disabled)
discovery==12.0.2 (disabled)
ecommerce==12.0.0 (disabled)
license==12.0.1 (disabled)
mfe==12.0.2 (disabled)
minio==12.0.4 (disabled)
notes==12.0.0 (disabled)
webui==12.0.1 (disabled)
xqueue==12.1.0 (disabled)
mpo@vmi651616:~$

And after the reboot, every custom plugin installations goes fine. ( Not needs to restart server)

mpo@vmi651616:~$ tutor plugins list
Failed to load entrypoint 'testplg = tutortestplg.plugin' from distribution tutor-contrib-testplg 0.1.0
Traceback (most recent call last):
  File "/home/mpo/.local/bin/tutor", line 8, in <module>
    sys.exit(main())
  File "/home/mpo/.local/lib/python3.8/site-packages/tutor/commands/cli.py", line 29, in main
    add_plugin_commands(cli)
  File "/home/mpo/.local/lib/python3.8/site-packages/tutor/commands/plugins.py", line 141, in add_plugin_commands
    for plugin in plugins.iter_installed():
  File "/home/mpo/.local/lib/python3.8/site-packages/tutor/plugins.py", line 401, in iter_installed
    yield from Plugins.iter_installed()
  File "/home/mpo/.local/lib/python3.8/site-packages/tutor/plugins.py", line 362, in iter_installed
    for plugin in PluginClass.iter_installed():
  File "/home/mpo/.local/lib/python3.8/site-packages/tutor/plugins.py", line 199, in iter_installed
    for plugin in cls.iter_load():
  File "/home/mpo/.local/lib/python3.8/site-packages/tutor/plugins.py", line 233, in iter_load
    yield cls(entrypoint)
  File "/home/mpo/.local/lib/python3.8/site-packages/tutor/plugins.py", line 220, in __init__
    super().__init__(entrypoint.name, entrypoint.load())
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2444, in load
    self.require(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2467, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (PyYAML 5.3.1 (/usr/lib/python3/dist-packages), Requirement.parse('pyyaml>=5.4.1'), {'kubernetes'})
mpo@vmi651616:~$ pip3 install pyyaml==5.4.1
Collecting pyyaml==5.4.1
  Downloading PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
     |████████████████████████████████| 662 kB 9.6 MB/s
Installing collected packages: pyyaml
Successfully installed pyyaml-5.4.1
**> mpo@vmi651616:~$ tutor plugins list**
**> testplg==0.1.0 (disabled)**
**> customplg==0.1.0 (disabled)**
mpo@vmi651616:~$ cookiecutter https://github.com/overhangio/cookiecutter-tutor-plugin.git
You've downloaded /home/mpo/.cookiecutters/cookiecutter-tutor-plugin before. Is it okay to delete and re-download it? [yes]:
plugin_name [myplugin]: anotherplg
package_name [tutor-contrib-anotherplg]:
module_name [tutoranotherplg]:
git_repo [https://github.com/myusername/tutor-contrib-anotherplg]:
author [John Doe]:
license [AGPLv3]:
mpo@vmi651616:~$ pip3 install -e tutor-contrib-anotherplg/
Obtaining file:///home/mpo/tutor-contrib-anotherplg
Requirement already satisfied: tutor in ./.local/lib/python3.8/site-packages (from tutor-contrib-anotherplg==0.1.0) (12.1.2)
Requirement already satisfied: mypy in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-anotherplg==0.1.0) (0.910)
Requirement already satisfied: appdirs in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-anotherplg==0.1.0) (1.4.4)
Requirement already satisfied: kubernetes in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-anotherplg==0.1.0) (18.20.0)
Requirement already satisfied: jinja2 in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-anotherplg==0.1.0) (3.0.1)
Requirement already satisfied: pycryptodome in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-anotherplg==0.1.0) (3.10.4)
Requirement already satisfied: click>=7.0 in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-anotherplg==0.1.0) (8.0.1)
Requirement already satisfied: pyyaml>=4.2b1 in ./.local/lib/python3.8/site-packages (from tutor->tutor-contrib-anotherplg==0.1.0) (5.4.1)
Requirement already satisfied: toml in ./.local/lib/python3.8/site-packages (from mypy->tutor->tutor-contrib-anotherplg==0.1.0) (0.10.2)
Requirement already satisfied: typing-extensions>=3.7.4 in ./.local/lib/python3.8/site-packages (from mypy->tutor->tutor-contrib-anotherplg==0.1.0) (3.10.0.2)
Requirement already satisfied: mypy-extensions<0.5.0,>=0.4.3 in ./.local/lib/python3.8/site-packages (from mypy->tutor->tutor-contrib-anotherplg==0.1.0) (0.4.3)
Requirement already satisfied: google-auth>=1.0.1 in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (2.1.0)
Requirement already satisfied: certifi>=14.05.14 in /usr/lib/python3/dist-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (2019.11.28)
Requirement already satisfied: setuptools>=21.0.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (45.2.0)
Requirement already satisfied: requests in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (2.26.0)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (1.14.0)
Requirement already satisfied: requests-oauthlib in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (1.3.0)
Requirement already satisfied: urllib3>=1.24.2 in /usr/lib/python3/dist-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (1.25.8)
Requirement already satisfied: python-dateutil>=2.5.3 in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (2.8.2)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in ./.local/lib/python3.8/site-packages (from kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (1.2.1)
Requirement already satisfied: MarkupSafe>=2.0 in ./.local/lib/python3.8/site-packages (from jinja2->tutor->tutor-contrib-anotherplg==0.1.0) (2.0.1)
Requirement already satisfied: rsa<5,>=3.1.4 in ./.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (4.7.2)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in ./.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (4.2.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in ./.local/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (0.2.8)
Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /usr/lib/python3/dist-packages (from requests->kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (2.8)
Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in ./.local/lib/python3.8/site-packages (from requests->kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (2.0.6)
Requirement already satisfied: oauthlib>=3.0.0 in ./.local/lib/python3.8/site-packages (from requests-oauthlib->kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (3.1.1)
Requirement already satisfied: pyasn1>=0.1.3 in ./.local/lib/python3.8/site-packages (from rsa<5,>=3.1.4->google-auth>=1.0.1->kubernetes->tutor->tutor-contrib-anotherplg==0.1.0) (0.4.8)
Installing collected packages: tutor-contrib-anotherplg
  Running setup.py develop for tutor-contrib-anotherplg
Successfully installed tutor-contrib-anotherplg

> mpo@vmi651616:~$ tutor plugins list
> testplg==0.1.0 (disabled)
> customplg==0.1.0 (disabled)
> anotherplg==0.1.0 (disabled)
mpo@vmi651616:~$

regis · September 27, 2021, 6:03am

I think that your issue is that you have multiple versions of the tutor executable on your machine. You have the tutor binary (in /usr/local/bin/tutor probably?) and the pip-installed one (in ~/.local/bin).

At any point, you can verify which tutor you are using by running which tutor. I strongly suggest that you keep only one and get rid of the binary (sudo rm /usr/local/bin/tutor). Rebooting the server might cause the PATH environment variable to be somehow modified, or your cache to be cleared (somehow). Rebooting only hides the actual issue, in this particular case.