Cairn-clickhouse Failed

Tutor Tutor Wizard Edition
1

hi @regis Our cairn-clickhouse failed to create users after we imported about 200 courses for initialization. Do you have any idea? Thank you.

$ tutor --version
tutor, version 11.2.5
$ tutor plugins list
license==11.0.0
cairn==11.0.6
$ tutor local run cairn-clickhouse cairn client
Creating tutor_local_cairn-clickhouse_run … done
chown: cannot read directory ‘/var/lib/clickhouse/store/64f/64f02898-3746-43fe-a964-71f8069ed43b/202202_73785_74744_507’: No such file or directory
chown: cannot access ‘/var/lib/clickhouse/store/64f/64f02898-3746-43fe-a964-71f8069ed43b/202202_74744_74744_0’: No such file or directory
chown: changing ownership of ‘/var/lib/clickhouse/store/64f/64f02898-3746-43fe-a964-71f8069ed43b/202202_73785_74746_509/ProfileEvents.size0.bin’: Stale file handle
chown: changing ownership of ‘/var/lib/clickhouse/store/64f/64f02898-3746-43fe-a964-71f8069ed43b/202202_73785_74746_509/columns.size0.bin’: Stale file handle
…
$ sudo ls -l $(tutor config printroot)/data/cairn/clickhouse/store/64f/64f02898-3746-43fe-a964-71f8069ed43b
total 1148
drwxr-x— 2 101 101 14336 Jan 30 15:58 202201_1_36874_3789
drwxr-x— 2 101 101 14336 Jan 31 14:14 202201_36875_46526_2838
drwxr-x— 2 101 101 14336 Feb 1 23:15 202202_51133_61709_2175
drwxr-x— 2 101 101 14336 Feb 2 17:02 202202_61710_69775_3076
drwxr-x— 2 101 101 14336 Feb 3 01:34 202202_69776_73784_1477
…

2

Hi @Xuebo,
What are the logs coming from the cairn-clickhouse container?

tutor local logs --tail=100 -f cairn-clickhouse
3

Thanks @regis. there’s same error in the log. and right now the clickhouse container keeps restarting.

$ tutor local logs --tail=100 -f cairn-clickhouse
Attaching to tutor_local_cairn-clickhouse_1
cairn-clickhouse_1 | chown: cannot read directory ‘/var/lib/clickhouse/store/64f/64f02898-3746-43fe-a964-71f8069ed43b/202202_75004_75734_205’: No such file or directory
cairn-clickhouse_1 | chown: cannot access ‘/var/lib/clickhouse/store/64f/64f02898-3746-43fe-a964-71f8069ed43b/202202_75736_75736_0’: No such file or directory
cairn-clickhouse_1 | chown: cannot access ‘/var/lib/clickhouse/store/64f/64f02898-3746-43fe-a964-71f8069ed43b/202202_75735_75735_0’: No such file or directory
cairn-clickhouse_1 | chown: cannot read directory ‘/var/lib/clickhouse/store/3cf/3cfb289e-fcc2-4763-9f2f-4706419c79b4/202202_94081_94081_0’: No such file or directory
…

4

Can you check the permissions of the files inside of the ~/.local/share/tutor/data/cairn/clickhouse/ folder?

sudo ls -lhn ~/.local/share/tutor/data/cairn/clickhouse/
sudo ls -lhn ~/.local/share/tutor/data/cairn/clickhouse/store
5

$ sudo ls -lhn ~/.local/share/tutor/data/cairn/clickhouse/
total 48K
drwxr-x— 2 101 101 6.0K Feb 3 06:24 access
drwxr-x— 5 101 101 6.0K Feb 3 06:23 data
drwxr-x— 2 101 101 6.0K Feb 3 06:22 dictionaries_lib
drwxr-x— 2 101 101 6.0K Feb 3 06:22 flags
drwxr-xr-x 2 101 101 6.0K Feb 3 06:22 format_schemas
drwxr-x— 2 101 101 6.0K Feb 3 06:23 metadata
drwxr-x— 2 101 101 6.0K Feb 3 06:31 metadata_dropped
drwxr-x— 2 101 101 6.0K Feb 3 06:22 preprocessed_configs
-rw-r----- 1 101 101 56 Feb 3 06:23 status
drwxr-x— 14 101 101 6.0K Feb 3 06:23 store
drwxr-xr-x 2 101 101 6.0K Feb 3 06:22 tmp
drwxr-xr-x 2 101 101 6.0K Feb 3 06:22 user_files
$ sudo ls -lhn ~/.local/share/tutor/data/cairn/clickhouse/store
total 48K
drwxr-x— 3 101 101 6.0K Feb 3 06:23 03b
drwxr-x— 3 101 101 6.0K Feb 3 06:23 065
drwxr-x— 4 101 101 6.0K Feb 3 06:23 177
drwxr-x— 3 101 101 6.0K Feb 3 06:22 27d
drwxr-x— 3 101 101 6.0K Feb 3 06:22 3b4
drwxr-x— 3 101 101 6.0K Feb 3 06:23 514
drwxr-x— 3 101 101 6.0K Feb 3 06:22 533
drwxr-x— 3 101 101 6.0K Feb 3 06:23 5c6
drwxr-x— 3 101 101 6.0K Feb 3 06:23 7c4
drwxr-x— 3 101 101 6.0K Feb 3 06:23 87d
drwxr-x— 3 101 101 6.0K Feb 3 06:23 b72
drwxr-x— 3 101 101 6.0K Feb 3 06:22 f03

6

Is there any file limitation on ClickHouse? It was good when we imported 30, 50 or a little more courses. This issue happened when we imported about 100 courses. I cleaned up /cairn/clickhouse/ folder and re-ran initialization and got same results.

7

No, there isn’t supposed to be any limit to the amount of data that you can add to Clickhouse. Can you please check whether there is any space left on the device?

df -lh
8

We have 99% storage space available for /data/ and tutor config root folders.
Now I can use Clickhouse-Client and cairn-clickhouse container for running sql to create accounts but it’s just not so convenient as tutor commands.

9

I’m not sure I understand. Is tutor local run cairn-clickhouse cairn client now working? Also, can you describe more precisely how you imported the courses? Was it via the studio “import” function?

10

A post was split to a new topic: Cairn: “relation report_schedule” does not exist

11

The tutor command “tutor local run cairn-clckhouse carin client” is still not working.
I used the following command to connect Clickhouse db and then ran sql queries:
docker run -it --rm --link tutor_local_cairn-clickhouse_1:clickhouse-server --net tutor_local_default yandex/clickhouse-client --host cairn-clickhouse --port 9000 --user $user --password $passwd
We did not have problem with course import in studio. it was good with both “mongorestore” and studio “import” function. Could this impact Cairn?
This clickhouse issue happened when we imported 100 courses data into Cairn for initializaiton with command “tutor local init --limit=cairn” (or “quickstart” for Cairn fresh installation)

12

I see that you are pointing “tutor_local_cairn-clickhouse_1” to “clickhouse-server” but you are connecting to --host cairn-clickhouse. Is this a typo?

I’m not sure that I understand the topology of your cluster. Are you running Clickhouse on a separate server or container?

Also, what is the output of the following commands:

docker ps | grep cairn
docker ps | grep clickhouse
13

Both “tutor_local_cairn-clickhouse” and “cairn-clickhouse” are working fine for that connection.
The clickhouse container connection with “docker run …” is not a problem.
Currently our problem is the tutor commands for clickhouse are not working.
tutor local run cairn-clickhouse cairn client
tutor local run cairn-clickhouse cairn createuser USERNAME

14

OK I think that I’m starting to understand what is going on. The clickhouse client is attempting to chown the data volume, as per this line in the entrypoint. It could be failing because there is a lot of data. To confirm this theory, please attempt to run:

tutor local run -e CLICKHOUSE_DO_NOT_CHOWN=1 cairn-clickhouse client

If I’m correct then this command should be successful.

15

The “CLICKHOUSE_DO_NOT_CHOWN=1” works! Thank you Regis.

16

Thanks for your feedback @Xuebo. This fix will be included in v13.0.3.