As part of our Native installation we had a few cronjobs running different scripts like transferring tracking logs to S3 for later analysis by Insights for example.
How do I achieve the same thing in Tutor since there does not seem to be a cron mechanism in the containers?
According to the documentation, the logs are persisted to disk by default in
$(tutor config printroot)/data/lms/logs/
and
$(tutor config printroot)/data/cms/logs/
I assume the Tutor way is to run a cronjob from the host in order to transfer the files to S3, right?
Since I am exploring Tutor right now, can someone confirm the different log files are rotated in these directories when they reach a certain size? That definitely needs to be done in order to determine which logs are new or not when I try to sync them to an S3 bucket
For example, under native you would find under /edx/var/log/tracking
tracking.log-20220218-1645175821.gz
tracking.log-20220218-1645179421.gz
tracking.log-20220218-1645183021.gz
tracking.log-20220218-1645186621.gz
Can anyone confirm the same is done in Tutor? Thanks.
Hi @sambapete
I’ve only been running Tutor for just over 3 weeks now so can’t confirm 100% but as far as I can tell there’s no logrotations happening on my instance running Debian 11. I did also login to each of the docker containers but don’t see any logrotates set up within the containers themselves, but I could be misinformed here
I’m planning on doing some work with this to set up rsyslog to push to a separate logging server, happy to give you whatever I come up with whenever I get around to it if you want but I don’t use AWS/S3 as I self-host on premises…
I can confirm that Tutor does not do log rotation. If you need log rotation for the tracking logs you should create a Tutor plugin that modifies the LOGGING configuration in edx-platform.
That will be a problem. It might even be a problem as large as my payment processor not being supported by the payment MFE.
Should I open a ticket? 
And that brings us another good question. Are the log files truncated at some point? They can become huge on a system with lots of users over the years.
If the Tutor plugin system does not allow you to build a plugin that implements log rotation, then yes it’s a problem and you should open a ticket. But my feeling is that creating such a plugin should be fairly simple.
No, the log files are not truncated by default. The reason for that is users might want to catch up on past events if they ever install an analytics system. So far, log size has not been an issue.
I used the scripts I found in the configuration repo to set up logrotate and send the logs to an s3 bucket.
From send-logs-to-s3 and edx_logrotate_tracking_log.
I created both /etc/logrotate.d/hourly/lms_tracking.log and /etc/logrotate.d/hourly/cms_tracking.log
/home/ubuntu/tutor_root/data/lms/logs/tracking.log {
size 1M
compress
create
dateext
dateformat -%Y%m%d-%s
missingok
nodelaycompress
notifempty
rotate 16000
postrotate
/usr/bin/killall -HUP rsyslogd
endscript
lastaction
/usr/local/bin/send-logs-to-object-store -d "/home/ubuntu/tutor_root/data/lms/logs" -b "mybucket" -p "logs/tracking/"
endscript
}
Oh, that would be a nicer way to do it. I’ve been playing with the idea of creating a “tutor-utils” plugin that could support some of the server admin tasks (log rotation, backups…).
I think the format of the tracking log is different. I have never used Insights so I don’t know for sure but it’s probably worth double checking @sambapete.
I believe you are right. What I found in the all.log and tracking.log files under $TUTOR_ROOT/data/logs in the LMS is closer to what is found in /edx/var/logs/lms/edx.log under Native than what is found and expected in /edx/var/logs/tracking/tracking.log under Native.
Don’t get me wrong, the logs in this format are useful from an administrative standpoint but not from an analytics standpoint when using Insights.
To be honest, I’ve never looked how the tracking.log files were produced. I wonder if there is a conversion made between the LMS logs and the tracking logs. And at this point maybe @regis could tell us if this is the case of if there is a reason why the tracking logs are not produced in the expected Open edX format.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.