Hello Everyone,
Anyone have faced similar issue earlier ?
Environment: Production
CPU: 4 Core
RAM: 16GB
Tutor version: 13.2.2
caddy:2.4.6
When I am trying to do Batch Enrollment for more than 15 learners in the instructor tab, It is displaying the request status as 502. Please see the below Screenshot.
CADDY LOGS
caddy_1 | {"level":"error","ts":1657645678.0171807,"logger":"http.log.error.log0","msg":"EOF","request":{"remote_addr":"103.81.92.231:25860","proto":"HTTP/2.0","method":"POST","host":"HOSTNAME","uri":"/courses/course-v1:abc+AM-INT+Batch1/instructor/api/students_update_enrollment","headers":{"Sec-Fetch-Site":["same-origin"],"Referer":["https://HOSTNAME/courses/course-v1:abc+AM-INT+Batch1/instructor"],"Accept-Language":["en-US,en;q=0.9,pt;q=0.8,gu;q=0.7"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\""],"Content-Type":["application/x-www-form-urlencoded; charset=UTF-8"],"X-Requested-With":["XMLHttpRequest"],"Accept":["application/json, text/javascript, */*; q=0.01"],"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Fetch-Mode":["cors"],"Cookie":["experiments_is_enterprise=false; sessionid=1|s6dz97prf5aozuhs0hd3yebto6zfo0v9|tVYhyf8NZwFK|Ijc2MTE2Mzc2ZWQ1ZWM0MjM4MzVhZjk0ZjNiMGU1N2E1ZWM3Y2U5NDUxNDhlZmJjMTZjNjM1MmI0YmQzOWU4NTgi:1o6opb:5Z2Shn37nvMcng5O_AYzIR4qITM; experiments_is_enterprise=false; edxloggedin=true; edx-user-info=\"{\\\"version\\\": 1\\054 \\\"username\\\": \\\"administrator\\\"\\054 \\\"header_urls\\\": {\\\"logout\\\": \\\"https://HOSTNAME/logout\\\"\\054 \\\"account_settings\\\": \\\"https://HOSTNAME/account/settings\\\"\\054 \\\"learner_profile\\\": \\\"https://HOSTNAME/u/administrator\\\"\\054 \\\"resume_block\\\": \\\"https://HOSTNAME/courses/course-v1:abc+SM-INT+23JUNE2022/jump_to/block-v1:abc+SM-INT+23JUNE2022+type@edx_sga+block@4b7afafaee184046a9ae3c92054827cc\\\"}\\054 \\\"user_image_urls\\\": {\\\"full\\\": \\\"https://HOSTNAME/static/images/profiles/default_500.4215dbe8010f.png\\\"\\054 \\\"large\\\": \\\"https://HOSTNAME/static/images/profiles/default_120.4a5e0900098e.png\\\"\\054 \\\"medium\\\": \\\"https://HOSTNAME/static/images/profiles/default_50.3455a6581573.png\\\"\\054 \\\"small\\\": \\\"https://HOSTNAME/static/images/profiles/default_30.deee7287e843.png\\\"}}\"; csrftoken=I5MAvcbDlKKZ42Ff178BLNFA3a5x1PlUVzoXGSXDvWOB67ayHwKHLXYu5JktHH0m; openedx-language-preference=en; sessionid=1|udtbudyg5ff8w8zk16y8igiu4kvalpqj|9zzIZmAoLDOd|ImQxZDkxYWU2ZjQyNmE5NWEwODVhNzA4YTI3Nzk2MDJhZmVjNDY0NmEzMDhjMTY1OWFiYjJjODEwZTNjMzRiNTYi:1oBIId:3Zu4-t7pWYkAWsGrHCwZa6oqjE4"],"Content-Length":["1050"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"],"X-Csrftoken":["I5MAvcbDlKKZ42Ff178BLNFA3a5x1PlUVzoXGSXDvWOB67ayHwKHLXYu5JktHH0m"],"Sec-Ch-Ua-Mobile":["?0"],"Origin":["https://HOSTNAME"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"HOSTNAME"}},"duration":60.042919962,"status":502,"err_id":"7bs6n3gtq","err_trace":"reverseproxy.statusError (reverseproxy.go:886)"}
caddy_1 | {"level":"error","ts":1657645678.0172203,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"103.81.92.231:25860","proto":"HTTP/2.0","method":"POST","host":"HOSTNAME","uri":"/courses/course-v1:abc+AM-INT+Batch1/instructor/api/students_update_enrollment","headers":{"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\""],"Content-Type":["application/x-www-form-urlencoded; charset=UTF-8"],"X-Requested-With":["XMLHttpRequest"],"Sec-Fetch-Site":["same-origin"],"Referer":["https://HOSTNAME/courses/course-v1:abc+AM-INT+Batch1/instructor"],"Accept-Language":["en-US,en;q=0.9,pt;q=0.8,gu;q=0.7"],"Accept":["application/json, text/javascript, */*; q=0.01"],"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip, deflate, br"],"Content-Length":["1050"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"],"X-Csrftoken":["I5MAvcbDlKKZ42Ff178BLNFA3a5x1PlUVzoXGSXDvWOB67ayHwKHLXYu5JktHH0m"],"Sec-Ch-Ua-Platform":["\"Linux\""],"Sec-Fetch-Mode":["cors"],"Cookie":["experiments_is_enterprise=false; sessionid=1|s6dz97prf5aozuhs0hd3yebto6zfo0v9|tVYhyf8NZwFK|Ijc2MTE2Mzc2ZWQ1ZWM0MjM4MzVhZjk0ZjNiMGU1N2E1ZWM3Y2U5NDUxNDhlZmJjMTZjNjM1MmI0YmQzOWU4NTgi:1o6opb:5Z2Shn37nvMcng5O_AYzIR4qITM; experiments_is_enterprise=false; edxloggedin=true; edx-user-info=\"{\\\"version\\\": 1\\054 \\\"username\\\": \\\"administrator\\\"\\054 \\\"header_urls\\\": {\\\"logout\\\": \\\"https://HOSTNAME/logout\\\"\\054 \\\"account_settings\\\": \\\"https://HOSTNAME/account/settings\\\"\\054 \\\"learner_profile\\\": \\\"https://HOSTNAME/u/administrator\\\"\\054 \\\"resume_block\\\": \\\"https://HOSTNAME/courses/course-v1:abc+SM-INT+23JUNE2022/jump_to/block-v1:abc+SM-INT+23JUNE2022+type@edx_sga+block@4b7afafaee184046a9ae3c92054827cc\\\"}\\054 \\\"user_image_urls\\\": {\\\"full\\\": \\\"https://HOSTNAME/static/images/profiles/default_500.4215dbe8010f.png\\\"\\054 \\\"large\\\": \\\"https://HOSTNAME/static/images/profiles/default_120.4a5e0900098e.png\\\"\\054 \\\"medium\\\": \\\"https://HOSTNAME/static/images/profiles/default_50.3455a6581573.png\\\"\\054 \\\"small\\\": \\\"https://HOSTNAME/static/images/profiles/default_30.deee7287e843.png\\\"}}\"; csrftoken=I5MAvcbDlKKZ42Ff178BLNFA3a5x1PlUVzoXGSXDvWOB67ayHwKHLXYu5JktHH0m; openedx-language-preference=en; sessionid=1|udtbudyg5ff8w8zk16y8igiu4kvalpqj|9zzIZmAoLDOd|ImQxZDkxYWU2ZjQyNmE5NWEwODVhNzA4YTI3Nzk2MDJhZmVjNDY0NmEzMDhjMTY1OWFiYjJjODEwZTNjMzRiNTYi:1oBIId:3Zu4-t7pWYkAWsGrHCwZa6oqjE4"],"Sec-Ch-Ua-Mobile":["?0"],"Origin":["https://HOSTNAME"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"HOSTNAME"}},"common_log":"103.81.92.231 - - [12/Jul/2022:17:07:58 +0000] \"POST /courses/course-v1:abc+AM-INT+Batch1/instructor/api/students_update_enrollment HTTP/2.0\" 502 0","user_id":"","duration":60.042919962,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}
caddy_1 | {"level":"error","ts":1657645678.0172203,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"103.81.92.231:25860","proto":"HTTP/2.0","method":"POST","host":"HOSTNAME","uri":"/courses/course-v1:abc+AM-INT+Batch1/instructor/api/students_update_enrollment","tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"HOSTNAME"}},"user_id":"","duration":60.042919962,"size":0,"status":502}
I have tried to do following Caddyfile changes with regards to timeout but it seems there is a hard limit of 1 minute and it is also the default value. We have tried to decrease the timeout in caddy file and it works but if we try to increase it to more than 1 min then it doesn’t pick up the values.
Increased the timeout to 2 minutes, but the request is timed out exactly at 1 min and give the error 502.
Do not see any errors on LMS side. After looking at the data directly in database it seems the enrollment were successful. Only on the UI it shows the enrolment failed message.
Anyone have any idea or any pointers on how can I resolve this issue ? Looking options to bypass the 1 min limit set on Caddy side or to handle it gracefully. Any help is appreciated.
Thanks & Regards
Kunj