Loomio
Thu 18 May 2023 1:25PM

"Internal server error" if Loomio's not been used in a while

J Josh Public Seen by 47

Hi,

I've set up a self-hosted instance of loomio using loomio-deploy. The initial feedback from our committee has been extremely positive! It's looking like it's going to be extremely helpful - thank you for such a great tool.

But I have noticed, if it's not been used in a little while, Loomio becomes a little sleepy. ie after a day of not much usage, the next morning it will be very slow to load, we see "internal server error" messages for a few minutes. And then it's back to normal again.

There doesn't seem to be huge CPU or memory pressure on the server at this time. I assume some internal processes are suspended after inactivity and take a few seconds to recover? I'm quite new to docker-compose, so I'm not sure the best way to interrogate the logfiles to understand what's happening. Is this a known phenomena?

I'm tempted to have a cronjob wget our Loomio site every half hour, see if that resolves it, but I know it's not ideal!

Thanks for any help.

RG

Robert Guthrie Thu 18 May 2023 6:45PM

Hi @Josh there will be a clear reason for this problem - in my experience we don't have an issue where Loomio gets "sleepy".. it tends to either work perfectly for years, or have a problem that needs fixed.

`docker-compose logs -f` is an excellent tool. Run that, then make requests against the server, and you should see the error.

You might want to consider adding Sentry to your app so error reports get emailed to you.

J

Josh Tue 23 May 2023 3:18PM

Hi @Robert Guthrie, thanks for the quick reply and the helpful advice. I was able to run `docker-compose logs -f` while replicating the problem.

This was the output:

loomio-app   | source=rack-timeout id=35d6557d-1ba1-4a4b-8e37-37092375ac85 timeout=15000ms state=ready
loomio-app   | source=rack-timeout id=35d6557d-1ba1-4a4b-8e37-37092375ac85 timeout=15000ms service=15130ms state=timed_out
loomio-app   | method=GET path=/committee format=html controller=GroupsController action=show status=500 error='Rack::Timeout::RequestTimeoutException: Request ran for longer than 15000ms ' duration=17141.83 view=0.00 db=170.88
loomio-app   |   
loomio-app   | Rack::Timeout::RequestTimeoutException (Request ran for longer than 15000ms ):
loomio-app   |   
loomio-app   | app/services/task_service.rb:59:in `parse_tasks'
loomio-app   | app/services/task_service.rb:55:in `parse_and_update'
loomio-app   | app/models/concerns/has_rich_text.rb:43:in `block (2 levels) in is_rich_text'
loomio-app   | app/helpers/current_user_helper.rb:34:in `set_last_seen_at'
loomio-app   | app/helpers/locales_helper.rb:8:in `use_preferred_locale'
loomio-app   | app/helpers/locales_helper.rb:3:in `process_time_zone'
loomio-app   | source=rack-timeout id=35d6557d-1ba1-4a4b-8e37-37092375ac85 timeout=15000ms service=17930ms state=completed
loomio-app   | source=rack-timeout id=bc0947a7-b4ef-446b-807a-80d9a8152a33 timeout=15000ms state=ready
loomio-app   | source=rack-timeout id=03a3d02a-b1b3-4cf4-9373-2708a7eaced6 timeout=15000ms state=ready
loomio-app   | source=rack-timeout id=b4f9814f-f6c3-4df7-a7d7-2766461a59ea timeout=15000ms state=ready
loomio-app   | source=rack-timeout id=03a3d02a-b1b3-4cf4-9373-2708a7eaced6 timeout=15000ms service=15009ms state=timed_out
loomio-app   | source=rack-timeout id=03a3d02a-b1b3-4cf4-9373-2708a7eaced6 timeout=15000ms service=15074ms state=completed
loomio-app   | source=rack-timeout id=bc0947a7-b4ef-446b-807a-80d9a8152a33 timeout=15000ms service=22771ms state=timed_out
loomio-app   | source=rack-timeout id=b4f9814f-f6c3-4df7-a7d7-2766461a59ea timeout=15000ms service=22249ms state=timed_out
loomio-app   | source=rack-timeout id=bc0947a7-b4ef-446b-807a-80d9a8152a33 timeout=15000ms service=22846ms state=completed
loomio-app   | source=rack-timeout id=b4f9814f-f6c3-4df7-a7d7-2766461a59ea timeout=15000ms service=22336ms state=completed

We saw this error message in the frontend:

RG

Robert Guthrie Wed 24 May 2023 1:44AM

@Josh I've got no idea (yet) about this one.

Check that PUMA_WORKERS is equal to the number of CPU cores.

You could change RACK_TIMEOUT_SERVICE_TIMEOUT to 30 and see if that allows the request to complete.

If your server has less than 4gb ram, then it might be a good idea to give it 4gb.