Heads up on social.coop server space
Nick S
Public
Seen by 314
Just to report on the last server outage whilst I'm thinking about it....
This last one was resolved by @fardog :raised_hands: who happened to be awake at the right time. He discovered that it was because it was running low on disk space on the main (root) partition.
He pruned some docker image cruft, but it's still currently at 93% full.
Now I can't explain exactly why it's so full, but obviously it's something we need to do something about or our server will start dying all the time. It's not the Mastodon database, that's on another (also 64% full) disk.
(Perhaps the growth was related to the recent Mastodon influx I've been hearing about, but either way we should expect more users and more tooting...)
Sorting this out may require taking the server down for a while, I suspect.
There's also backups, I notice the database backup file has quadrupled in size since about June (2G -> 8.4G), which probably needs investigation. I say 'backup' because we currently just have a manual backup of the database, and it's only run when someone remembers to. In order to protect ourselves from various trousers-falling-down scenarios we might encounter, we need an automated back-up, ideally generational, which also means more 10s - 100s of gigabytes of (off-server) disk space.
Does anyone here know much about analysing Mastodon instances, or know someone who does?
And this touches on the issue of spending funds, which is a different issue but I'll mention here: perhaps we should allocate a budget to working groups, which they can spend at their discretion without the need to go back to the main / finance group?
For those with git.coop accounts, you can see the tickets I created on the recent outage and the disk space question here. I suggest we keep the technical discussion there as much as possible to spare those here who have been overwhelmed by Loomio chat. :) Anyone who wants an account can sign up following the instructions here https://git.coop/social.coop/
Nick S Fri 17 Aug 2018 12:48PM
I think it would make total sense, obvs.
I checked the last budget a while back, it looks like our existing two servers cost $26/month, so another one wouldn't break the bank ATM, since IIRC we have $300/month left over. Apart from this and disk space I don't know what other expenses we expect.
Hmm... I suppose there's paying people to manage things. But this is a whole other question, and I don't think we have nearly enough to pay going rates multiplied by the hours needed, it'd need to be bounties and thank-you gifts.
Michele Kipiel Fri 17 Aug 2018 2:56PM
Then I'd say let's consider getting that extra server, especially if it can make managing things easier for you!
Antoine-Frédéric Raquin Mon 20 Aug 2018 11:20PM
Hello, as Ubuntu 16.04 is the recommended platform for Mastodon, why don't we just dogfood a Mastodon snap? I proposed to document it and I got no feedback, so I gave up.
Gil Scott Fitzgerald Tue 21 Aug 2018 12:32AM
I'd be willing to assist but I can't promise any amount of time.
Antoine-Frédéric Raquin Tue 21 Aug 2018 12:38AM
Me too, but I CAN promise that I won't be available between wednesday and saturday.
Do you want to chat on XMPP? Would you tell me your time zone? I live in France (it's 20 to 3am here).
Antoine-Frédéric Raquin Tue 21 Aug 2018 12:41AM
I don't know much about docker, but maybe getting rid of it could help about the database. (Probably not but who knows)
And I really need to pay my cotisation ASAP tbh
Unless the Mastodon backend isn't reliable enough for this?
Gil Scott Fitzgerald Tue 21 Aug 2018 12:43AM
I will spin up an XMPP box - I'm UTC-6 but I'm sure we can figure something out. I'm busy until next week though.
Nick S Tue 21 Aug 2018 10:12AM
Hi @proton - Was the suggestion for XMPP chat because it's preferable somehow to the matrix chat server we already use, or simply because you weren't aware of it?
https://riot.im/app/#/room/#SocialCoop:matrix.org
If the latter perhaps we should mention it somewhere more prominent, although I'd need to go and check where it's missing from. (See also my mention of a getting-started document in another thread... might make sense if we try and keep all this info in one place as much as possible, but also I'm aware that the order of access to the git.coop server and this loomio group might not always be the same.)
Nick S Tue 21 Aug 2018 10:14AM
Also @proton - regarding snaps, we don't currently use them, we use Docker. I don't think it'd be a trivial thing to change that at this point, but you could certainly try and make a case for doing so when we get the opportunity. You'd need to explain to those of us not familiar with snaps why they're better than the alternatives.
Michele Kipiel · Fri 17 Aug 2018 10:41AM
As already discussed in the chatroom, would it make sense to setup a staging/dev server? What would be the cost of getting one?