Heads up on social.coop server space

I was suggesting XMPP because :

1) I was not aware of the Matrix chat ;

2) there's no reason to discuss about snapcraft in a Matrix chatroom, as it would mainly flood the chatrooms already here, IMO. I prefer to have 5 private conversations on 5 different sync channels (that anyone is free to join) than to have 5 conversations in the same Matrix chatroom, for purposes of clarity ;

3) I'm not sure that Riot runs on my computer.

The riot.im client (see my link above) should run in all major web browsers, but if not please let us know because we don't want to pick tech which isn't fairly universally accessible for our public channel.

However, if you want a private discussion with a specific person, you're right, of course you can pick any tech you agree on.

And a snapcraft discussion could just be on a new thread here in the Loomio Tech WG group, I didn't mean to imply you needed to discuss that in our public chat room.

It doesn't run on a 2000s Intel CPU without overheating the CPU, whereas xmpp-client, Gajim, Dino… run without much problems on i3 (obviously not on Windows though). I'm not on my own computer (whose motherboard melted) so it's fine, but this exact trend of building software over Electron or in a browser just because it works over all platforms isn't really on point on ecology and accessibility.

It's also not that much accessible because the Riot client has a terrible UX, and I just think that making it the de-facto standard for free software, cooperative, or union work is a bad idea.

I'd simply recommend rocket.chat instead of Matrix (if not XMPP, but that's just because the clients and servers are objectively better regardless of the protocol itself); I don't know if officially using Matrix doesn't push people out of the decision process and I'd like to question that.

Never heard of Snap till now... However, here is current status of Snap package for Mastodon — https://github.com/tootsuite/mastodon/issues/1068... Appears non-existent at the moment.

Hi all,

Disk space has always been a problem, mainly due to limited budget.... However, there are a couple of things that you missed that will free up further space:

— docker system prune -a — this gets rid of everything that is superfluous, including old containers and volumes. Docker doesn't automatically remove previously ran containers or volumes, unless you say so. It is safe to do this as we mount volumes that need persisting to host, this allows you to not care about Docker preserving volumes or containers.

– NUM_DAYS=7 rake mastodon:media:remove_remote. All media is uploaded locally first, resized/optimised with Paperclip and then pushed to DreamObjects. I'm not sure if Mastodon runs the aforementioned rake task regularly, it is possible it does but from my experience there is always media available to delete when I run it, this frees up a lot of space.

Finally, looking at the disk /var/lib/docker takes up 27GB of space alone, which suggests it is this media that is taking up space. I am now running this rake task in a detached screen, please no reboots : ) until I give the all clear.

Hope this helps.

@wulee @fardog

Thanks I've just added this to the ticket here to be tried and written up somewhere

Additionally, I just came across this —
https://github.com/tootsuite/documentation/blob/d9ecbee47d6c09afbff8cf1280e29018872936b3/Running-Mastodon/Production-guide.md#remote-media-attachment-cache-cleanup

It appears that the cache is also made up of images from other instances we are connected to, scary....

@victormatekole , can we, or should we try to move the /var/lib/docker folder off the root partition? Is it eventually going to get too big?

@wulee good question and probably not a bad strategy as it does tend to bloat. Theoretically there should be no issues but I would never assume with Docker, let me do some digging.

That being said I am wondering if we should consider getting a root server. I run my services via Digital Ocean (mission critical) and Hetzner where I have a couple of root servers(less critical and resource heavy). Hetzner are super cheap and you get a lot for very little, service ain't too bad either and is based in Germany. With an extra $15 or so we can get a couple of terabytes and not too shabby CPU. Nursing a 100GB with a social network of a 1000+ users seems pretty tight.

Not sure what you mean? The servers we have are already root servers.
And there's pros/cons to 100GB of SSD storage vs 1000GB of SATA storage.
Of course it's probably time to add storage space and/or upgrade the server (with a bigger root partition). I have no objections if you all want to switch to another provider either, though it's worth putting together a comparison table.

Sorry, "root servers" implies dedicated hardware/servers (not virtual), as far as I understood Scaleway is a cloud service? You are correct regarding SATA vs SSD. However, Hetzner will allow you mixed setups, we can have SSD for Postgres and SATA for lesser demanding parts of the stack. Either way, I am sure we'd pay less per GB than on Scaleway. But as I suggested earlier there is always a trade-off — having a dedicated server means we look after the hardware, if a disk breaks we have to call Hetzner to replace, from experience they are reasonably fast, in this case.

Nonetheless, I've always felt a 100GB was never enough for our growth rate and requirements long-term. Hetzner was just an example, as I know them but I have no bias. I just wanted to start a conversation, where growth rate, performance and cost are carefully considered.

E.g. piece of hardware:

Intel Core i7-2600
2x HDD SATA 1,5 TB
HDD1x SSD 240 GB
RAM 32GB DDR3
€45.38 / mth

I wonder if we could just throw postgres in RAM?

As indicated in the docs 'trunk' is a dedicated server, and 'toot' is VPS:
https://git.coop/social.coop/tech/operations/wikis/infrastructure-overview

I see ... Do they support upgrades of the disk and perhaps memory?

BTW — how do I get an account to git.coop? Just tried to register under my email address but was denied.

Hi @victormatekole, check out this guide on the steps to get your git.coop account: https://git.coop/social.coop/general/wikis/getting-an-account

I think one of our milestones should be the capability (duplicated amongst several people) to rebuild the server in the event it dies or gets hacked.

In order to learn how to do this, we need a server (or servers) to practice on.

I'd call this a "staging server".

IMO spend the money for a good experience and fewer headaches later

Disk consumption is now 80% by the way but there is more that can be trimmed from the media cache, I think someone restarted the ruby app and thus the job I started got killed.

Wasn't me, honest!

In general I aim to go to the riot.im channels to check if anything's going on on our servers, or to announce it on the public channel if I'm there doing something. I suggest this'd be a good policy for everyone to follow, to help avoid tripping each other up by mistake.

• open channel: https://riot.im/app/#/room/#SocialCoop:matrix.org
• encrypted private channel: https://riot.im/app/#/room/#tech.social.coop:matrix.org

Also, I should add, if this was running in a docker container, I have been noticing a lot of 'dying and restarting' events when browsing the datadog account Mayel (I think) set up to monitor our servers. (Maybe it was you originally, however you did say it was unused and should be removed, and it seems to be a new free account).

If you have any experience interpreting these, I'd be interested what you think...

And anyone else on the tech team who's interested, go and have a look, it's quite impressive. I can either paste the credentials into the tech group's private channel, or maybe I'll get time to get keryringer set up.

Glad you are finding Datadog useful, it is pretty amazing tool! I thought it should be killed as I understood they were removing their free option or at least limiting it to 30 days... I may have got that wrong, last time I checked I could not gain access with my current credentials for social.coop. If you send me the credentials I'd be happy to give my 2 cents...

Social.coop returning 502 bad gateway. @victormatekole @wulee

Thanks. As I mentioned in the chat channel, it seems to have resolved itself...

There've been a bunch of outages like this, in which there's a 502 or similar, and a pingometer/pingdom notification, which mysteriously resolves itself. I'm a bit of a newbie with docker, but it looks like one of the containers will die and then restart. I'd like to know why this happens, I'm still researching that. Maybe @mayel or @victormatekole or one of the other admins will be able to shed some light on that, but at least it isn't currently a critical problem (and I don't think it's a disk related problem).

Timezones: I think we have admins who can fix server issues in the EU and US timezones (assuming they're not indisposed for some reason). Do we have anyone in the Asian timezones in between who could do this?

When I have chance to look at Datadog I will check to see what maybe the root cause. When I look at mem. consumption there is only 200mb free, I wonder if we are hitting some memory limits, which is common with Rails apps as they tend to be resource heavy and leak memory especially from poorly written 3rd-party packages.

I was trying to get the memory/CPU load overlayed with docker events, to see if they correlate. I think I managed it and concluded that the memory grows and then gets resets when there's an event, but this is across the whole system, and yet doesn't imply that memory causes the events rather than vice versa.

Hi, one option you could consider for hosting is buying your own hardware, if you can raise the capital, you could get a 1U server with a lot of RAM and SSDs and HDDs which could run everything (assuming you run a hypervisor on it and multiple virtual servers) and have space for development servers and backups (though you would probably also want backups elsewhere) and then colocate it with a hosting co-operative. Most new servers come with a three warranty — it would make sense to budget for renewing it after 3 years, however at that point the old machine could be used as a backup as, in my experience, servers can generally be run for about ten years.

I like the idea of owning bare metal! Some cost-benefit analysis would have to be performed but I suspect it would be cheaper in the long-run as the network grows in numbers.