Shifting backend servers to new provider
This post refers to this slack thread. Parts of it are copied from there.
Most of our current bill from Google is caused by the backend server which runs the database (where we store results after the app has sent them to firebase). This cost was £1086 for June 22 to September 19, or ~£362/month. This includes a spike due to ToggleCorp's work, but on average, the server costs about £300/month (335€). Most of this bill is the backend server (say ~300€). And of this, the majority is hard disk space, which @benni has had to keep increasing in the last few months to store the growing number of results the app is generating.
We (Benni and I) are planning to do work next week to improve the way the data is stored. It is currently very redundant, and we know there is room to shrink this dramatically, which would reduce the volume stored for the entire history of MapSwipe and make new data more efficient to store. In practice, this means reducing our monthly bill, and making future mapathons cheaper as well.
The second element of the job is to move the servers from Google to a different provider (firebase stays for now, there is no easy to get rid of it). Google servers are very expensive, and we can get better machines for a lot less money.
With those 2 elements, I expect to shrink the monthly bill from the current ~300€ to less than 100€. A side effect of this being that we would get a much smaller bill from Google (~£20-50 depending on app activity) paid by BRC as currently, and the rest, from a new provider, would have to be paid out of OpenCollective, unless one of the partner orgs volunteers to cover that bill.
We currently use 2 backend servers:
production: 2 CPUs, 7.5GB RAM, 825GB SSD disk space
development: 1 CPU, 3.75GB RAM, 250GB SSD disk space
Considering that the upcoming community dashboard statistics API will incur some extra load on the servers, it would make sense to switch to slightly bigger machines (or machines able to scale up on demand), at least for production.
This thread is to propose moving from the current situation:
BRC pays a monthly bill to Google for firebase + backend servers of around GBP 300-350
to a new situation:
BRC still pays a monthly bill to Google for firebase only. The expected average cost would be below GBP 50, but spikes are possible due to increased activity, for instance during a big swiping event like we had in April.
The backend servers are rented from a different provider (to be defined) and paid out of OpenCollective money. The expected bill would be below 100 EUR from initial exploration (as an example, a server rented from Hetzner, a provider I use for other work, with characteristics similar to our current production machine would cost ~ 55€/month, the development server ~20€/month, this is before considering storage size reduction).
Worth noting that there are ways to push the cost of servers even further down, but this would come with more maintenance work, and at our current scale, it would make no sense.
If this proposal is accepted by the governance team, then we would start paying this new bill out of OC sometime around mid-October.
As a safety net, should we find out in October that the actual cost is much higher than what I'm expecting in this post, we would update the proposal, or simply cancel it entirely, but I think the risk is very low.
I will be posting a separate related thread about the work itself to make these changes, and paying for work time+travel out of OC for this. This current thread is not meant to imply anything about the other one.
Laurent Savaete Wed 5 Oct 2022 8:51AM
True about the need for money. But this is also true of the current situation with BRC paying the bill. @Johnny and @Andrew Braye would know better, but my impression is that this will not last forever either. And finding a smaller amount of money is usually easier than a bigger one ;)
Just to clarify, the cost of these backend servers should not have spikes the same way firebase currently does when we're running a big mapathon. That will remain on the google bill. For the servers, the cost should be pretty stable, or increase every now and then when we need to increase disk space to store all the results we keep producing. As a very crude example, if we were to run a huge event in March next year, maybe our firebase bill would jump to $1000 for that month and go back to the average $50/month after that, and the monthly bill for servers would increase $5-10/month "forever after that". If the event is really big, we might need to temporarily increase the size of the production server (and cost for a few days, probably $10-20 one-off). This is partly why we need to work out how to decide what donation to get for these events.
At some point, it might make sense to archive old projects and their results in a different way which would be less accessible, but also cheaper, but I don't think we've reached that point yet.
As for the actual cost, I did a bit of research yesterday. The best price I saw so far is with Hetzner, but the exact cost will depend on how big the data is after we've restructured the data. Assuming we divide that size by 3, we should be paying just under 50€/month total, for both servers. We will know the initial cost next week, when we setup the new servers. This should be stable over the first few months, and mostly easy to anticipate afterwards as spikes should be minimal.
Jorieke Vyncke Wed 5 Oct 2022 9:42AM
Hey @Laurent Savaete thanks a lot for the explanation on the spikes. That means that we would be able to plan relatively easily the costs for the sever. Which is great!
What we would like to archive from when is indeed a good question too to ask ourselves.
Keep us up to date about the actual amount. If it would be around the 50EUR per month, it certainly is a thumbs up on the proposal for me. 👍
Jorieke Vyncke · Tue 4 Oct 2022 6:43PM
Hello @Laurent Savaete and @Benni , this looks like a sound proposal to me.
The question is if we would have enough money in OpenCollective every year, but I guess as long as MSF is contributing yearly a larger amount, and we have some bits and pieces from others like the mapathon organiser that can cause a spike it might work out pretty well.
Having an overview of what all is payed would be good to have as well, but I assume that we would be easily able to have an overview in the OpenCollective on how much with payed into the server provider.
For finding out the actual cost, would we really know this after one month, or do we need two, three months to see if all is more or less stable?