and remove your orphans. that's safe

root@loomio:~/loomio-deploy# docker-compose up -d db

Creating loomio-db

root@loomio:~/loomio-deploy# docker exec loomio-db su - postgres -c 'pgdump -O -Fc loomioproduction -f /pgdumps/loomio_production.dump'

pgdump: [custom archiver] could not open output file "/pgdumps/loomioproduction.dump": Permission denied

root@loomio:~/loomio-deploy#

docker exec -ti loomio-db su - postgres -c 'pg_dump loomio_production' > loomio_production.sql

How about that one? do you know anything about pg_dump and what flag to use?

@Rob Guthrie Looks like that worked:

root@loomio:~/loomio-deploy# ls -lah|grep -i loom

-rw-r--r-- 1 root root 14M Nov 21 12:28 loomio_production.sql

so I need my uploads folder and that file to bring to a new install - I will grab that and spin up another VM on the same IP - how far should I go with the install before checking back in how to get this database imported?

I would also take a copy of your pgdata folder, you may not need to import the sql if that comes along too. the sql just means you can easily move the data between postgresql versions if the pgdata dir is incompatible with it's new host

@Rob Guthrie I am getting the same error on an 18.04 fresh install. Not sure what is causing this, but I have not copied anything over from the other machine. Following the install instructions at https://github.com/loomio/loomio-deploy - the only thing I did differently is I installed docker directly following instructions at https://docs.docker.com/install/linux/docker-ce/ubuntu/ because I was getting some errors about not finding the docker binary using the snap install. This is all fresh and clean install.

root@loomio:~/loomio-deploy# docker-compose run app rake db:setup

..... lots of scrolling text ...

7f586c967000-7f586c968000 rw-p 00000000 00:00 0

7fff70f43000-7fff71742000 rw-p 00000000 00:00 0 [stack]

7fff717e1000-7fff717e4000 r--p 00000000 00:00 0 [vvar]

7fff717e4000-7fff717e6000 r-xp 00000000 00:00 0 [vdso]

ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

[NOTE]

You may have encountered a bug in the Ruby interpreter or extension libraries.

Bug reports are welcome.

For details: https://www.ruby-lang.org/bugreport.html

SEGV received in ILL handler

what cpu architecture is your server? I wonder if this is 32bit or 64bit issue

root@loomio:~/loomio-deploy# uname -a

Linux loomio 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x8664 x8664 x86_64 GNU/Linux

Sorry, I'm stumped right now. I'll keep thinking. Your .env file uses "latest" for the loomio image tag right? These images are running well on my boxes with the sameish uname results.

 4.18.0-25-generic #26-Ubuntu SMP Mon Jun 24 09:32:08 UTC 2019 x8664 x8664 x86_64 GNU/Linux

Nothing in the env file says anything about image tag or "latest".

no, the "dot env" file ie: .env

root@loomio:~/loomio-deploy# more .env
LOOMIO_CONTAINER_IMAGE=loomio/loomio
LOOMIO_CONTAINER_TAG=latest
MAILIN_CONTAINER_IMAGE=loomio/mailin-docker
MAILIN_CONTAINER_TAG=latest
POSTGRES_CONTAINER_TAG=9.6

I have seen this issue recently with someone else. So It's a thing. I'll see if I can find some energy to look properly over the next few days.

I am poking around in the dark a bit, could it be the ruby version? Whatever it is the update broke it in the same way on the old box as the new fresh 18.04 install.

is it a cloud VPS?

It is a KVM virtual machine running on a bare metal server I have in a DC

Oh I know! Maybe you can just build it yourself!

put a copy of the loomio repo into your loomio-deploy folder:

git clone [email protected]:loomio/loomio.git (or maybe the https url?)

and modify docker-compose.yml to look a bit like this (I've added build and commented image)

hmmm I guess I can go down that road, but then updates will be a hassle it would seem. Although this update has brought the server down and not sure I can recover it so hard to be any worse I guess.

It’s super easy to update the source. Hit pull origin master. And you can always switch back to an image later if they start working for you.

okay I will give it a shot - sorry to take your time and I appreciate your availability. I was an original contributor to the project in the first crowd funding and I love Loomio. I am always trying to get orgs I work with to use it, and I hope I can get this back up in time to not have people think it is a bad idea to use it.

Keep those backup files safe and we can do it. The migrations might need hand holding but I know it’s possible.

@Rob Guthrie

So here is an issue - the command that is failing is:

docker-compose run app rake db:setup

I may not be following the logic completely, but I do not think this is even spinning up (sorry I do not have a lot of experience with what terms to use for docker components) worker, it spins up loomio-redis and loomio-db and then it gets an illegal instruction at

root@loomio:~/loomio-deploy# docker-compose run app rake db:setup

Creating loomio-redis ... done

Creating loomio-db ... done

/usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:112: [BUG] Illegal instruction at 0x00007fd0f83b97a2

ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]

-- Control frame information -----------------------------------------------

c:0084 p:---- s:0462 e:000461 CFUNC :open

c:0083 p:0022 s:0456 e:000455 BLOCK /usr/local/bundle/gems/ffi-1.11.1/lib/ffi/library.rb:112 [FINISH]

If I am understanding what you asked me to do above, I am replacing the container of loomio-worker with some local build command, but I am erroring out before we ever run worker. Were you expecting me to change all the containers with local build commands? Apologies this is all a bit new to me. I realize I may be missing something simple.

Am I following? And does it make sense that the above change made no difference? I get the same thing when I try to build the DB.

It leaves loomio-redis and loomio-db up, I need to close them down.

[NOTE]

You may have encountered a bug in the Ruby interpreter or extension libraries.

Bug reports are welcome.

For details: https://www.ruby-lang.org/bugreport.html

SEGV received in ILL handler

root@loomio:~/loomio-deploy# docker-compose down

Stopping loomio-deployapprun_1 ... done

Stopping loomio-db ... done

Stopping loomio-redis ... done

Removing loomio-deployapprun_1 ... done

Removing loomio-db ... done

Removing loomio-redis ... done

Yes. Swap both Loomio containers to build and leave others as is. If you’re getting the same error as the images that’s interesting but sad

@Rob Guthrie

Okay, so I am making progress. I changed app and worker to build, and that seemed to be enough to get everything to start up. I tried with just app, but worker was crashing, so I changed that back to build. I can now get a login screen on the app when I start it.

Prior to starting it I tared up the pgdata folder and moved it over to the new server and replaced the pgdata folder, and then ran the rake migrate command which seemed to work (had never worked before) and then copied the env file over from the old server. I tarred up and replaced the upload folder.

I was getting errors and would not start with the new env file, and I looked at it and decided I should copy the DeviseSecret and SecretCookieToken entries to the old env file from the new one and that worked.

Now I get a login screen, which allows me to use google to login as I could on the old server, but after I login I get a "Something went wrong" loomio screen.

Feel like I am getting close.

On closer examination I do not believe the app rake migrate command is actually working. I deleted pgdata and reverted back to the old database several times now. When I run the commmand:

docker-compose run app rake db:migrate

it starts to do the migration:

Migrating to AddAnnouncements (20171114224912)
== 20171114224912 AddAnnouncements: migrating =================================
-- create_table(:announcements, {:id=>:integer})
   -> 2.5373s
== 20171114224912 AddAnnouncements: migrated (2.5375s) ========================

Migrating to AddAuthorToAnnouncement (20171115232230)
== 20171115232230 AddAuthorToAnnouncement: migrating ==========================
-- add_column(:announcements, :author_id, :integer, {:default=>false, :index=>true})
   -> 2.3210s
== 20171115232230 AddAuthorToAnnouncement: migrated (2.3212s) =================

Then it errors out:

Migrating to UpdateMeetingScores (20180416223444)

== 20180416223444 UpdateMeetingScores: migrating ==============================

rake aborted!

StandardError: An error has occurred, this and all later migrations canceled:

unknown attribute 'score_counts' for PollOption.

/usr/local/bundle/gems/activemodel-5.2.3/lib/activemodel/attributeassignment.rb:53:in `assignattribute'

When I just spin up the server after this partial migration, the server will run, and it lets me login with google (the email login button doesn't work) and I get the "something went wrong" screen.

A couple of times going through various slightly different order of operations I have gotten it to log me in and then it thinks I am a new user (so obviously the database is not getting imported).

Sorry for the mess, but I have certainly gotten much closer now, though we still seem to be a bit off.

Ok great!! Congratulations!! Almost there. I’ll pick this up tomorrow

@Rob Guthrie

Thanks. I have been trying various things to get this to work, might need to start fresh on the new install but I believe I have been following the logic of what is happening so it should be okay though I have made many modifications to env files and deleted and restored pgdata many times and have been running it in various states trying to figure out what is going on. Often I end up in a state where it looks like the database server is either not starting or it cannot be reached by the other containers.

I have daily backups of the database and they are even sent off the old server, so having the data from before any of this happened is no problem (I have a cron script running pgdump every day and backing up the uploads folder and mailing that all off of the server as well as copying it to a remote FTP).

@Rob Guthrie

I renamed the loomio-deploy on the new server and started fresh - I can get everything to run according to the install (seems to all be working - could be some failing going on but I don't see anything that jumps out to me in the logs) by replacing the containers for app and worker with a local build.

How to import the database from the old server and get it to do a migration etc., and the exact way to bring in what I need from the other env file (primarily the google authentication and my email settings) - are left to do. Also I assume I need to restore my uploads folder.

I have it set up and ready to run as a fresh install, will no modifications to anything, and it boots and runs as a clean install (asks me for email login and assumes I am a new user). Letsencrypt sets up the cert, everything seems good. I just have it sitting spun down as I do not want my users logging into it.

I have my env file from the old server, a tarred up version of the pgdata folder. I am afraid that database folder may have something wrong with it from trying to get the old server to do a migration and get it to work over there, but since app would never run from the containers I am not sure if the migration ever did anything (I do not believe it did). I also have a pgdump from prior to any issues when the old server was running fine (November 20th). I also have a tar of the uploads folder sitting on the new server ready to go.

I have the old server available and running on a different IP so I can grab anything from there that might be needed (and I could possibly try to local builds over there and see if I can get it to run, but at this point I thought it would be better to get the 18.04 server running).

I thought this would be the best state to start from to try to restore the old site. Let me know how you think it is best to proceed.

Thanks again for your help.

Right, the issue I'm thinking we need to focus in on is the migrations, right? You're saying your system works up until: 20180416223444 UpdateMeetingScores

That migration is only to adjust the values on timepolls, which we can do later if necessary anyway.

Reading on https://stackoverflow.com/questions/8810732/how-do-you-skip-failed-migrations-rake-dbmigrate I see that you can insert a record into the migrations table to say that it's already been run, essentially skipping the migration.

So are you able to use the open psql console command from the readme to run

insert into schema_migrations (version) values ('20180416223444');

and see if that does the trick?

I will give that a shot. The problem for me is that while I have quite a bit of experience at using databases from the CLI in a normal environment, I am often confused by the process of spinning up the docker containers and using commands from inside them to take action and when and why one chooses to do that in some situations.

Can you tell me the method you would use to get into the console? Do we do this just from the CLI of the host machine OS or is there some way I am spinning up a docker container and accessing the psql command line from there? I do not even have the psql tools installed on the host so I would need to do that if we are talking about manipulating the database from there, or else can you give me at least a starting point to build the command that would open a psql terminal interactively to my CLI with the docker container?

this uses psq in the container. from the loomip-deploy readme:

docker exec -ti loomio-db su - postgres -c 'psql loomio_production'

Hi, just wondering how you got on. You're so close now!

@Michael McKinsey I've just found a likely reason for the illegal instruction bug. The next version of the docker image should work for you.

@Rob Guthrie

Okay, this was quite an adventure. It looks like I have it all working now. The manual insert into the migrate was needed in order to finish the migration, and I ran app and worker as build instead of in the images. I ended up using the actual pgdata tar of the old database data rather than importing the backups.

It looks like everything is working as it should, I was in the middle of a rather time sensitive proposal with a group of people, so we dealt with emails and I am trying to get them to use loomio now, we shall see shortly if it is actually working correctly, but so far so good.

Thank you for all the help getting this problem solved, not sure why my database was broken so that it could not finish the migrate. I imagine that would have stopped me even if the new images had not been crashing.

I will try to keep my install more current, I certainly have learned more about how Loomio works on the internals than I ever wanted to know. The version 2 is different enough that at first I thought something was broken, but it seems to be working and looking at the loomio site, it is just a different layout in the new version. Looks good and the new features I was trying to get (change the invitation link, and remove a member from a group) seem to be working as hoped.

Your support and help with this has really been phenomenal. I don't know enough about the structure of the loomio team to know who you are, or why you were helping me, but whatever made that happen I am really grateful for the help and there is little/no chance I would have been able to save my install and keep using loomio without you helping me through this problem.

I'm the lead developer and designer on Loomio. There is one other part time developer who helps keep me sane. It's a 4 employee company.

There are not a lot of people running self hosted Loomio's, and I want to be sure to support those brave people who do attempt it, where I can.

We only managed to release Loomio 2 because of our user community reporting bugs and giving feedback during a long public beta period.

Thanks for using Loomio, and reporting your journey, you've improved things for the next person.