@flaburgan - as a non-techy, what would separating the federation from the rest of the code involve?

ok, sorry just saw that separate discussion and it seems to be something that has been agreed on pretty unanimously. Ignore comment!

@nickdowson no problem ;)

I see the link in my vote didn't catch. Here it is properly:

Put federation into a layer

Just to show there is already an agreement to do this, so L3MNcakes's practical proposal of forming a working group builds on this previous agreement.

@seantilleycommunit :: Good points, but not necessarily show stoppers. If we could even get 2-3 devs working on this to start out, I think we could take it a long way. I'm one! =) It would be nice if we had a developer who knows the code well to help out, but in the event that doesn't happen, I'll happily be the one to tear through the code until I understand what's going on. I would love to see some of the newer devs jump in to help figure it out too! I know the task is a daunting one, but it has to be done by somebody if we want our app to succede. This is one of the core pieces of our network and it's really holding us back on the features we can implement. I really hate telling users, "We can't do that because the distributed part of our distributed network sucks." I'd much rather start moving forward on this with limited manpower than continue waiting for a magical influx of dev power.

@l3mncakes If you want to take a look at our federation system, by all means take a look.

Here's some wiki entries that may be useful to anyone interested:

• Federation Message Semantics

• Federation Protocol Overview

• Diaspora Message Passing

@seantilleycommunit Sweet! Thanks for listing those resources here. Should be very helpful.

Come on guys, do we really need a proposal to make a plan? :)

Absolutely, Tom. It's a means of agreeing a plan and getting something done.

@tomscott haha, I thought it would be best way to get the ball rolling on this and since it purposes doing things a little differently than I've seen them done before, I wanted to make sure the community had a say in the decision. C=

meanwhile on github:
https://github.com/Raven24/diaspora-federation

;)

I hope nobody feels left out by me starting ahead on a clean-room implementation for federation... but it really is not such a complicated or huge task for it to require elaborated documents or planning.

My plan of stuff to implement for this gem is:
- DO NOT BUILD ON RAILS (bonus: lightning fast specs) [done]

• get xml de/serialization on database-unrelated entities to work in both directions [done]

• build a structure resembling what is currently known as 'to_diaspora_xml' [done]

• put payload in salmon envelope and sign/encrypt that as it is currently done in D* [pending]

• simulate sending/receiving inside a proof-of-concept test-app [pending]

@florianstaudacher Really cool stuff! Watching, will keep an eye on this. Do you think the availability of federation in a gem might, in theory, allow a developer the ability to incorporate Diaspora federation into just about any Rails app? :)

@florianstaudacher Cool! Thanks for jumping on that. I don't think planning or documentation is ever a bad thing, and it also gives an opportunity for others to learn new stuff and help out if they want, but I appreciate that you're taking the time to get stuff done.

Fantastic work, Florian.

@florianstaudacher thank you for dealing with that!

@seantilleycommunit in theory, yes, you could re-use the gem in other ruby projects. but keep in mind that the gem just represents the "business logic". you'd have to handle the http routes and the database persistence in the app - that's really out of scope for the gem.

• put payload in salmon envelope and sign/encrypt that as it is currently done in D* [done]

;)

Started trying to put together some better documentation around this. Currently working off the articles Sean posted here and trying to make them more readable and useful. Not done by any means yet, but just a heads up on what I'm doing.

https://docs.google.com/document/d/1QEhIx10zFnBgAtf_GL6AMT9GvWjxfPE28QFb2rPIbHU/edit?usp=sharing

alright, the code in my github repo should now have every entity we use for federation, also the ones with nested entities inside (e.g. StatusMessage -> Photo, Location).
they can be de-/serialized from/to XML and are meant to be put in a Salmon envelope that can be de-/encrypted.
I still want to implement some rudimentary validation classes, so the received objects can be sanity-checked before they are being handled by an actual app, but apart from that I think we could start to think about how to test this on its own and with another instance of D* running the current federation code.

@florianstaudacher please ping me when you think the code is stable enough to be pulled on my pod.

... it will need a lot more work before we can actually run a pod with it.
not only do we have to give it a decent test period, also a good part of the diaspora code will have to be adapted just to use the new gem.

@florianstaudacher I'll be more than happy to help you with this. I'll try to give a look at it this weekend, just let me know what needs to be done

Well, to anyone who would like to help, I can suggest to review my code and the docs I write with it.

I may be an intermediate-advanced programmer but that certainly doesn't mean everything I write is pure gold. I want this to be as good as it can get, so we have a rock-solid library we can use for Diaspora* and possibly to have something other projects could use to integrate federation into their software.

Anyway, this will become the new reference implementation of our protocol, so this has to be impeccable, readable and understandable.

@florianstaudacher Would it be possible to move these updated docs into the federation section of our wiki at some point? It'd be really great to brush up what little information we have on how the protocol works, and update it with all the information you've put together. :)

Hey, without understanding the technical stuff, it seems like there's been a lot of progress with this.
Seems to me like an agreed standard for decentralised social network federation should be a major outcome that the diaspora project should be working towards - and that this needs to be something that can be used by other social networks. I think it would be really good to work with a few other distributed social networks (movim and gnustatus come to mind as to my mind projects with potential) from an early stage to develop a protocol that can be flexibly and easily extended and some kind of process/foundation that becomes the agreed way of agreeing and improving this.
Now might be a good time to start thinking about this could be made happen? Working with groups such as the free network foundation (thefnf.org), the free software foundation, the electronic frontier foundation and others might be a way forward?

On a totally different note, any thoughts on how diaspora should deal with linking to profiles? On facebook etc. someone can give the http address to their profile which anyone can click on, look at and then press to subscribe/friend/etc. On diaspora this is more complicated if the profile is hosted on a different pod - plus being logged in or out and consistency of how sites look becomes a bit of an issue. Plugins or third party cookies might be a way of solving this but bring up obvious privacy issues... (sorry if I've not explained this issue very well...)

Nick, you can use http://dia.so/ to get a URL for your profile which will work on any pod.

ah, thanks @goob I didn't realise this, will check that out!

It's still a third-party tool ;) Something like this should be in the future integrated to the Foundation(tm) and officially supported in the code.

Also I'm fantasizing of a link in posts to take the user to "the post on my pod". For example when someone gives me a link to a post on another pod, I click it, I cannot reply or reshare unless I find it on my pod. Just needs some ajax query, and of course post must exist on the other pod.

Has this descision https://www.loomio.org/discussions/612?proposal=463 been worked into GitHub issues? Or how will it be moved into actionable items?

@wilhelm , Florian's code, linked on this thread, is the start of the solution to the proposal you link to. See https://github.com/Raven24/diaspora-federation

alright, I'm quasi-finished with what I had planned to include in the gem.
Only one more design/architectural issue I'd like to bring up and solve with the wisdom of the crowd:

We have a lot of XML in the protocols around D*, so I have chosen the 'ox' gem to handle XML parsing and generation. It promises to be fast and has all the features we need...
Except for one occasion, where we use hCard - which is HTML and uses classes to label its data fields inside standard HTML elements. Ox doesn't know CSS or XPath selectors, so I can't select the elements based on the used class names directly. At that point I brought in libxml-ruby, which has support for XPath, which is enough to solve the problem in this case.

Personally I feel it's unclean to include another lib just for that one use case, and now I'm torn between leaving it like that, or just switching to libxml or possibly nokogiri entirely...

Ideas?

I think we should switch to Nokogiri. While it may not be the fastest library, it's the most feature complete and battle tested one, so using it should be future proof.

@florianstaudacher -- I don't really know much about the various libs themselves, but I agree that it feels dirty to include a separate lib for just that case. My first instinct is to modify ox so it could support that use case and then we only have to use one lib. Would this be feasible?

If you feel that ox performs better for the other cases, it might be worth considering whether Diaspora's hCard could be parsed using XML. I've had a look online and found suggestions that this is possible with hCard. This may not be a good idea in Diaspora's case, or may not be a clean solution in itself, but if you'd prefer to use ox, it might be worth considering whether it could work.

Apologies if I'm speaking out of turn, not really understanding the technical issues. It just struck me as a potential alternative approach to switching libraries, if you really like that particular gem.

Well, as far as performance goes, our XML documents are at most a few KB in size, so any speed variances should be almost undetectable.

Modifying Ox to support advanced selectors is way out of scope for this task and I don't think it's in the spirit of the lib - being mostly fast and lightweight.

I thought about parsing the hCard as XML, but I decided against it, since the idea behind XML is structure and hCard as such has no structure.

I guess I'll be moving to Nokogiri, it seems the least controversial... ;)

(in case anybody missed my post on d*)

okay, porting to nokogiri is done. now it's time to give the code a thorough review. so if you have any ruby knowledge, look it through and give me feedback on anything that might jump in your eyes.

documentation and readme will follow later

I've just started a new discussion with some ideas for adding the possibility of pulls to Diaspora's push model for federation. I didn't want to start a new discussion on this thread, which might confuse things.

Hope it's useful.

I created a new discussion for what protocol we use & working with other social networks:

https://www.loomio.org/discussions/6233

another update: the Readme contains more infos, and I think the docs should be mostly finished.
So if you get a chance, you can have a look at the docs (http://rdoc.info/github/Raven24/diaspora-federation/master/frames) and tell me if everything is understandable and usable for implementing the gem.