Federation layer/module system - define terms/requirements

So now that we've agreed that we need to put the federation into its own layer, separated from the web front-end, I think the next step is to define exactly what we mean by that, and start building up requirements and suggest possible solutions.

I'll get the ball rolling. First, some definitions (feel free to add/correct):

protocol: A protocol is any agreed upon method of communicating between two interested parties. In this case, I would include any API as a particular kind of "protocol" in the general sense.

client: in any situation where there is a request/response interaction going on between two parties, the party making the request is the "client".

federation layer: The top level protocol that allows diaspora pods, client applications, and other social networks to talk to each other and discover what protocols are attached to a user or the pod itself, independent of any user. We say that the federation layer "links" to other protocols. The thing about which this post is written.

protocol link: A link to a particular protocol or version of an actionable protocol. Given knowledge of the protocol and a generic "link" to an appropriate endpoint (a plain URI), a client of that protocol should be able to do something useful with it.

format protocol: A protocol that defines the structure of some particular content irrespective of the context that content is being used. ATOM, JSON, Activity Streams, Salmon I would consider to be format protocols, because they are not actionable by themselves - only when used in conjunction with another protocol. A particular protocol MAY support more than one content format protocol, but this is optional. A format protocol MAY be layered on top of one or more other format protocols, e.g. Activity Streams via JSON or ATOM.

protocol group: a collection of protocols that should be considered together as a unit. This does not include protocols that have their own mechanism for aggregating other protocols (such as OStatus, or the federation layer protocol itself). This is a possible convenience feature of the federation layer protocol to group and version a collection of protocol links when that collection doesn't have it's own aggregation protocol.

Here are the requirements I can think of. Again, please add/correct as you see fit:

The federation layer MUST be able to reference nearly any type of protocol, even protocols that are not aware of the federation layer. This includes non HTTP-based protocols. For instance, it should allow you to link to a telnet server, or an XMPP server, or an instance of an OpenSim server - you name it. If you can make a URI for it, it should be supported.

The federation layer MUST support multiple versions of any particular protocol or protocol group. That means it must have a means of identifying a protocol uniquely by type and version.

The federation layer MUST NOT expect any particular protocol to be present except itself. For instance, authentication should be outside the scope of the federation layer, and should instead be a linked protocol underneath it.

Linked protocols MAY have dependencies on other linked protocols, but the federation layer MUST NOT be responsible for facilitating these dependencies. Instead, this MUST be facilitated by the client and the protocols involved. For instance, one protocol may require an authorization token, which an authorization protocol can provide. The client must then retrieve the token from one protocol and pass it to another. A client may even pass a protocol link to another protocol.

How would, as an example, Webfinger be situated in relation to:

1 federation layer
2 protocol link
3 format protocol
4. protocol group
?

Woah, slow down. It's not like that we're currently a team of a hundred people working on it and need that clear communication. The first thing we need to do is in this sector is separating the code into a layer, really on a code level, not a conceptual one. This is the one required step to even think about the next ones. Putting the code into a separate layer allows to do isolated testing without any influence by the application using it, it would even allow to build a suite of test servers spawning up some EC2 instances exchanging a predefined queue of messages and put expectations and requirements on the protocol in an automated way. When we have something like this we can start identifying the real problems in the protocol, try to reproduce issues we see on production deploys etc. Then we can start thinking on howto improve it, then we can start thinking on howto structure this layer, ideally without changing it's public API (code level API, not protocol, not client!).

If we have the code level layer the next step is still not changing it though, but to make it versioned, find ways for discovery of the capabilities of the remote end. Otherwise we can't make breaking changes without leaving all the outdated installations behind. This issue becomes even more critical with us doing version releases of Diaspora now.

So, my brainstorming for the next immediate steps looks like this atm:

1. Move all federation related stuff under one module, including Webfinger. (unify logging of federation events)
2. Move federation related stuff from AR models into dedicated (proxy) objects.
3. Mechanism for deferred processing, maybe just via the delegator pattern.
4. AR model creation through callbacks which have an input defined by the layer but logic defined by the app. If possible a single "adapter" file.
5. Refactor webfinger/hcard to a genric identity discovering API not bound to the technology, internal API wise, do not change the actual mechanism yet.
6. Create a single object defining the needed API to talk between the layer and the app, do not make any calls from the app deeper into the layer. Do not make any call from the layer into the app.
7. Create a rails engine (or do it devise style) to provide the routes for .well-known/host-meta, webfinger, the hcard and receive, in short PublicsController. I think the atom feed is part of oStatus so that too maybe.
8. Make everything a gem.

Single, isolated "baby" steps.

Lets not make terms upfront without knowing exactly what's going to happen. Lets keep Diaspora's development spirit and find out what needs to be done just right before doing it, not one or two years earlier.

I've been looking at message notifications due to a bug I've hunted down. I don't know much about the other code but it made me wonder what is the boundary of federation refactoring? For example right now local notifications and remote notifications follow almost exactly the same path through dispatch.rb. I can sort of see why this was done, but the fact is that resque messages are enqueued for notifications even when a private message is sent only to people on the local pod. That shouldn't need to happen just to maintain the abstraction and it's a bit expensive performance-wise. Somehow we'd want things like notification to be receivers of messages that could be generated locally or remotely - and that message interface in the application really shouldn't need an async layer just to conform to the same interface. I guess what I'm trying to say is that in this case I'd propose re-factoring to go further than just extracting the remote interface, because that maybe would leave some really confusing stuff in its wake.

We actually have a second route for local delivery going through resque: local_dispatch. This one creates the ShareVisibillities of a new post. It was done since for large pod it might need several hundred, for things like Diaspora HQ even thousands, of objects to be created. Separating these kind of stuff from the federation layer while doing the extraction sounds reasonable to me, I agree.

Hi, I'm new here.
Seeing as how I'm new to this software and still trying to define some context, I would like to ask that anyone stop me if I'm intruding or lowering the 'intelligence' level of the conversation.
I agree with what SleepyDaddy is saying here (possibly because I'm still learning what you all mean with some of your terminology/jargon as it relates to this software). It reminds me of rfc's. Yes, I used to be a network geek. It seems to me that one of the advantages of the rfc system and defining communication protocols to that level of detail, is that it saves a lot of the semantic problems in discussions like these.
I would be glad to volunteer for boring and repetitive typing of documentation. It would help me understand the system and help get everyone on the same page. All it requires is that people put up with the occasional silly question.
=D

Hello. Is implementing the federation layer as an external application/service makes any sense? I would like to help, but I don't use Ruby/Rails.

a railsgirls team implemented a federation test-app last summer using the new gem I wrote to test interoperability of the old server code with the code from the gem.
I think that could be a place to start - the code is simple, and written as cucumber features, so there is always a scenario behind each test case.
see: https://github.com/Team-D/federation-testbed

@jonnehass do we btw have a tracker for this comment above: https://www.loomio.org/d/JdU0ns1x/federation-layer-module-system-define-terms-requirements#comment-6108 ?

Not that I remember one.