In JGroups, messages are delivered in the order in which they were sent by a given member. So when member X sends messages 1-3 to the cluster, then everyone will deliver them in the order X1 -> X2 -> X3 ('->' means 'followed by').
When a different member Y delivers messages 4-6, then they will get delivered in parallel to X's messages ('||' means 'parallel to'):
X1 -> X2 -> X3 || Y4 -> Y5 -> Y6
This is good, but what if X has 100 HTTP sessions and performs session replication ?
All modifications to the sessions are sent to the cluster, and will get delivered in the order in which they were performed.
The problem here is that even updates to different sessions will be ordered, e.g. if X updates sessions A, B and C, then we could end up with the following delivery order (X is omitted for brevity):
A1 -> A2 -> B1 -> A3 -> C1 -> C2 -> C3
This means that update 1 to session C has to wait until updates A1-3 and B1 have been processed; in other words, an update has to wait until all updates ahead of it in the queue have been processed !
This unnecessarily delays updates: since updates to A, B and C and unrelated, we could deliver them in parallel, e.g.:
A1 -> A2 -> A3 || B1 || C1 -> C2 -> C3
This means that all updates to A are delivered in order, but parallel to updates to B and updates to C.
How is this done ? Enter the SCOPE protocol.
SCOPE delivers messages in the order in which they were sent within a given scope. Place it somewhere above NAKACK and UNICAST (or SEQUENCER).
To give a message a scope, simply use Message.setScope(short). The argument should be as unique as possible, to prevent collisions.
The use case described above is actually for real, and we anticipate using this feature in HTTP session replication / distribution in the JBoss application server !
More detailed documentation of scopes can be found at [1]. Configuration of the SCOPE protocol is described in [2].
This is yet an experimental feature, so feedback is appreciated !
[1] Scopes
[2] The SCOPE protocol
Hm, SCOPE sounds fine, but what's wrong with sending http session replication messages as OOB? SCOPE should be fine when having multiple caches using the same JChannel. Each cache will receive its own scope and they do not block each other when sending messages.
ReplyDeleteOOB also delivers messages from the sender concurrently, but there is no ordering whatsoever. If you use synchronous replication, then this is not a problem, because the next RPC will only be invoked when we have all results, but with asynchronous replication, ordering is not guaranteed. I assume that, in most cases, modifications to a given session should be delivered in the order in which they were performed.
ReplyDeleteYes, multiple caches is another good use case for scopes. We plan to do that in Infinispan, for example.
ReplyDeleteexcellent feature Bela!
ReplyDeleteI bet we'll use it :-)
Cool, feedback always appreciated !
ReplyDelete