Sunday, October 06, 2013

JGroups 3.4.0.Final released

I'm happy to announce that JGroups 3.4.0.Final has been released !

The major features, bug fixes and optimizations are:
  • View creation and coordinator selection is now pluggable
  • Cross site replication can have more than 1 site master
  • Fork channels: light weight channels
  • Reduction of memory and wire size for views, merge views, digests and various headers
  • Various optimizations for large clusters: the largest JGroups cluster is now at 1'538 nodes !
  • The license is now Apache License 2.0
3.4.0.Final can be downloaded from SourceForge [1] or Maven (central). The complete list of issues is at [2]. Below is a summary of the changes.
Enjoy !

[1] https://sourceforge.net/projects/javagroups/files/JGroups/3.4.0.Final
[2] https://issues.jboss.org/browse/JGRP/fixforversion/12320869


Changes:
****************************************************************************************************
Note that the license was changed from LGPL 2.1 to AL 2.0: https://issues.jboss.org/browse/JGRP-1643
****************************************************************************************************


New features
============


Pluggable policy for picking coordinator
----------------------------------------
[https://issues.jboss.org/browse/JGRP-592]

View and merge-view creation is now pluggable; this means that an application can determine which member is
the coordinator.
Documentation: http://www.jgroups.org/manual/html/user-advanced.html#MembershipChangePolicy.


RELAY2: allow for more than one site master
-------------------------------------------
[https://issues.jboss.org/browse/JGRP-1649]

If we have a lot of traffic between sites, having more than 1 site master increases performance and reduces stress
on the single site master


Fork channels: private light-weight channels
--------------------------------------------
[https://issues.jboss.org/browse/JGRP-1613]

This allows multiple light-weight channels to be created over the same (base) channel. The fork channels are
private to the app which creates them and the app can also add protocols over the default stack. These protocols are
also private to the app.

Doc: http://www.jgroups.org/manual/html/user-advanced.html#ForkChannel
Blog: http://belaban.blogspot.ch/2013/08/how-to-hijack-jgroups-channel-inside.html



Kerberos based authentication
-----------------------------
[https://issues.jboss.org/browse/JGRP-1657]

New AUTH plugin contributed by Martin Swales. Experimental, needs more work


Probe now works with TCP too
----------------------------
[https://issues.jboss.org/browse/JGRP-1568]

If multicasting is not enabled, probe.sh can be started as follows:
probe.sh -addr 192.168.1.5 -port 12345
, where 192.168.1.5:12345 is the physical address:port of a node.
Probe will ask that node for the addresses of all other members and then send the request to all members.



Optimizations
=============

UNICAST3: ack messages sooner
-----------------------------
[https://issues.jboss.org/browse/JGRP-1664]

A message would get acked after delivery, not reception. This was changed, so that long running app code would not
delay acking the message, which could lead to unneeded retransmission by the sender.


Compress Digest and MutableDigest
---------------------------------
[https://issues.jboss.org/browse/JGRP-1317]
[https://issues.jboss.org/browse/JGRP-1354]
[https://issues.jboss.org/browse/JGRP-1391]
[https://issues.jboss.org/browse/JGRP-1690]

- In some cases, when a digest and a view are the same, the members
  field of the digest points to the members field of the view,
  resulting in reduced memory use.
- When a view and digest are the same, we marshal the members only
  once (in JOIN-RSP, MERGE-RSP and INSTALL-MERGE-VIEW).
- We don't send the digest with a VIEW to *existing members*; the full
  view and digest is only sent to the joiner. This means that new
  views are smaller, which is useful in large clusters.
- JIRA:  https://issues.jboss.org/browse/JGRP-1317
- View and MergeView now use arrays rather than lists to store
  membership and subgroups
- Make sure digest matches view when returning JOIN-RSP or installing
  MergeView (https://issues.jboss.org/browse/JGRP-1690)
- More efficient marshalling of GMS$GmsHeader: when view and digest
  are present, we only marshal the members once


Large clusters:
---------------
- https://issues.jboss.org/browse/JGRP-1700: STABLE uses a bitset rather than a list for STABLE msgs, reducing
  memory consumption
- https://issues.jboss.org/browse/JGRP-1704: don't print the full list of members
- https://issues.jboss.org/browse/JGRP-1705: suppression of fake merge-views
- https://issues.jboss.org/browse/JGRP-1710: move contents of GMS headers into message body (otherwise packet at
  transport gets too big)
- https://issues.jboss.org/browse/JGRP-1713: ditto for VIRE-RSP in MERGE3
- https://issues.jboss.org/browse/JGRP-1714: move large data in headers to message body



Bug fixes
=========

FRAG/FRAG2: incorrect ordering with message batches
[https://issues.jboss.org/browse/JGRP-1648]

Reassembled messages would get reinserted into the batch at the end instead of at their original position


RSVP: incorrect ordering with message batches
---------------------------------------------
[https://issues.jboss.org/browse/JGRP-1641]

RSVP-tagged messages in a batch would get delivered immediately, instead of waiting for their turn


Memory leak in STABLE
---------------------
[https://issues.jboss.org/browse/JGRP-1638]

Occurred when send_stable_msg_to_coord_only was enabled.


NAKACK2/UNICAST3: problems with flow control
--------------------------------------------
[https://issues.jboss.org/browse/JGRP-1665]

If an incoming message sent out other messages before returning, it could block forever as new credits would not be
processed. Caused by a regression (removal of ignore_sync_thread) in FlowControl.



AUTH: nodes without AUTH protocol can join cluster
--------------------------------------------------
[https://issues.jboss.org/browse/JGRP-1661]

If a member didn't ship an AuthHeader, the message would get passed up instead of rejected.


LockService issues
------------------
[https://issues.jboss.org/browse/JGRP-1634]
[https://issues.jboss.org/browse/JGRP-1639]
[https://issues.jboss.org/browse/JGRP-1660]

Bug fix for concurrent tryLock() blocks and various optimizations.


Logical name cache is cleared, affecting other channels
-------------------------------------------------------
[https://issues.jboss.org/browse/JGRP-1654]

If we have multiple channels in the same JVM, the a new view in one channel triggers removal of the entries
of all other caches




Manual
======

The manual is at http://www.jgroups.org/manual-3.x/html/index.html.

Monday, September 30, 2013

New record for a large JGroups cluster: 1538 nodes

The largest JGroups cluster so far has been 536 nodes [1], but since last week we have a new record: 1538 nodes !

The specs for this cluster are:
  • 4 X Intel Xeon CPU E3-1220 V2 @ 3.10GHz 4GB: 8 boxes of 13 members = 104
  • 1 X Intel Celeron M CPU 440 @ 1.86GHz 1GB: 8 boxes of 3 members = 24
  • 1 X Intel Celeron M CPU 440 @ 1.86GHz 2GB: 141 boxes of 10 members =1410
    • Total = 1538 members
  • JGroups 3.4.0.Beta2 (custom build with JGRP-1705 [2] included)
  • 120 MB of heap for each node
The time to form a 1538 node cluster was around 115 seconds and it took a total 11 views (due to view bundling being enabled).

Note that the physical memory listed for the hardware above is shared by all the nodes on that box, e.g. in the last HW config of 141 boxes, 10 nodes share the 2GB of physical memory; each member has 120 MB max of heap available.

The picture below shows the management GUI. The last line shows member number 1538, which took roughly 11 seconds to join (due to view bundling) and the total time for the cluster to form was ca. 115 seconds.




[1] http://belaban.blogspot.ch/2011/04/largest-jgroups-cluster-ever-536-nodes.html
[2] https://issues.jboss.org/browse/JGRP-1705

Wednesday, August 21, 2013

How to hijack a JGroups channel inside Infinispan / JBoss ... (and get away with it !)

Have you ever used an Infinispan or JBoss (WildFly) cluster ?
If no --> skip this article.

Have you ever had the need to send messages around the cluster, without resorting to RpcManager offered by Infinispan or HAPartition provided by WildFly ?
If no --> skip this article.

Have you ever wanted to hijack the JGroups channel used by Infinispan and WildFly and use it for your own purposes ?
If no --> skip this article.

I recently added light-weight channels [1] to JGroups. They provide the ability to send and receive messages over an existing channel, yet those messages are private to the light-weight channel. And the hijacked channel doesn't see those private messages either.

This is good when an application wants to reuse an existing channel, and doesn't want to create/configure/maintain its own channel, which would increase resource consumption.

Also, applications can create many (hundreds) of light-weight channels, as they don't use a lot of resources and are easy and quick to create and destroy.

Even better: if an application would like to add capabilities to an existing stack, e.g. atomic counters, a distributed execution service or distributed locking, then it can define the protocol(s) it wants to add and the private channel will add these. Again, the hijacked channel is unaffected by this.

There's documentation on this at [1], so let me show you how to hijack a channel inside of an Infinispan application. The application can be downloaded from [3].

The (original but edited) code that creates an Infinispan cache looks like this:

    protected void start() throws Exception {
        mgr=new DefaultCacheManager("infinispan.xml");
        cache=mgr.getCache("clusteredCache");

        eventLoop();
    }


It creates a CacheManager, then creates a Cache instance off of it. This is it. Now, we want to grab the JGroups channel and hijack it to run a Draw instance on it (HijackTest:53, [4]):


    protected void start() throws Exception {
        mgr=new DefaultCacheManager("infinispan.xml");
        cache=mgr.getCache("clusteredCache");
        Transport   tp;

        ForkChannel fork_ch;
        tp=cache.getAdvancedCache().getRpcManager().getTransport();
        Channel main_ch=((JGroupsTransport)transport).getChannel();
        fork_ch=new ForkChannel(main_ch, 

                                "hijack-stack", 
                                "lead-hijacker",
                                true,

                                ProtocolStack.ABOVE,
                                FRAG2.class);
        Draw draw=new Draw((JChannel)fork_ch);
        fork_ch.connect("ignored");
        draw.go();

        eventLoop();
    }


The code in bold was inserted. So what does it do ? It grabs the JGroups channel from the Infinispan cache, creates a light-weight (fork) channel  and runs Draw [5] over it. Draw is a replicated whiteboard GUI application. To explain this step-by-step:
  • First the Transport instance is retrieved from the cache, then the JGroups channel from it
  • Next a ForkChannel is created. It is passed the hijacked JGroups channel, the name of the newly created fork stack ("hijack-stack") and the name of the fork channel ("lead-hijacker"). Then we state that we want to dynamically insert a FORK protocol if it doesn't exist, above FRAG2. FORK [2] is needed to create fork channels
  • Next, we create an instance of Draw (shipped with JGroups) on the newly created fork channel
  • Then we connect the fork channel and call go() on Draw which starts up the GUI
The result is shown below:




The hijacked application is a text-based Infinispan perf test (shown in the 2 shell windows). The hijacking Draw windows are shown above. Pressing and moving the mouse around will send multicasts of coordinates/color around the cluster over the fork channel, but the hijacked channel will not notice anything.

Seem like we got away with hijacking the JGroups channel after all ! :-)



[1] http://www.jgroups.org/manual/html/user-advanced.html#ForkChannel

[2] https://issues.jboss.org/browse/JGRP-1613

[3] https://github.com/belaban/HijackTest

[4] https://github.com/belaban/HijackTest/blob/master/src/main/java/org/dubious/HijackTest.java#L53

[5] https://github.com/belaban/JGroups/blob/master/src/org/jgroups/demos/Draw.java


Wednesday, July 31, 2013

BYOC: Bring Your Own Coordinator

I've finally resolved one of the older issues in JGroups, created 6 years ago: pluggable policy for picking the coordinator [1].

This adds the ability to let application code create the new View on a join, leave or crash-leave, or the MergeView on a merge.

Why is this important ?

There are 2 scenarios that benefit from this:
  1. The new default merge policy always picks a previous coordinator if available, so that moving around of the coordinatorship is minimized. Because singleton services (services which are instantiated only on one node in the cluster, usually on the coordinator) need to shutdown services and start them every time the coordinatorship moves, minimizing this means less work for singletons.
  2. A node with a coordinator role has more work to do, e.g. join/leave/merge handling, plus it may be running some singleton services. Some systems will therefore prefer bulkier machines (faster CPU, more memory) to run the coordinator. But so far, this hasn't been possible, as JGroups always preferred every node equally. This changes now, and the puggable policy allows for pinning the coordinatorship to a number of machines (a subset of the cluster nodes).
The documentation is at [2]. For questions and feedback, please use the mailing list. 
Cheers,

[1] https://issues.jboss.org/browse/JGRP-592

[2] http://www.jgroups.org/manual/html/user-advanced.html#MembershipChangePolicy

Monday, July 29, 2013

First JGroups release under Apache License 2.0

I'm happy to announce that I just released 3.4.0.Alpha1, which is the first release under the Apache License 2.0 !

In order to change the license, I had to track down all contributors who still have code in the code base and get their permissions [1], which was time consuming.

Given that JGroups came into existence in 1998, even though a lot of code has since been added/removed/replaced, there are still quite a number of people who still have contributions in the current master.

Finding some of those folks turned out to be hard. I ran into a few cases of abandoned email addresses, and sometimes googling for "firstname lastname" led me to people with the same name but no connection to JGroups whatsoever...

Anyway, the change is done now and I wanted to release a first alpha under AL 2.0 as soon as possible, so other projects can start using JGroups with the new license.

Although 'alpha' suggests bad code quality, the opposite is actually true: this release is very stable, at least as stable as the latest 3.3.x. The major changes are listed below.
Enjoy !

RELAY2: allow for more than one site master
-------------------------------------------
[https://issues.jboss.org/browse/JGRP-1649]

If we have a lot of traffic between sites, having more than 1 site master 
increases performance and reduces stress on the single site master


Kerberos based authentication
-----------------------------
[https://issues.jboss.org/browse/JGRP-1657]

New AUTH plugin contributed by Martin Swales. Experimental, needs more work


Probe now works with TCP too
----------------------------
[https://issues.jboss.org/browse/JGRP-1568]

If multicasting is not enabled, probe.sh can be started as follows:
probe.sh -addr 192.168.1.5 -port 12345
, where 192.168.1.5:12345 is the physical address:port of a node.
Probe will ask that node for the addresses of all other members and 
then send the request to all members.



Optimizations
=============

UNICAST3: ack messages sooner
-----------------------------
[https://issues.jboss.org/browse/JGRP-1664]

A message would get acked after delivery, not reception. This was changed, 
so that long running app code would not delay acking the message, which 
could lead to unneeded retransmission by the sender.


Bug fixes
=========

FRAG/FRAG2: incorrect ordering with message batches
[https://issues.jboss.org/browse/JGRP-1648]

Reassembled messages would get reinserted into the batch at the end instead of 
at their original position


RSVP: incorrect ordering with message batches
---------------------------------------------
[https://issues.jboss.org/browse/JGRP-1641]

RSVP-tagged messages in a batch would get delivered immediately,
instead of waiting for their turn


Memory leak in STABLE
---------------------
[https://issues.jboss.org/browse/JGRP-1638]

Occurred when send_stable_msg_to_coord_only was enabled.


NAKACK2/UNICAST3: problems with flow control
--------------------------------------------
[https://issues.jboss.org/browse/JGRP-1665]

If an incoming message sent out other messages before returning, it could 
block forever as new credits would not be processed.
Caused by a regression (removal of ignore_sync_thread) in FlowControl.



AUTH: nodes without AUTH protocol can join cluster
--------------------------------------------------
[https://issues.jboss.org/browse/JGRP-1661]

If a member didn't ship an AuthHeader, the message would get passed up
instead of rejected.


LockService issues
------------------
[https://issues.jboss.org/browse/JGRP-1634]
[https://issues.jboss.org/browse/JGRP-1639]
[https://issues.jboss.org/browse/JGRP-1660]

Bug fix for concurrent tryLock() blocks and various optimizations.


Logical name cache is cleared, affecting other channels
-------------------------------------------------------
[https://issues.jboss.org/browse/JGRP-1654]

If we have multiple channels in the same JVM, the a new view in one channel
triggers removal of the entries of all other caches



Manual
======

The manual is at http://www.jgroups.org/manual-3.x/html/index.html.

The complete list of features and bug fixes can be found at 
http://jira.jboss.com/jira/browse/JGRP.


Bela Ban, Kreuzlingen, Switzerland
Aug 2013


[1] https://issues.jboss.org/browse/JGRP-1643




Tuesday, May 28, 2013

JGroups to investigate adopting Apache License 2.0

We've started to investigate whether JGroups can change its license from LGPL 2.1 [1] to Apache License (AL) 2.0 [2]. If possible, the next JGroups release will be licensed under AL 2.0.

The reason for the switch is that AL 2.0 is more permissive than LGPL 2.1 and can be integrated with other projects / products more easily. Also, AL 2.0 is one of the most frequently used licenses in the open source world [3], and you can't go wrong with that, can you ? :-)

I've received quite a number of requests to change the license over the years, from the open source community and from companies trying to integrate JGroups into their products.

This change should increase acceptance of JGroups with other open source projects or commercial products.

So what would the impact of this change be ?
  • For projects / products using a current (<= 3.3) JGroups release, nothing would change (they're still licensed under LGPL 2.1)
  • For projects / products upgrading to the AL licensed release (once it's out), nothing would change; the AL 2.0 license is even more permissive
Let me repeat this, nothing would change except that LGPL skeptics should now be able to use JGroups.

Note that Infinispan is looking into making the same change, see [4] for details. As Infinispan consumes JGroups, it is important for JGroups to have the same license as Infinispan, or else integration would be a nightmare; it's hard (if not impossible) for an AL project to consume an LGPL project.

Changing the license would be done in the next JGroups release. Whether this will be 3.4 or whether a license change warrants going directly to a 4.0 is still undecided.

We're currently working with Red Hat's legal department and the community to see whether this switch is possible and what needs to be done to make it happen.

Opinions ? Questions ? Feedback ? Please post to this blog, or the JGroups mailing list.

Cheers,

[1] http://opensource.org/licenses/LGPL-2.1
[2] http://opensource.org/licenses/Apache-2.0
[3] http://osrc.blackducksoftware.com/data/licenses
[4] http://infinispan.blogspot.co.uk/2013/05/infinispan-to-adopt-apache-software.html



Tuesday, May 07, 2013

JGroups 3.3.0.Final released

I'm happy to announce that JGroups 3.3.0.Final was released today.

It contains quite a few optimizations and new features, the most prominent ones being:
  • Message batching: messages received as bundles by the transport are passed up as batches. Compared to passing individual messages up the stack, the advantage is that we have to acquire resources (such as locks) only once per batch instead of once per message. This reduces the number of lock acquisitions and should lead to less context switching and better performance in contended scenarios.
  • Asynchronous Invocation API: This allows the recipient of a message in MessageDispatcher or RpcDispatcher to make the delivering thread return immediately (making it available for other requests) and to send the response later. The advantage is that it is the application which now decides how to deliver messages (e.g. sequentially or in parallel), and not JGroups. The documentation is here: http://www.jgroups.org/manual-3.x/html/user-building-blocks.html#AsyncInvocation
  • UNICAST3: this is a new reliable unicast protocol, combining the advantages of UNICAST (immediate delivery) and UNICAST3 (speed). It is the default in 3.3.
  • New internal thread pool: to be used for internal JGroups messages only. This way, important internal JGroups messages are not impeded by the delivery of application messages ahead of them in delivery order. While this pool is not supposed to be used by application messages, it will help for example to decrease unneeded blocking (by credit messages getting queued up behind application messages), or reduce false suspicions (due to heartbeats getting handled too late, or even getting dropped).
  • RELAY2 improvements: RELAY2 is the protocol which relays traffic between geographically separate sites. This will be the topic of my talk at Red Hat Summit in June.
  • New timer implementation: a better, simpler and faster implementation; the default in 3.3.
  • New message bundler: this new bundler handles sending of individual messages and message batches equally well. Default in 3.3.
A more detailed version of these release notes can be found at [1]. The documentation can be found at [2]. The JIRA is at [3].

Please post questions and feedback as usual on the mailing list.


[1] https://github.com/belaban/JGroups/blob/3.3/doc/ReleaseNotes-3.3.0.txt

[2]  http://www.jgroups.org/manual-3.x/html/index.html

[3] https://issues.jboss.org/browse/JGRP

Monday, April 22, 2013

JBossWorld 2013 in Boston is around the corner

FYI,

I'm going to have a talk at JBossWorld / Red Hat Summit 2013 in Boston on June 13: http://www.redhat.com/summit/sessions/index.html#54.

This will be about a feature in JBoss Data Grid (JDG) which provides geographic failover between sites. I'm going to run a couple of JDG clusters (Boston, London and San Francisco), and initially route all clients to London. Then, as London shuts down at the end of their day, I'll route all clients over to Boston, and finally to San Francisco.

I have a demo that can be run by anyone with internet access, in their browser. Users can punch in some data, and will see their clients fail over between sites seamlessly, without data loss.

Hope to see / meet some of you in Boston !
Cheers,

Tuesday, February 05, 2013

Performance of message batching


A quick heads up on performance of message batching [1] [2]: I ran MPerf and UPerf and got the results as shown at the bottom of [1].

The tests were run on a 4 node cluster; with cluster sizes of 6 and 8, I ran 2 processes on the same physical box.

MPerf shows that a slightly better perf for 2 and 4 nodes, but a significantly (10%) better perf when running more than 1 process on the same box (6 and 8 nodes). I think the reason is that under contention, the property of message batching to acquire fewer locks comes in to reduce lock contention.

UnicastTestRPC shows exactly the same perf for the old (no message batching) and the new code (with message batching). The main reason here is that we use synchronous RPCs and one sender, which doesn't take advantage of message batching at all, as no message bundles are sent across the wire.

UPerf shows a significantly better perf for 4 nodes (11%) and 8 nodes (16% better). I guess the reason here is that we do make use of message batching as we have multiple sender threads and higher contention than in the previous test.

This is not the end of the line, as I haven't implemented message batching in protocols above NAKACK2 and UNICAST2: currently, messages are sent up in batches from the transport to NAKACK2 (multicast messages) or UNICAST2 (unicast messages), but from there on, they're sent up individually.
This will get changed in [3], but because this is a lot of work and will affect many classes, I thought I split the work in two parts.

The first part has been merged with master (3.3) and it would be interesting to get feedback from people trying this out !

Cheers,

[1] https://issues.jboss.org/browse/JGRP-1564

[2] http://belaban.blogspot.ch/2013/01/buy-one-get-many-for-free-message.html

[3] https://issues.jboss.org/browse/JGRP-1581

Wednesday, January 30, 2013

Buy one, get many for free: message batching in JGroups

Just a quick heads-up of what's going on in JGroups 3.3 with message batching.

Currently, when the transport receives a message bundle (say 20 messages), it passes the bundle to the regular thread pool (OOB messages are never bundled). The thread which handles the bundle grabs each message, adds it to the retransmission table, and then removes as many messages as possible and passes them up one-by-one.

So, although we have a message bundle, we don't process it as a bundle, but rather add each message individually and then pass them up up one-by-one.

Message batching [1] changes this. It reads a message bundle directly into 2 MessageBatch instances: one for OOB messages and one for regular messages. (This already shows that OOB messages are now bundled, too, but more on this later). The OOB MessageBatch is passed to the OOB thread pool for handling, the regular batch to the regular pool.

A message batch is nothing more than a list of messages.

A message batch is handled by only 1 thread: the thread passes the entire batch up the stack. Each protocol can remove messages it consumes (e.g. FD_ALL), change messages in-place (e.g. COMPRESS or ENCRYPT), remove and add messages (e.g. FRAG2), remove messages and pass up a new batch (NAKACK2, UNICAST2) or even do nothing (SIZE, STATS).

The advantage is that a protocol can now handle many messages at once, amortizing (e.g.) lock acquisition costs. For example, NAKACK2 adds all 20 messages to the retransmission table at once, thereby acquiring the table lock only once rather than 20 times. This means that we incur the cost of 1 lock acquition, instead of 20. It goes without saying that this will also reduce lock contention, at least in this particular case, even if the lock duration will be slightly longer than before.

I'll present some performance numbers soon, but so far preliminary performance tests look promising !

So while message bundling queues messages and sends them across the wire as a list, but stops at the receiver's transport; message batching takes this idea further and passes that bundle up all the way to the channel. (Note that this will require another receive() callback in the Receiver, but this will be transparent by default).

Message batching will allow other cool things to happen, e.g.
  • OOB messages will be bundled too now. If no bundling is desired, tag a message as DONT_BUNDLE.
  • We can simplify the message bundler (on the sender side), see [2]. As a result, I might even be able to remove all existing 4 message bundlers. As you know, I like removing stuff, and making code easier to read !
  • RPC responses can be bundled [3]
  • UNICAST2 can now ack the 'last message' [4]
Cheers,


[1] https://issues.jboss.org/browse/JGRP-1564

[2] https://issues.jboss.org/browse/JGRP-1540

[3] https://issues.jboss.org/browse/JGRP-1566

[4] https://issues.jboss.org/browse/JGRP-1548

Wednesday, January 02, 2013

SUPERVISOR: detecting faults and fixing them automatically

I've added a new protocol SUPERVISOR [1] to master, which can periodically check for certain conditions and correct them if necessary. This will be in the next release (3.3) of JGroups.

You can think of SUPERVISOR as an automated rule-based system admin.

SUPERVISOR was born out of a discussion on the mailing list [2] where a bug in FD caused the failure detection task in FD to be stopped, so members would not get suspected and excluded anymore. This is bad if the suspected member was the coordinator itself, as new members would not be able to join anymore !

Of course, fixing the bug [3] was the first priority, but I felt that it would be good to also have a second line of defense that detected problems in a running stack. Even if a rule doesn't fix the problem, it can still be used to detect it and alert the system admin, so that the issue can be fixed manually.

The documentation for SUPERVISOR is here: [4].


[1] https://github.com/belaban/JGroups/blob/master/src/org/jgroups/protocols/rules/SUPERVISOR.java

[2] https://sourceforge.net/mailarchive/message.php?msg_id=30218296

[3] https://issues.jboss.org/browse/JGRP-1559

[4] http://www.jgroups.org/manual-3.x/html/user-advanced.html#Supervisor



Cross site replication: demo on YouTube

FYI,

I've recently published a video on cross-site replication [1] on Youtube: [2].

The video shows how to set up and configure cross-site replication in Infinispan, although the focus of the video is on running the performance test [3].

Cheers, and a belated happy new year to everyone !


[1] https://docs.jboss.org/author/display/ISPN/Cross+site+replication

[2] https://www.youtube.com/watch?v=owOs430vLZo

[3] https://github.com/belaban/IspnPerfTest