Monday, August 28, 2017

JGroups workshops in Rome and Berlin in November

I'm happy to announce that I've revamped the JGroups workshop and I'll be teaching 2 workshops in November: Rome Nov 7-10 and Berlin Nov 21-24.

The new workshop is 4 days (TUE-FRI). I've updated the workshop to use the latest version of JGroups (4.0.x), removed a few sections and added sections on troubleshooting/diagnosis and handling of network partitions (split brain).

Should be a fun and hands-on workshop!

To register and for the table of contents, navigate to [1].

To get the early-bird discount, use code EARLYBIRD.



Monday, June 26, 2017

Non-blocking JGroups

I'm happy to announce that I just released JGroups 4.0.4.Final!

It's prominent feature is non-blocking flow control [1,2].

Flow control makes sure that a fast sender cannot overwhelm a slow receiver in the long run (short spikes are tolerated) by adjusting the send rate to the receive rate. This is done by giving each sender credits (number of bytes) that it is allowed to send until it has to block.

A receiver sends new credits to the sender once it has processed a message.

When no credits are available anymore, the sender blocks until it gets new credits from the receiver(s). This is done in UFC (unicast flow control) and MFC (multicast flow control).

Flow control and TCP have been the only protocols that could block (UDP and TCP_NIO2 are non-blocking).

Non-blocking flow control now adds protocols UFC_NB and MFC_NB. These can replace their blocking counterparts.

Instead of blocking sender threads, the non-blocking flow control protocols queue messages when not enough credits are available to send them, allowing the sender threads to return immediately.

When fresh credits are received, the queued messages will be sent.

The queues are bound to prevent heap exhaustion; setting attribute max_queue_size (bytes) will queue messages up to max_queue_size bytes and then block subsequent attempts to queue until more space is available. Of course, setting max_queue_size to a large value will effectively make queues unbounded.

Using MFC_NB / UFC_NB with a transport of UDP or TCP_NIO2, which also never block, provides a completely non-blocking stack, where sends never block a JChannel.send(Message). If RpcDispatcher is used, there are non-blocking methods to invoke (even synchronous) RPCs, ie. the ones which return a CompletableFuture.

Other features of 4.0.4 include a new message bundler and a few bug fixes (e.g. the internal thread pool was not shut down). For the list of all JIRAs see [3].





Monday, May 08, 2017

Running an Infinispan cluster with Kubernetes on Google Container Engine (GKE)

In this post, I'm going to show the steps needed to get a 10 node Infinispan cluster up and running on Google Container Engine (GKE).

The test we'll be running is IspnPerfTest and the corresponding docker image is belaban/ispn_perf_test on dockerhub.

All that's needed to run this youselves is a Google Compute Engine account, so head on over there now and create one if you want to reproduce this demo! :-)

Alternatively, the cluster could be run locally in minikube, but for this post I chose GKE instead.

Ready? Then let's get cracking...

First, let's create a 10-node cluster in GKE. The screen shot below shows the form that needs to be filled out to create a 10 node cluster in GKE. This results in 10 nodes getting created in Google Compute Engine (GCE):

As shown, we'll use 10 v16-cpu instances with 14GB of memory each.

Press "Create" and the cluster is being created:

If you logged into your Google Compute Engine console, it would show the 10 nodes that are getting created.

When the cluster has been created, click on "Connect" and execute the "gcloud" command that's shown as a result:

gcloud container clusters get-credentials ispn-cluster \ --zone us-central1-a --project ispnperftest

We can now see the 10 GCE nodes:
[belasmac] /Users/bela/kubetest$ kubectl get nodes
NAME                                          STATUS    AGE       VERSION
gke-ispn-cluster-default-pool-59ed0e14-1zdb   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-33pk   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-3t95   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-5sn9   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-9lmz   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-j646   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-k797   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-q80q   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-r96s   Ready     4m        v1.5.7
gke-ispn-cluster-default-pool-59ed0e14-zhdj   Ready     4m        v1.5.7

Next we'll run an interactive instance of IspnPerfTest and 3 non-interactive (no need for a TTY) instances, forming a cluster of 4. First, we start the interactive instance. Note that it might take a while until Kubernetes has downloaded image belaban/ispn_perf_test from dockerhub. When done, the following is shown:

[belasmac] /Users/bela/kubetest$ kubectl run infinispan --rm=true -it --image=belaban/ispn_perf_test
If you don't see a command prompt, try pressing enter.

GMS: address=infinispan-749052960-pl066-27417, cluster=default, physical address=

GMS: address=infinispan-749052960-pl066-18029, cluster=cfg, physical address=
created 100,000 keys: [1-100,000]
[1] Start test [2] View [3] Cache size [4] Threads (100)
[5] Keys (100,000) [6] Time (secs) (60) [7] Value size (1.00KB) [8] Validate
[p] Populate cache [c] Clear cache [v] Versions
[r] Read percentage (1.00)
[d] Details (true)  [i] Invokers (false) [l] dump local cache
[q] Quit [X] Quit all

Now we'll start 3 more instances, the definition of which is taken from a Yaml file (ispn.yaml):

## Creates a number of pods on kubernetes running IspnPerfTest
## Run with: kubectl create -f ispn.yaml
apiVersion: extensions/v1beta1
kind: Deployment
  name: ispn
  namespace: default
  replicas: 3
        run: ispn-perf-test
      hostNetwork: false
      - args:
        - -nohup
        name: ispn-perf-test
        image: belaban/ispn_perf_test
        # imagePullPolicy: IfNotPresent

The YAML file creates 3 instances (replicas: 3):

belasmac] /Users/bela/kubetest$ kubectl create -f ispn.yaml
deployment "ispn" created

Running "kubectl get pods -o wide" shows:

[belasmac] /Users/bela/kubetest$ kubectl get pods -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP          NODE
infinispan-749052960-pl066   1/1       Running   0          4m   gke-ispn-cluster-default-pool-59ed0e14-5sn9
ispn-1255975377-hm785        1/1       Running   0          46s   gke-ispn-cluster-default-pool-59ed0e14-r96s
ispn-1255975377-jx70d        1/1       Running   0          46s   gke-ispn-cluster-default-pool-59ed0e14-1zdb
ispn-1255975377-xf9r8        1/1       Running   0          46s   gke-ispn-cluster-default-pool-59ed0e14-3t95

This shows 1 infinispan instance (interactive terminal)  and 3 ispn instances (non-interactive).

We can now exec into one of the instances are run to verify that a cluster of 4 has formed:

[belasmac] /Users/bela/kubetest$ kubectl exec -it ispn-1255975377-hm785 bash
bash-4.3$ -addr localhost                                                                                                                                         
-- sending probe request to /
-- sending probe request to /
-- sending probe request to /
-- sending probe request to /

#1 (287 bytes):
view=[infinispan-749052960-pl066-27417|3] (4) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319]
version=4.0.3-SNAPSHOT (Schiener Berg)

#2 (287 bytes):
view=[infinispan-749052960-pl066-27417|3] (4) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319]
version=4.0.3-SNAPSHOT (Schiener Berg)

#3 (284 bytes):
view=[infinispan-749052960-pl066-27417|3] (4) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319]
version=4.0.3-SNAPSHOT (Schiener Berg)

#4 (292 bytes):
view=[infinispan-749052960-pl066-27417|3] (4) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319]
version=4.0.3-SNAPSHOT (Schiener Berg)

4 responses (4 matches, 0 non matches)

The 'view' item shows the same view (list of cluster members) across all 4 nodes, so the cluster has formed successfully. Also, if we look at our interactive instance, pressing [2] shows the cluster has 4 members:

-- local: infinispan-749052960-pl066-18029
-- view: [infinispan-749052960-pl066-18029|3] (4) [infinispan-749052960-pl066-18029, ispn-1255975377-xf9r8-34843, ispn-1255975377-jx70d-20952, ispn-1255975377-hm785-48520]

This means we have view number 4 (0-3) and 4 cluster members (the number in parens).

Next, let's scale the cluster to 10 members. To do this, we'll tell Kubernetes to scale the ispn deployment to 9 instances (from 3):

[belasmac] /Users/bela/IspnPerfTest$ kubectl scale deployment ispn --replicas=9
deployment "ispn" scaled

After a few seconds, the interactive terminal shows a new view containing 10 members:

** view: [infinispan-749052960-pl066-27417|9] (10) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319, ispn-1255975377-6191p-9724, ispn-1255975377-1g2kx-5547, ispn-1255975377-333rl-13052, ispn-1255975377-57zgl-28575, ispn-1255975377-j8ckh-35528, ispn-1255975377-lgvmt-32173]

We also see a minor inconvenience when looking at the pods:

[belasmac] /Users/bela/jgroups-docker$ lubectl get pods -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP          NODE
infinispan-749052960-pl066   1/1       Running   0          13m   gke-ispn-cluster-default-pool-59ed0e14-5sn9
ispn-1255975377-1g2kx        1/1       Running   0          1m   gke-ispn-cluster-default-pool-59ed0e14-k797
ispn-1255975377-333rl        1/1       Running   0          1m   gke-ispn-cluster-default-pool-59ed0e14-9lmz
ispn-1255975377-57zgl        1/1       Running   0          1m   gke-ispn-cluster-default-pool-59ed0e14-q80q
ispn-1255975377-6191p        1/1       Running   0          1m   gke-ispn-cluster-default-pool-59ed0e14-5sn9
ispn-1255975377-hm785        1/1       Running   0          10m   gke-ispn-cluster-default-pool-59ed0e14-r96s
ispn-1255975377-j8ckh        1/1       Running   0          1m   gke-ispn-cluster-default-pool-59ed0e14-j646
ispn-1255975377-jx70d        1/1       Running   0          10m   gke-ispn-cluster-default-pool-59ed0e14-1zdb
ispn-1255975377-lgvmt        1/1       Running   0          1m   gke-ispn-cluster-default-pool-59ed0e14-33pk
ispn-1255975377-xf9r8        1/1       Running   0          10m   gke-ispn-cluster-default-pool-59ed0e14-3t95

The infinispan pod  and one of the ispn pods have been created on the same GCE node gke-ispn-cluster-default-pool-59ed0e14-5sn9. The reason is that they are different deployments, and so GKE deploys them in an unrelated manner. Had all pods been created in the same depoyment, Kubernetes would have assigned pods to nodes in a round-robin fashion.

This could be fixed by using labels, but I didn't want to complicate the demo. Note that running more that one pod per GCE node will harm performance slightly...

Now we can run the performance test by populating the cluster (grid) with key/value pairs ([p]) and then running the test ([1]). This inserts 100'000 key/value pairs into the grid and executes read operations on every node for 1 minute (for details on IspnPerfTest consult the Github URL given earlier):

 Running test for 60 seconds:
1: 47,742 reqs/sec (286,351 reads 0 writes)
2: 56,854 reqs/sec (682,203 reads 0 writes)
3: 60,628 reqs/sec (1,091,264 reads 0 writes)
4: 62,922 reqs/sec (1,510,092 reads 0 writes)
5: 64,413 reqs/sec (1,932,427 reads 0 writes)
6: 65,050 reqs/sec (2,341,846 reads 0 writes)
7: 65,517 reqs/sec (2,751,828 reads 0 writes)
8: 66,172 reqs/sec (3,176,344 reads 0 writes)
9: 66,588 reqs/sec (3,595,839 reads 0 writes)
10: 67,183 reqs/sec (4,031,168 reads 0 writes)

done (in 60020 ms)

all: get 1 / 1,486.77 / 364,799.00, put: 0 / 0.00 / 0.00

======================= Results: ===========================
ispn-1255975377-1g2kx-51998: 100,707.90 reqs/sec (6,045,193 GETs, 0 PUTs), avg RTT (us) = 988.79 get, 0.00 put
ispn-1255975377-xf9r8-34843: 95,986.15 reqs/sec (5,760,705 GETs, 0 PUTs), avg RTT (us) = 1,036.78 get, 0.00 put
ispn-1255975377-jx70d-20952: 103,935.58 reqs/sec (6,239,149 GETs, 0 PUTs), avg RTT (us) = 961.14 get, 0.00 put
ispn-1255975377-j8ckh-11479: 100,869.08 reqs/sec (6,054,263 GETs, 0 PUTs), avg RTT (us) = 987.95 get, 0.00 put
ispn-1255975377-lgvmt-26968: 104,007.33 reqs/sec (6,243,144 GETs, 0 PUTs), avg RTT (us) = 960.05 get, 0.00 put
ispn-1255975377-6191p-15331: 69,004.31 reqs/sec (4,142,053 GETs, 0 PUTs), avg RTT (us) = 1,442.04 get, 0.00 put
ispn-1255975377-57zgl-58007: 92,282.75 reqs/sec (5,538,903 GETs, 0 PUTs), avg RTT (us) = 1,078.14 get, 0.00 put
ispn-1255975377-333rl-8583: 99,130.95 reqs/sec (5,949,542 GETs, 0 PUTs), avg RTT (us) = 1,004.08 get, 0.00 put
infinispan-749052960-pl066-18029: 67,166.91 reqs/sec (4,031,358 GETs, 0 PUTs), avg RTT (us) = 1,486.77 get, 0.00 put
ispn-1255975377-hm785-48520: 79,616.70 reqs/sec (4,778,196 GETs, 0 PUTs), avg RTT (us) = 1,254.87 get, 0.00 put

Throughput: 91,271 reqs/sec/node (91.27MB/sec) 912,601 reqs/sec/cluster
Roundtrip:  gets min/avg/max = 0/1,092.14/371,711.00 us (50.0=938 90.0=1,869 95.0=2,419 99.0=5,711 99.9=20,319 [percentile at mean: 62.50]),
            puts n/a

As suspected, instances infinispan-749052960-pl066-18029 and ispn-1255975377-6191p-15331 show lower performance than the other nodes, as they are co-located on the same GCE node.

The Kubernetes integration in JGroups is done by KUBE_PING which interacts with the Kubernetes master (API server) to fetch the IP addresses of the pods that have been started by Kubernetes.

The KUBE_PING protocol is new, so direct problems, issues, configuration questions etc to the JGroups mailing list.


Tuesday, February 21, 2017

JGroups 4.0.0.Final

I'm happy to announce that JGroups 4.0.0.Final is out!

With 120+ issues, the focus of this version is API changes, the switch to Java 8 and thus the use of new language features (streams, lambdas) and optimizations.

API changes

Fluent configuration

JChannel ch=new JChannel("config.xml").name("A").connect("cluster");

Removed deprecated classes and methods

Removed support for plain string-based channel configuration


Use of Java 8 features

E.g. replace Condition with Predicate etc

Removed classes

MessageDispatcher / RpcDispatcher changes

Use of CompletableFuture instead of NotifyingFuture

TCP: remove send queues

New features

Deliver message batches 


Receiver (ReceiverAdapter) now has an additional callback receive(MessageBatch batch). This allows JGroups to pass an entire batch of messages to the application rather than passing them up one by one.



Plus fixed security issues in the refactored code. Removed ENCRYPT.

Measure round-trip times for RPCs via probe


Keys 'rpcs' and 'rpcs-details' dump information about average RTTs between individual cluster members

Change message bundler at runtime


Message bundlers can be changed at runtime via probe. This is useful to see the effect of different bundlers on performance, even in the same test run.


DELIVERY_TIME: new protocol to measure delivery time in the application


Exposes stats via JMX and probe

RELAY2: sticky site masters


When we have multiple site masters, messages from the same member should always be handled by the same site master. This prevents reordering of messages at the receiver.

Multiple elements in bind_addr





This tries to bind to eth2 first, then to, then to an interface that starts with en0, and

finally to loopback.

Useful when running in an environment where the IP addresses and/or interfaces are not known before.

Multiple receiver threads in UDP


Multiple threads can receive (and process) messages from a datagram socket, preventing queue buildups.




RpcDispatcher: don't copy the first anycast


Reduction of memory size of classes

Remove one buffer copy in COMPRESS


Replace Java serialization with JGroups marshalling


Some internal classes still used Java serialization, which opens up security holes

(google 'java serialization vulnerability').

Faster marshalling / unmarshalling of messages

TCP: reduce blocking


Message bundler improvements


E.g. removal of SingletonAddress: not needed anymore as shared transports have been removed, too.

Protocol: addition of up(Message) and down(Message) callbacks


This massively reduces the number of Event creations.

TransferQueueBundler: remove multiple messages


Instead of removing messages one-by-one, the remover thread now removes as many messages as are in the queue (contention) into a local queue (no contention) and then creates and sends message batches off of the local queue.

Single thread pool


There's only a single thread pool for all typs of messages, reducing the maintenance overhead of 4 thread pools and the configuration required.

The internal thread pool is still available (size to the number of cores), but not configurable.
A ForkJoinPool can be used instead of the thread pool (which can be disabled as well).

Timer: tasks can indicate that they will not block


If a task calls execute() with an argument blocking==false, the task will be executed by the timer's main thread, and not be passed on to the timer's thread pool. This reduces the number of threads needed and therefore the number of context switches.

Headers are resized unnecessarily


ByteArrayDataOutputStream: expand more conservatively


Reading ints and longs creates unnecessary buffers


Ints and longs are read into a byte[] array first, then parsed. This was changed to read the values and add them to the resulting ints or longs.

Should reduce overall memory allocation as ints and longs are used a lot in headers.

Table.removeMany() creates unneeded temp list


This method is used a lot in NAKACK2 and UNICAST3. The change was to read messages directly into the resulting MessageBatch instead of a temp list and from there into the batch.

Reduce in-memory size of UnicastHeader3


Reduced size from 40 -> 32 bytes.

Cache result of log.isTraceEnabled()


This was done mainly for protocols where log.isTraceEnabled() was used a lot, such as TP, NAKACK2 or UNICAST3.

Note that the log level can still be changed at runtime.

Added MessageProcessingPolicy to define assigning of threads to messages or batches


This only applies only to regular (not OOB or internal) messages. Make sure that only one message per member is processed at a given time by the thread pool.

This reduces the number of threads needed.

UNICAST3 / NAKACK2: more efficient adding and removing of messages / batches to/from tables


Simpler algorithm and removal of one lock (= less contention)

Bug fixes

GMS sometimes ignores view bundling timeout


UFC and MFC headers get mixed up


Although indepent protocols, the protocol ID was assigned by the superclass, so replenish and credit messages would get mixed up, leading to stuttering in the sending of credits.

Flow control: replenish credits after message delivery


MERGE3: merge is never triggered


This is an edge case that was not covered before: every subgroup coordinator has some other member as coord:


MPING: restart fails


UNICAST3 drops all messages until it receives one with first==true


This caused a bug in ASYM_ENCRYPT.

ASYM_ENCRYPT: message batches are not handled correctly


SYM_ENCRYPT: allow for other keystores besides JCEKS


E.g. pcks#12 or jks

ASYM_ENCRYPT encrypts an empty buffer into a null buffer


This caused an NPE.


On Sourceforge:,
or via Maven:

The manual is at

The complete list of features and bug fixes can be found at

Bela Ban, Kreuzlingen, Switzerland
March 2017