Tuesday, December 22, 2020

Running jgroups-raft as a service

This is a short tutorial on running a Raft cluster [1] in Kubernetes. It shows how to run a jgroups-raft cluster of 3 nodes, then connects to it with a client.

Running the jgroups-raft cluster

This is very simple with Kubernetes:

kubectl apply -f https://raw.githubusercontent.com/belaban/jgroups-raft/master/conf/rsm.yaml

This downloads belaban/jgroups-raft:blog and starts 3 StatefulSet instances. The instances are named jgroups-raft-0, jgroups-raft-1 and jgroups-raft-2. The persistent data is stored in /mnt/data. Note that the load balancer fronting the 3 instances is listening on port 1965:

netstat -na -f inet |grep 1965
tcp46      0      0  *.1965                 *.*                    LISTEN 

We can look at the cluster with probe:

kubectl exec jgroups-raft-2 probe.sh
#1 (176 bytes):
local_addr=jgroups-raft-2
physical_addr=10.1.0.207:58801
view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]
cluster=rsm
version=5.1.3.Final (Stelvio)

#2 (176 bytes):
local_addr=jgroups-raft-1
physical_addr=10.1.0.206:35596
view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]
cluster=rsm
version=5.1.3.Final (Stelvio)

#3 (176 bytes):
local_addr=jgroups-raft-0
physical_addr=10.1.0.205:46824
view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]
cluster=rsm
version=5.1.3.Final (Stelvio)

This shows the 3 instances, all having the same view (jgroups-raft-1|2). This shows that the cluster has formed correctly.

Running the client

This is a bit more involved. We could clone the jgroups-raft repo and build the client from source, but for this tutorial, we'll simply download the relevant JARs (jgroups-raft, JGroups) from maven central.

mkdir lib

curl -o ./lib/jgroups.jar https://repo1.maven.org/maven2/org/jgroups/jgroups/5.1.3.Final/jgroups-5.1.3.Final.jar

curl -o ./lib/raft.jar https://repo1.maven.org/maven2/org/jgroups/jgroups-raft/1.0.1.Final/jgroups-raft-1.0.1.Final.jar

java -cp "./lib/*" org.jgroups.raft.client.ReplicatedStateMachineClient

The client connects to the load balancer listening on port 1965, which redirects the request to one of the 3 instances. It can be used to modify/view the replicated state maintained by jgroups-raft, e.g.:

[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit

1
key: name
value: Bela
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit

1
key: id
value: 500
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit

4
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit

{name=Bela, id=500}
5
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit


index (term): command
---------------------
21 (11379): put(name, Bela)
22 (11379): put(id, 500)

v
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit

local address: jgroups-raft-0
view: [jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]

[1] adds a key and value to the replicated state, 4 shows the entire state and [5] shows the log. Press 'v' to see the cluster view.

 

Conclusion

Using Kubernetes is a quick way to to run a 3-node jgroups-raft cluster as a service. The demo above showed ReplicatedStateMachine, but - of course - other services are possible, too. For instance, one could write a Yaml file which starts a replicated counter service easily.

On the client side, a simple protocol to set/get and remove data was implemented. A more sophisticated client could for example use gRPC for the communication between client and service.

Questions and feedback please to the mailing list [3].

Enjoy!


[1] http://belaban.github.io/jgroups-raft/

[2] https://hub.docker.com/repository/docker/belaban/jgroups-raft

[3] https://groups.google.com/g/jgroups-raft