This is a short tutorial on running a Raft cluster [1] in Kubernetes. It shows how to run a jgroups-raft cluster of 3 nodes, then connects to it with a client.
Running the jgroups-raft cluster
This is very simple with Kubernetes:
kubectl apply -f https://raw.githubusercontent.com/belaban/jgroups-raft/master/conf/rsm.yaml
This downloads belaban/jgroups-raft:blog and starts 3 StatefulSet instances. The instances are named jgroups-raft-0, jgroups-raft-1 and jgroups-raft-2. The persistent data is stored in /mnt/data. Note that the load balancer fronting the 3 instances is listening on port 1965:
netstat -na -f inet |grep 1965
tcp46 0 0 *.1965 *.* LISTEN
We can look at the cluster with probe:
kubectl exec jgroups-raft-2 probe.sh
#1 (176 bytes):
local_addr=jgroups-raft-2
physical_addr=10.1.0.207:58801
view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]
cluster=rsm
version=5.1.3.Final (Stelvio)
#2 (176 bytes):
local_addr=jgroups-raft-1
physical_addr=10.1.0.206:35596
view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]
cluster=rsm
version=5.1.3.Final (Stelvio)
#3 (176 bytes):
local_addr=jgroups-raft-0
physical_addr=10.1.0.205:46824
view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]
cluster=rsm
version=5.1.3.Final (Stelvio)
This shows the 3 instances, all having the same view (jgroups-raft-1|2). This shows that the cluster has formed correctly.
Running the client
This is a bit more involved. We could clone the jgroups-raft repo and build the client from source, but for this tutorial, we'll simply download the relevant JARs (jgroups-raft, JGroups) from maven central.
mkdir lib
curl -o ./lib/jgroups.jar https://repo1.maven.org/maven2/org/jgroups/jgroups/5.1.3.Final/jgroups-5.1.3.Final.jar
curl -o ./lib/raft.jar https://repo1.maven.org/maven2/org/jgroups/jgroups-raft/1.0.1.Final/jgroups-raft-1.0.1.Final.jar
java -cp "./lib/*" org.jgroups.raft.client.ReplicatedStateMachineClient
The client connects to the load balancer listening on port 1965, which redirects the request to one of the 3 instances. It can be used to modify/view the replicated state maintained by jgroups-raft, e.g.:
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit
1
key: name
value: Bela
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit
1
key: id
value: 500
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit
4
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit
{name=Bela, id=500}
5
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit
index (term): command
---------------------
21 (11379): put(name, Bela)
22 (11379): put(id, 500)
v
[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit
local address: jgroups-raft-0
view: [jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]
[1] adds a key and value to the replicated state, 4 shows the entire state and [5] shows the log. Press 'v' to see the cluster view.
Conclusion
Using Kubernetes is a quick way to to run a 3-node jgroups-raft cluster as a service. The demo above showed ReplicatedStateMachine, but - of course - other services are possible, too. For instance, one could write a Yaml file which starts a replicated counter service easily.
On the client side, a simple protocol to set/get and remove data was implemented. A more sophisticated client could for example use gRPC for the communication between client and service.
Questions and feedback please to the mailing list [3].
Enjoy!
[1] http://belaban.github.io/jgroups-raft/
[2] https://hub.docker.com/repository/docker/belaban/jgroups-raft
[3] https://groups.google.com/g/jgroups-raft
