tag:blogger.com,1999:blog-198350542024-03-17T08:34:48.581+01:00Belas BlogBela Banhttp://www.blogger.com/profile/01830789377474906550noreply@blogger.comBlogger107125tag:blogger.com,1999:blog-19835054.post-86500370828387639752023-09-12T13:04:00.000+02:002023-09-12T13:04:56.730+02:00JGroups 5.3 released<p>I just released JGroups 5.3.</p><p>The 5.2 branch (last stable release: 5.2.19) is stable and will only be modified when bug fixes are backported from 5.3. All new develpoment will be done on the 5.3 branch.</p><p>The major new feature of the 5.3 release is RELAY3, which provides <b>asymmetric routing between sites</b>. This means that a given site doesn't need to be connected to all other sites, e.g.:<br /></p><p><span style="font-family: courier;">A <--> B <--> C <--> D</span> <br /></p><p>Site A is connected to site B, B to C and C to D. If a member in site D wants to send a message M to a member in site A, then M needs to be forwarded to C, then to B which forwards M to the member in A.<br /></p><p>RELAY3 accepts the same configuration as RELAY2, but cannot be used with RELAY2.</p><p>The documentation is at [2].</p><p>Enjoy!<br /></p><p><br /></p><p>[1] <a href="https://issues.redhat.com/projects/JGRP/versions/12343297">https://issues.redhat.com/projects/JGRP/versions/12343297</a></p><p>[2] <a href="http://www.jgroups.org/manual5/index.html#Relay3Advanced">http://www.jgroups.org/manual5/index.html#Relay3Advanced</a></p><p><br /></p>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-6119677496175442052023-04-28T09:37:00.000+02:002023-04-28T09:37:07.775+02:00 Support for TLS/SSL in TCP<div style="text-align: left;">In version 5.2.15 (to be released soon), TLS can be enabled in TCP via a simple configuration change:</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><span style="font-family: courier;"><TCP<br /> tls.enabled="true"<br /> tls.client_auth="NEED"<br /> tls.keystore_path="good-server.jks"<br /> tls.keystore_password="password"<br /> tls.keystore_alias="server"</span></div><div style="text-align: left;"><span style="font-family: courier;">...</span></div><div style="text-align: left;"><span style="font-family: courier;">/></span></div><div style="text-align: left;"><br /></div><div style="text-align: left;">This installs an SSLSocketFactory into TCP, creating SSLSockets instead of Sockets and SSLServerSockets instead of ServerSockets.</div><div style="text-align: left;"> </div><div style="text-align: left;">This is an alternative to SYM_ENCRYPT.</div><div style="text-align: left;"> </div><div style="text-align: left;">Details can be found in [1]. <br /></div><div style="text-align: left;"> </div><div style="text-align: left;">Cheers,<br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">[1] <a href="http://www.jgroups.org/manual5/index.html#TLS">http://www.jgroups.org/manual5/index.html#TLS</a></div><div style="text-align: left;"><br /></div>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-69293171801385727402021-05-26T10:06:00.001+02:002021-05-26T10:06:47.120+02:00JGroups 5.1.7 released<p>I'm happy to announce that 5.1.7 has been released!</p><p>The major new features are FD_SOCK2 [1] and VERIFY_SUSPECT2 [2].<br /></p><p>The complete list of features and bug fixes is at [4].</p><p></p><p>Here's a short description of the major changes/additions:</p><p><br /></p><h4 style="text-align: left;">FD_SOCK2</h4><p>This is a rewrite of FD_SOCK, which was created 20 (!) years ago. The old protocol has worked surprisingly well, given its brittle and complex design. FD_SOCK2 should be much more robust, as I've eliminated the cache between ports and members, and code which maintains this cache.</p><p>Also, FD_SOCK2 (re-)uses NioServer, which means that we'll use 1 (select) thread instead of 3 in FD_SOCK.<br /></p><p>Compared to FD_SOCK's 1235 LOC, FD_SOCK2 has 723 LOC with the same functionality.</p><p><br /></p><h4 style="text-align: left;">VERIFY_SUSPECT2</h4><p>The major change over VERIFY_SUSPECT is that VERIFY_SUSPECT2 bundles SUSPECT events sent up the stack. This reduces the <a href="https://issues.redhat.com/browse/JGRP-2556">problem</a> where view installation runs into a timeout waiting for acks from crashed members.</p><p>When X crashed, and then Y crashed a few milliseconds later, then VERIFY_SUSPECT would have sent up events SUSPECT(X) and then SUSPECT(Y), whereas VERIFY_SUSPECT2 sends up SUSPECT(X,Y) *if* X and Y crashed in the same time window (1s by default).</p><p>This speeds up the installation of the new view, especially when multiple members have crashed.<br /></p><p><br /></p><h4 style="text-align: left;">No need to use jmx= or op= in probe<br /></h4><p>This is only syntatic sugar, but now we can shorten <span style="font-family: courier;">probe.sh jmx=UDP.bind</span> to <span style="font-family: courier;">probe.sh UDP.ping</span> and <span style="font-family: courier;">probe.sh op=TCP.printConnections</span> to <span style="font-family: courier;">probe.sh TC.printConnections[]</span>. This comes in handy when switching between attributes and operations. JIRA: [3]<br /></p><p><br /></p><p>[1] <a href="https://issues.redhat.com/browse/JGRP-2521">https://issues.redhat.com/browse/JGRP-2521</a><br /></p><p>[2] <a href="https://issues.redhat.com/browse/JGRP-2558">https://issues.redhat.com/browse/JGRP-2558</a></p><p>[3] <a href="https://issues.redhat.com/browse/JGRP-2413">https://issues.redhat.com/browse/JGRP-2413</a> <br /></p><p>[4] <a href="https://issues.redhat.com/projects/JGRP/versions/12355552">https://issues.redhat.com/projects/JGRP/versions/12355552</a></p><p><br /></p>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-1661347791335260752020-12-22T17:08:00.000+01:002020-12-22T17:08:11.832+01:00Running jgroups-raft as a service<p>This is a short tutorial on running a Raft cluster [1] in Kubernetes. It shows how to run a jgroups-raft cluster of 3 nodes, then connects to it with a client.</p><h4 style="text-align: left;">Running the jgroups-raft cluster<br /></h4><p>This is very simple with Kubernetes:</p><p><span style="font-size: x-small;"><span style="font-family: courier;">kubectl apply -f https://raw.githubusercontent.com/belaban/jgroups-raft/master/conf/rsm.yaml</span></span></p><p>This downloads belaban/jgroups-raft:blog and starts 3 <b>StatefulSet</b> instances. The instances are named jgroups-raft-0, jgroups-raft-1 and jgroups-raft-2. The persistent data is stored in /mnt/data. Note that the load balancer fronting the 3 instances is listening on port 1965:</p><p><span style="font-size: x-small;"><span style="font-family: courier;">netstat -na -f inet |grep 1965<br />tcp46 0 0 *.1965 *.* LISTEN </span></span><br /></p><p>We can look at the cluster with probe:</p><p><span style="font-size: x-small;"><span style="font-family: courier;">kubectl exec jgroups-raft-2 probe.sh<br />#1 (176 bytes):<br />local_addr=jgroups-raft-2<br />physical_addr=10.1.0.207:58801<br />view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]<br />cluster=rsm<br />version=5.1.3.Final (Stelvio)<br /><br />#2 (176 bytes):<br />local_addr=jgroups-raft-1<br />physical_addr=10.1.0.206:35596<br />view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]<br />cluster=rsm<br />version=5.1.3.Final (Stelvio)<br /><br />#3 (176 bytes):<br />local_addr=jgroups-raft-0<br />physical_addr=10.1.0.205:46824<br />view=[jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]<br />cluster=rsm<br />version=5.1.3.Final (Stelvio)</span></span><br /></p><p>This shows the 3 instances, all having the same view (jgroups-raft-1|2). This shows that the cluster has formed correctly.<br /></p><h4 style="text-align: left;">Running the client</h4><p>This is a bit more involved. We could clone the jgroups-raft repo and build the client from source, but for this tutorial, we'll simply download the relevant JARs (jgroups-raft, JGroups) from maven central.</p><p><span style="font-size: x-small;"><span style="font-family: courier;">mkdir lib </span></span></p><p><span style="font-size: x-small;"><span style="font-family: courier;">curl -o ./lib/jgroups.jar https://repo1.maven.org/maven2/org/jgroups/jgroups/5.1.3.Final/jgroups-5.1.3.Final.jar </span></span></p><p><span style="font-size: x-small;"><span style="font-family: courier;">curl -o ./lib/raft.jar https://repo1.maven.org/maven2/org/jgroups/jgroups-raft/1.0.1.Final/jgroups-raft-1.0.1.Final.jar </span></span></p><p><span style="font-size: x-small;"><span style="font-family: courier;">java -cp "./lib/*" org.jgroups.raft.client.ReplicatedStateMachineClient</span></span></p><p>The client connects to the load balancer listening on port 1965, which redirects the request to one of the 3 instances. It can be used to modify/view the replicated state maintained by jgroups-raft, e.g.:</p><p><span style="font-size: x-small;"><span style="font-family: courier;">[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit</span></span><br /><span style="font-size: x-small;"><span style="font-family: courier;"><br />1<br />key: name<br />value: Bela<br />[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit<br /><br />1<br />key: id<br />value: 500<br />[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit<br /><br />4<br />[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit<br /><br />{name=Bela, id=500}<br />5<br />[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit<br /><br /><br />index (term): command<br />---------------------<br />21 (11379): put(name, Bela)<br />22 (11379): put(id, 500)<br /><br />v<br />[1] add [2] get [3] remove [4] show all [5] dump log [6] snapshot [v] view [x] exit<br /><br />local address: jgroups-raft-0<br />view: [jgroups-raft-1|2] (3) [jgroups-raft-1, jgroups-raft-0, jgroups-raft-2]<br /></span></span><br /></p><p>[1] adds a key and value to the replicated state, 4 shows the entire state and [5] shows the log. Press 'v' to see the cluster view.</p><p> </p><h4 style="text-align: left;">Conclusion</h4><p>Using Kubernetes is a quick way to to run a 3-node jgroups-raft cluster as a service. The demo above showed ReplicatedStateMachine, but - of course - other services are possible, too. For instance, one could write a Yaml file which starts a replicated counter service easily.</p><p>On the client side, a simple protocol to set/get and remove data was implemented. A more sophisticated client could for example use <a href="https://grpc.io/">gRPC</a> for the communication between client and service.<br /></p><p>Questions and feedback please to the mailing list [3].</p><p>Enjoy!<br /></p><p><br /></p><p>[1] <a href="http://belaban.github.io/jgroups-raft/">http://belaban.github.io/jgroups-raft/</a></p><p>[2] <a href="https://hub.docker.com/repository/docker/belaban/jgroups-raft">https://hub.docker.com/repository/docker/belaban/jgroups-raft</a></p><p>[3] <a href="https://groups.google.com/g/jgroups-raft">https://groups.google.com/g/jgroups-raft</a></p><p> <br /></p>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-19237487383411427422020-11-24T10:11:00.002+01:002020-11-24T10:11:54.590+01:00I hate distributed locks!<p>I hate distributed locks!</p><p>Adding distributed locks to JGroups has been the second biggest mistake that I've made (next to MuxRpcDispatcher). Unfortunately, a lot of people are using them in JGroups... </p><p>Distributed locks don't work. Or, at least, don't work the way developers expect them to.<br /></p><h3 style="text-align: left;">TL;DR</h3><ul style="text-align: left;"><li>Assumption that distributed locks have the same semantics as local locks</li><li>Multiple cluster members in JGroups can hold the same lock when there is a partition</li><li>The try-lock-finally-unlock pattern is unsuitable for taking locks away from a holder</li><li>Some scenarios are better handled by using transactions instead</li><li>Even consensus-based implementations have their share of problems</li></ul><h3 style="text-align: left;">Distributed locks</h3><p style="text-align: left;">The simplest distributed lock implementation is a (fixed) lock server (<b>DLM</b>) which cluster members contact to acquire or release locks. The server may persist locks in a database, so it still knows which members hold locks on a restart. This may be okay for applications that can tolerate unavailability of the lock server and/or the database. If your application is fine with this, then there's no need to read on!<br /></p><p style="text-align: left;">However, the single lock server may quickly become a bottleneck when many members want to acquire or release locks. It is also a single point of failure. This is not the distributed lock implementation I'm talking about (although it works).</p><p style="text-align: left;">What I'm talking about are <b>clustered distributed locks</b>; in today's distributed systems, lock information is typically replicated to all or a subset of the cluster members. This avoids the database bottleneck, but brings with it its own slew of problems.</p><p style="text-align: left;"><b>The biggest problem is that distributed system can have partitions</b>.</p><p style="text-align: left;">Partitions occur when cluster members are separated from each other; when members falsely suspect each other of having crashed, for example because of a long GC cycle, an exhausted thread pool (so heartbeats are dropped), a flaky NIC, or a router dropping packets.</p><p style="text-align: left;">The example I'll use is a cluster {A,B,C} <span style="font-family: courier;"></span>being partitioned into {A} and {B,C}. A suspects and removes members B and C, because it thinks they died, and members B and C remove A for the same reason. Note that B and C can still talk to each other.</p><p style="text-align: left;"> </p><h3 style="text-align: left;">Distributed locking in JGroups <br /></h3><p style="text-align: left;">How does the JGroups lock service [2] handle partitions? Well, <b>it doesn't!</b></p><p style="text-align: left;">Let's assume A holds lock L before the partition. C wants to acquire L, but has to wait until A releases it, or crashes. Now partition {A} | {B,C} occurs. B becomes coordinator in {B,C} and C is able to acquire L. A remains coordinator in {A}.<br /></p><p style="text-align: left;"><b>This means that both A and C now hold lock L!</b></p><p style="text-align: left;">This is because the lock service implementation in JGroups favors availability over consistency (<b>AP</b>, see [1] for details). <br /></p><p style="text-align: left;">This may be acceptable for some applications, but I suspect it's not ok for most. Typically, locks are used to make sure only 1 thread in the cluster accesses a shared resource, or performs an action that modifies some shared state. Unless these actions are idempotent, running them multiple times may wreak havoc.</p><p style="text-align: left;">Even worse, what happens when the partition heals? Now A and C are both holding lock L. The Lock interface mandates code like this:</p><p style="text-align: left;"><span style="font-family: courier;">mylock.lock();</span></p><p style="text-align: left;"><span style="font-family: courier;">try {</span></p><p style="text-align: left;"><span style="font-family: courier;"> // do some work <br /></span></p><p style="text-align: left;"><span style="font-family: courier;">}</span></p><p style="text-align: left;"><span style="font-family: courier;">finally {</span></p><p style="text-align: left;"><span style="font-family: courier;"> mylock.unlock(); <br /></span></p><p style="text-align: left;"><span style="font-family: courier;">}</span><br /></p><p style="text-align: left;">Because both A and C might be doing some work in the try-clause, the following issues arise:</p><ol style="text-align: left;"><li>How do you stop one of the two (the member which is <b>not</b> supposed to hold the lock anymore)?</li><li>Do you interrupt the thread? </li><li>What happens to the work that has already been done, ie. state changes?</li></ol><p>Point #1 shows that the try-lock-finally-unlock pattern is not a good one when it comes to taking locks away (forcefully) from a member. <b>Actually, I'm not sure a good interface exists at all for forcefully removing locks from a member!</b></p><p>Point #3 is about what should be done with the work done until the point of lock removal? Should it be rolled back? Have other threads seen the changes so far?</p><p>The try-lock-finally-unlock pattern does not guarantee ACIDity, so perhaps people who are using the lock service in this manner should replace it with distributed transactions, and roll back the transaction on lock removal (a.k.a transaction abort)?</p><p><br /></p><h3 style="text-align: left;">Consensus to the rescue?</h3><p style="text-align: left;">Perhaps we should <b>prevent</b> multiple members from being able to acquire the same lock in the first place, instead of taking away locks? How about we use a consensus based system like <a href="http://belaban.github.io/jgroups-raft/">jgroups-raft</a> [3]? This is a Raft [4] implementation built on top of JGroups.</p><p style="text-align: left;">Consensus means that a change to the system can only be made when the <b>majority</b> of the members agree. In the case of {A,B,C}, at least 2 members have to agree on a change, otherwise the change won't be applied ('committed' in Raft terms).</p><p style="text-align: left;">In terms of CAP [1], this means that <b>consistency (CP)</b> is favored over <b>availability</b>.<br /></p><p style="text-align: left;">We now introduce a <b>compare-and-swap command to acquire a lock</b>, which acquires lock L only if it is null. When a majority of the members is able to commit the command, then all members will record the outcome either as successful or failed.<br /></p><p style="text-align: left;">For example, if "L" is not set, A is able to acquire the lock by calling <b>compareAndSwap("L", null, "A")</b>. This means atomically set "L" to "A" if "L" is null.</p><p style="text-align: left;">When the partition {A} | {B,C} occurs, A still holds the lock <b>but is not able to release it</b> because the compareAndSwap("L", "A", "null") operation will fail as it doesn't get a majority.</p><p style="text-align: left;">In the other partition {B,C}, B may become leader, but nobody will be able to acquire the lock as the compare-and-swap operation will fail because "L" is not null, but set to "A".<br /></p><p style="text-align: left;"><b>This means that members which want to acquire L have to wait until the partition has healed and A releases the lock. Even worse: if A crashed, it would be to be restarted, in order to release the lock! This is not better than using a DistributedLockManager (DLM) described at the top of this blog!<br /></b></p><p>We have to ask ourselves whether locks make sense in a consensus-based system. If we make changes to a shared state, then - instead of using locks - we should use the consensus-based system directly to make changes. After all, consensus will serialize state changes, which is what locks promise.<br /></p><p> </p><h3 style="text-align: left;">Conclusion</h3><p style="text-align: left;">AP (JGroups lock service) and CP (jgroups-raft) lie at the opposite ends of the reliability spectrum, and applications have to choose between them.<br /></p><p style="text-align: left;">Like choosing between pest and
cholera, when using the lock service in JGroups, due to the AP properties, we can have multiple holders of the same lock.<br /></p><p style="text-align: left;">With jgroups-raft and its CP properties, only 1 member holds a lock at any given time, but members trying to acquire a lock may potentially be blocked for a long time.</p><p style="text-align: left;"><br /></p><p style="text-align: left;">[1] <a href="https://en.wikipedia.org/wiki/CAP_theorem">https://en.wikipedia.org/wiki/CAP_theorem</a></p><p style="text-align: left;">[2] <a href="http://www.jgroups.org/manual5/index.html#LockService">http://www.jgroups.org/manual5/index.html#LockService</a></p><p style="text-align: left;">[3] <a href="http://belaban.github.io/jgroups-raft/">http://belaban.github.io/jgroups-raft/</a><br /></p><p style="text-align: left;">[4] <a href="https://raft.github.io/">https://raft.github.io/</a></p><p style="text-align: left;"><br /></p>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-19310100840487169002020-09-04T10:17:00.000+02:002020-09-04T10:17:31.402+02:00One size fits all JGroups?<p>We're getting one step closer to having just a single JGroups program that runs in any environment! There are 3 things that need to be made to make this possible:</p><ol style="text-align: left;"><li>Multiple discovery protocols: this allows for multiple discovery protocols to be present in the same configuration. For example, DNS_PING or KUBE_PING to run in Kubernetes environments, MPING when IP multicasting is available, TCPPING for a static list of members etc. DONE: [1]</li><li>Multiple transports: this can run a UDP and TCP transport side by side. If IP multicasting is not available, we can fall back to TCP. Or, even if multicasting is available, use TCP for one-to-one messages and UDP for one-to-all messages. NOT DONE yet: [2]<br /></li><li>Use GraalVM to compile this down to a native executable. This could be shipped in a Docker image, so it could be run anywhere Docker/Kubernetes is available. NOT DONE yet.<br /></li></ol><p>Step #3 is optional, but would help for quick startup times.</p><p>Step #2 is not really needed if we know that all environments run in a cloud where IP multicasting is not supported, so we can ship configs with TCP as transport. But if we know that some customers deploy locally, where IP multicasting is available, and others in environments where multicasting is disabled, or in clouds, then multiple transports will be helpful, as we can ship and support a single configuration.</p><p>Step #1 is probably the most important one: there are ~13-15 discovery protocols available today, reflecting the wide range of different environments. Being able to ship a config that includes multiple discovery protocols allows us to support a single configuration for many different customers.</p><p>In the future, we could think of code that looks at unused/inactive discovery protocols, or even transports, and removes them after some time. Kind of like just-in-time (JIT) optimizations in the JVM...</p><p>Feature [1] will be in 5.1. If you want to try this out today, head over to Github [3], clone the JGroups repo and generate your own JAR.</p><p>Cheers,<br /></p><p><br /></p><p>[1] <a href="https://issues.redhat.com/browse/JGRP-2230">https://issues.redhat.com/browse/JGRP-2230</a></p><p>[2] <a href="https://issues.redhat.com/browse/JGRP-1424">https://issues.redhat.com/browse/JGRP-1424</a></p><p>[3] <a href="https://github.com/belaban/JGroups">https://github.com/belaban/JGroups</a> <br /></p><p> <br /></p><p><br /></p>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com1tag:blogger.com,1999:blog-19835054.post-74289081602158660732020-08-06T10:06:00.000+02:002020-08-06T10:06:11.655+02:00JGroups 5.0.0.Final released<div>I'm happy to announce that JGroups 5.0.0.Final has been released!</div><div><br /></div><div>The new features are described in [1]. Below's a list of the major JIRAs:</div><div><ul style="text-align: left;"><li><a href="https://issues.redhat.com/browse/JGRP-2218">https://issues.redhat.com/browse/JGRP-2218</a>: this is the most important change in 5.0.0: it changes Message into an interface and allows for different implementations of Message</li><li><a href="https://issues.redhat.com/browse/JGRP-2450">https://issues.redhat.com/browse/JGRP-2450</a>: support for virtual threads (fibers). If the JDK (probably 16 and higher) supports virtual threads, then they can be enabled by setting use_fibers to true in the transport. This will effectively bypass the thread pool(s) and use virtual threads instead. See [2] for details.</li><li><a href="https://issues.redhat.com/browse/JGRP-2451">https://issues.redhat.com/browse/JGRP-2451</a>: FD_ALL3 is a more efficient failure detection protocol; counts messages received from P as heartbeats, and P suppresses heartbeats when sending messages. This should reduce traffic on the network</li><li><a href="https://issues.redhat.com/browse/JGRP-2462">https://issues.redhat.com/browse/JGRP-2462</a>: implementation of Random Early Drop (RED) protocol, which starts dropping messages on the send side when the queue becomes full. This prevents message storms (by unneeded retransmission requests when messages are not received) and/or blocking</li><li><a href="https://issues.redhat.com/browse/JGRP-2402">https://issues.redhat.com/browse/JGRP-2402</a>: new protocol SOS to captures vital stats and dump them to a file periodically</li><li><a href="https://issues.redhat.com/browse/JGRP-2401">https://issues.redhat.com/browse/JGRP-2401</a>: versioned configuration. Stacks won't start if the versions of JGroups and the configuration differ (not for micro versions). This prevents use of old/outdated configurations with a newer JGroups release</li><li><a href="https://issues.redhat.com/browse/JGRP-2476">https://issues.redhat.com/browse/JGRP-2476</a>: more efficient marshalling of classes. Reduces size of RPCs in RpcDispatcher<br /></li></ul></div><div>The documentation can be found at [3].</div><div>Enjoy!<br /></div><div><br /></div><div><br /></div><div>[1] <a href="http://belaban.blogspot.com/2020/01/first-alpha-of-jgroups-50.html">http://belaban.blogspot.com/2020/01/first-alpha-of-jgroups-50.html</a></div><div>[2] <a href="http://belaban.blogspot.com/2020/07/double-your-performance-virtual-threads.html">http://belaban.blogspot.com/2020/07/double-your-performance-virtual-threads.html</a></div><div>[3] <a href="http://www.jgroups.org/manual5/index.html">http://www.jgroups.org/manual5/index.html</a><br /></div><div><br /></div><div><br /></div>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-35721618635562136612020-07-17T09:49:00.002+02:002020-07-17T11:44:33.337+02:00Double your performance: virtual threads (fibers) and JDK 15/16!If you use UDP as transport and want to <b>double your performance</b>: read on!<br />
<br />
If you use TCP, your performance won't change much. You might still be interested in what more recent JDKs and <a href="https://openjdk.java.net/projects/loom/">virtual threads</a> (used to be called 'fibers') will bring to the table.<br />
<br />
<h3>
Virtual threads </h3>
<br />
Virtual threads are lightweight threads, similar in concept to the old Green Threads, and are managed by the JVM rather than the kernel. Many virtual threads can map to the same OS native (carrier) thread (only one at a time, of course), so we can have millions of virtual threads.<br />
<br />
Virtual threads are implemented with continuations, but that's a detail. What's important is that all blocking calls in the JDK (LockSupport.park() etc) have been modified to <b>yield</b> rather than block. This means that we don't waste the precious native carrier thread, but simply go to a non-RUNNING state. When the block is over, the thread is simply marked as RUNNABLE again and the scheduler continues the continuation where it left off.<br />
<br />
Main advantages:<br />
<ul>
<li>Blocking calls don't need to be changed, e.g. into reactive calls</li>
<li>No need for thread pools: simply create a virtual thread</li>
<li>Fewer context switches (reduced/eliminated blocking calls) </li>
<li>We can have lots of virtual threads</li>
</ul>
It will be a while until virtual threads show up in your JDK, but JGroups has already added support for it: just set <span style="font-family: "courier new", "courier", monospace;">use_fibers="true"</span> in the transport. If the JVM supports virtual threads, they will be used, otherwise we fall back to regular native threads.<br />
<br />
<br />
<h3>
UDP: networking improvements </h3>
While virtual threads bring advantages to JGroups, the other performance increase can be had by trying a more recent JDK.<br />
<br />
Starting in <a href="http://openjdk.java.net/jeps/373">JDK 15</a>, the implementation of DatagramSockets and MulticastSockets has been changed to delegate to DatagramChannels and MulticastChannels. In addition, virtual threads are supported.<br />
<br />
This increases the performance of UDP which uses DatagramChannels and MulticastChannels. <br />
<br />
The combination of networking code improvements and virtual threads leads to astonishing results for UDP, read below. <br />
<br />
<h3>
Performance</h3>
I used <a href="http://www.jgroups.org/manual5/index.html#UPerf">UPerf</a> for testing on a cluster of 8 (physical) boxes (1 GBit ethernet), with JDKs 11, 16-ea5 and 16-loom+2-14. The former two use native threads, the latter uses virtual threads.<br />
<br />
As can be seen in [1], UDP's performance goes from <b>44'691</b> on JDK 11 to <b>81'402</b> on JDK 16-ea5; that's a whopping <b>82% increase</b>! Enabling virtual threads increases the performance between 16-ea5 and 16-loom+2-14 to <b>88'252</b>, that's another <b>8%</b>!<br />
<br />
<b>The performance difference between JDK 11 and 16-loom is 97%!</b><br />
<br />
The difference in TCP's performance is miniscule; I guess because the TCP code was already optimized in JDK 11.<br />
<br />
Running in JDK 16-loom+2-14 shows that UDP's performance is now on par with TCP, as a matter of fact, <b>UDP is even 3% faster than TCP</b>!<br />
<br />
If you want to try for yourself: head over to the <a href="https://github.com/belaban/JGroups">JGroups Github repo</a> and create the JAR (ant jar). Or wait a bit: I will soon release 5.0.0.Final which contains the changes.<br />
<br />
Not sure if I want to backport the changes to the 4.x branch...<br />
<br />
Enjoy! <br />
<br />
[1] <a href="https://drive.google.com/file/d/1Ars1LOM7cEf6AWpPwZHeIfu_kKLa9gv0/view?usp=sharing">https://drive.google.com/file/d/1Ars1LOM7cEf6AWpPwZHeIfu_kKLa9gv0/view?usp=sharing</a><br />
<br />
[2] <a href="http://openjdk.java.net/jeps/373">http://openjdk.java.net/jeps/373</a><br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-41743768433227800732020-06-30T15:50:00.003+02:002020-06-30T15:50:55.493+02:00New Netty transportI'm happy to announce that Baizel Mathew and Steven Wong have written a new transport protocol using Netty!<br />
<br />
Read Baizel's announcement here: [1]; for the code look here: [2].<br />
<br />
I anticipate that the Netty transport will replace TCP_NIO2 over time.<br />
<br />
The maven coordinates are:<br />
<br />
<span style="font-family: "Courier New", Courier, monospace;"><dependency><br /> <groupId>org.jgroups</groupId><br /> <artifactId>jgroups-netty</artifactId><br /> <version>1.0.0.Alpha2</version><br /></dependency></span> <br />
<br />
Thanks, Baizel and Steven, for your contribution!<br />
<br />
<br />
[1] <a href="https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/jgroups-dev/R3yxmfhcqMk/ugked7zaAgAJ">https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/jgroups-dev/R3yxmfhcqMk/ugked7zaAgAJ</a><br />
[2] <a href="https://github.com/jgroups-extras/jgroups-netty">https://github.com/jgroups-extras/jgroups-netty</a><br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-39405526339172220252020-04-20T12:45:00.000+02:002020-04-20T12:45:59.000+02:00Hybrid clouds with JGroups and SkupperThis is a follow-up post on [<a href="http://belaban.blogspot.com/2019/12/spanning-jgroups-kubernetes-based.html">1</a>], which showed how to connect two Kubernetes-based hybrid clouds (Google GKE and AWS EKS) with JGroups' TUNNEL and GossipRouter.<br />
<br />
Meanwhile, I've discovered <a href="https://skupper.io/">Skupper</a>, which (1) simplifies this task and (as a bonus) (2) encrypts the data exchanged between different clouds.<br />
<br />
In this post, I'm going to provide step-by-step instructions on how to connect a Google Kubernetes Engine (GKE) cluster with a cluster running on my local box.<br />
<br />
To run the demo yourself, you must have Skupper installed and a GKE account. However, any other cloud provider works, too.<br />
<br />
For the local cloud, I'm using docker-desktop. Alternatively, minikube could be used.<br />
<br />
So let's get cracking, and start the GKE cluster. To avoid having to switch contexts with kubectl all the time, I suggest start 2 separate shells and set KUBECONFIG for the public (GKE) cloud to a copy of config:<br />
<br />
Shell 1 (GKE): <span style="font-family: "Courier New", Courier, monospace;">cp .kube/config .kube/gke; export KUBECONFIG=$HOME/.kube/gke</span><br />
<br />
Now start a GKE cluster (in shell 1):<br />
<span style="font-family: "Courier New", Courier, monospace;">gcloud container clusters create gke --num-nodes 4</span><br />
<br />
NOTE: if you use a different cloud, simply start your cluster and set kubectl's context to point to your cluster. The rest of the instructions below apply regardless of the specific cloud. <br />
<br />
This sets the Kubernetes context (shell 1):<span style="font-family: "Courier New", Courier, monospace;"> </span><br />
<span style="font-family: "Courier New", Courier, monospace;">kubectl config current-context<br />gke_ispnperftest_us-central1-a_gke </span><br />
<span style="font-size: small;"><span style="font-family: "Courier New", Courier, monospace;"></span></span><br />
In shell 2, confirm that the context is local:<br />
<span style="font-family: "Courier New", Courier, monospace;">kubectl config current-context<br />docker-desktop </span><br />
<span style="font-family: "Courier New", Courier, monospace;"> </span><br />
<br />
This shows Kubernetes is pointing to docker-desktop.<br />
<br />
Let's now start a GossipRouter in both clouds. To do this, we have to modify the YAML used in [1] slightly:<br />
<span style="font-family: "Courier New", Courier, monospace;">curl https://raw.githubusercontent.com/belaban/jgroups-docker/master/yaml/gossiprouter.yaml > gossiprouter.yaml</span><br />
<br />
Now comment lines 42-43:<br />
<span style="font-family: "Courier New", Courier, monospace;">spec:<br /># type: LoadBalancer<br /># externalTrafficPolicy: Local</span><br />
<br />
This is needed by Skupper which requires a service to be exposed as a <b>ClusterIP</b> and not a <b>LoadBalancer</b>.<br />
<br />
Now deploy it <b>in both shells</b>:<br />
<span style="font-family: "Courier New", Courier, monospace;">kubectl apply -f gossiprouter.yaml<br />deployment.apps/gossiprouter created<br />service/gossiprouter created</span><br />
<br />
Now it is time to initialize Skupper <b>in both shells</b>:<br />
<span style="font-family: "Courier New", Courier, monospace;">skupper init</span><br />
<span style="font-family: "Courier New", Courier, monospace;">Waiting for LoadBalancer IP or hostname...<br />Skupper is now installed in namespace 'default'. Use 'skupper status' to get more information.</span><br />
<br />
This installs some pods and services/proxies:<br />
<span style="font-family: "Courier New", Courier, monospace;">kubectl get po,svc<br />NAME READY STATUS RESTARTS AGE<br />pod/gossiprouter-6d6dcd6d79-q9p2f 1/1 Running 0 4m6s<br />pod/skupper-proxy-controller-dcf99c6bf-whns4 1/1 Running 0 86s<br />pod/skupper-router-7976948d9f-b58wn 1/1 Running 0 2m50s<br /><br />NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br />service/gossiprouter ClusterIP 10.27.252.196 <none> 8787/TCP,9000/TCP,12001/TCP 4m6s<br />service/kubernetes ClusterIP 10.27.240.1 <none> 443/TCP 27m<br />service/skupper-controller LoadBalancer 10.27.241.112 35.223.80.171 8080:30508/TCP 2m49s<br />service/skupper-internal LoadBalancer 10.27.243.17 35.192.126.100 55671:30671/TCP,45671:31522/TCP 2m48s<br />service/skupper-messaging ClusterIP 10.27.247.95 <none> 5671/TCP 2m49s</span><br />
<br />
Next, we create a connection token in one of the clouds. This creates a file containing a certificate and keys that allows a Skupper instance in one cluster to connect to a Skupper instance in another cluster.<br />
<br />
<b>Note that this file must be kept secret as it contains the private keys of the (server) Skupper instance!</b><br />
<br />
We only need to connect from one cloud to the other, Skupper will automatically create a bi-directional connection.<br />
<br />
Let's pick the public cloud (shell 1):<br />
<span style="font-family: "Courier New", Courier, monospace;">skupper connection-token gke.secret<br />Connection token written to gke.secret</span> <br />
<br />
We now need to copy this file to the other (local) cloud. In my example, I'm using the home directory, but in real-life, this would have to be done secretly.<br />
<br />
The local Skupper instance now uses this file to connect to the Skupper instance in the public cluster and establish an encrypted VPN tunnel:<br />
<span style="font-family: "Courier New", Courier, monospace;">kupper connect gke.secret <br />Skupper is now configured to connect to 35.192.126.100:55671 (name=conn1)</span><br />
<br />
Now, we have to expose the GossipRouter service in each cloud to Skupper, so Skupper can create a local proxy of the service that transparently connects to the other cloud, via a symbolic name:<br />
Shell 1:<br />
<span style="font-family: "Courier New", Courier, monospace;">skupper expose deployment gossiprouter --port 12001 --address gossiprouter-1</span><br />
Shell 2:<br />
<span style="font-family: "Courier New", Courier, monospace;">skupper expose deployment gossiprouter --port 12001 --address gossiprouter-2</span><br />
<br />
<b>The symbolic names gossiprouter-1 and gossiprouter-2 are now available to any pod in both clusters.</b><br />
<b>Traffic sent from the local cluster to gossiprouter-1 in the public cluster is transparently (and encryptedly) forwarded by Skupper between the sites!</b><br />
<br />
This means, we can set <span style="font-family: "Courier New", Courier, monospace;">TUNNEL_INITIAL_HOSTS</span> (as used in the bridge cluster) to<br />
<span style="font-family: "Courier New", Courier, monospace;">gossiprouter1[12001],gossiprouter-2[12001]</span>.<br />
<br />
This is used in <a href="http://bridge.xml/"></a><a href="https://github.com/belaban/jgroups-docker/blob/master/conf/bridge.xml">bridge.xml</a>:<br />
<span style="font-family: "Courier New", Courier, monospace;"><TUNNEL bind_addr="match-interface:eth0,site-local" gossip_router_hosts="${TUNNEL_INITIAL_HOSTS:127.0.0.1[12001]}"</span><br />
<span style="font-family: "Courier New", Courier, monospace;">...</span><br />
<br />
<br />
Let's now run <a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/demos/RelayDemo.java">RelayDemo</a> in the public and local clusters. This is the same procedure as in [1].<br />
<br />
Shell 1: <br />
<span style="font-family: "Courier New", Courier, monospace;">curl https://raw.githubusercontent.com/belaban/jgroups-docker/master/yaml/nyc.yaml > public.yaml</span><br />
<br />
Shell 2:<br />
<span style="font-family: "Courier New", Courier, monospace;">curl https://raw.githubusercontent.com/belaban/jgroups-docker/master/yaml/sfc.yaml > local.yaml</span><br />
<br />
In both YAML files, change the number of replicas to 3 and the value of TUNNEL_INITIAL_HOSTS to <span style="font-family: "Courier New", Courier, monospace;">"gossiprouter-1[12001],gossiprouter-2[12001]"</span>.<br />
<br />
Then start 3 pods in the public (NYC) and local (SFC) clusters:<br />
Shell 1:<br />
<span style="font-family: "Courier New", Courier, monospace;">kubectl apply -f public.yaml<br />deployment.apps/nyc created<br />service/nyc created</span><br />
<br />
Shell 2:<br />
<span style="font-family: "Courier New", Courier, monospace;">kubectl apply -f local.yaml<br />deployment.apps/sfc created<br />service/sfc created</span><br />
<br />
Verify that there are 3 pods running in each cluster.<br />
<br />
Let's now run RelayDemo on the local cluster:<br />
Shell 2: <br />
<span style="font-family: "Courier New", Courier, monospace;">> kubectl get pods |grep sfc-<br />sfc-7f448b7c94-6pb9m 1/1 Running 0 2m44s<br />sfc-7f448b7c94-d7zkp 1/1 Running 0 2m44s<br />sfc-7f448b7c94-ddrhs 1/1 Running 0 2m44s</span><br />
<span style="font-family: "Courier New", Courier, monospace;"><br /></span>
<span style="font-family: "Courier New", Courier, monospace;">> kubectl exec -it sfc-7f448b7c94-6pb9m bash<br />bash-4.4$ relay.sh -props sfc.xml -name Local<br /><br />-------------------------------------------------------------------<br />GMS: address=Local, cluster=RelayDemo, physical address=10.1.0.88:7801<br />-------------------------------------------------------------------<br />View: [sfc-7f448b7c94-6pb9m-4056|3]: sfc-7f448b7c94-6pb9m-4056, sfc-7f448b7c94-ddrhs-52643, sfc-7f448b7c94-d7zkp-11827, Local<br />: hello<br />: << hello from Local<br /><< response from sfc-7f448b7c94-6pb9m-4056<br /><< response from sfc-7f448b7c94-ddrhs-52643<br /><< response from sfc-7f448b7c94-d7zkp-11827<br /><< response from Local<br /><< response from nyc-6b4846f777-g2gqk-7743:nyc<br /><< response from nyc-6b4846f777-7jm9s-23105:nyc<br /><< response from nyc-6b4846f777-q2wrl-38225:nyc</span><br />
<br />
<br />
We're first listing all pods, then exec into one of them.<br />
<br />
Next, we're running RelayDemo and send a message to all members of the local and remote clusters. We can see that we get a response from self (Local) and the other 3 members of the local (SFC) cluster, and we also get responses from the 3 members of the remote public cluster (NYC).<br />
<br />
JGroups load-balances messages across one of the two GossipRouters. Each time, the router is remote, Skupper forwards the traffic transparently over its VPN tunnel to the other site.<br />
<br />
<br />
[1] <a href="http://belaban.blogspot.com/2019/12/spanning-jgroups-kubernetes-based.html">http://belaban.blogspot.com/2019/12/spanning-jgroups-kubernetes-based.html</a><br />
[2] <a href="https://skupper.io/">https://skupper.io/</a><br />
<br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-26371374788376354632020-01-28T10:14:00.000+01:002020-01-28T10:14:35.264+01:00First alpha of JGroups 5.0Howdy folks!<br />
<br />
Today I'm very happy to announce the first alpha version of JGroups 5.0!<br />
<br />
JGroups 5.0 has major API changes and I'd like people to try it out and give feedback before we release final.<br />
<br />
Note that there might still be more API changes before the first beta.<br />
<br />
So what's new in 5?<br />
<br />
The biggest change is that Message is now an interface [1] and we have a number of message classes implementing it, e.g.:<br />
<ul>
<li><a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/BytesMessage.java">BytesMessage</a>: this is the replacement for the old 4.x Message class, having a byte array as payload.</li>
<li><a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/ObjectMessage.java">ObjectMessage</a>: accepts an object as payload.</li>
<li><a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/NioMessage.java">NioMessage</a>: has an NIO ByteBuffer as payload.</li>
<li><a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/EmptyMessage.java">EmptyMessage</a>: this class has *no* payload at all! Useful when sending around messages that have only headers, e.g. heartbeats. Used mainly by JGroups internally. This class has a smaller memory footprint.</li>
<li><a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/CompositeMessage.java">CompositeMessage</a>: message type which carries other messages</li>
</ul>
The advantage is different message types is that rather than having to marshal payloads into a byte array, as in 4.x Messages, the payload is now added to the message without marshalling. Marshalling is only done just before sending the message on the network.<br />
<br />
This <a href="http://www.jgroups.org/manual5/index.html#_late_marshalling">late marshalling</a> saves one memory allocation.<br />
<br />
The other advantage is that applications can <a href="http://www.jgroups.org/manual5/index.html#MessageFactory">register their own messages types</a>. This means that we can control how a message is created, e.g. using off-heap memory rather than heap memory.<br />
<br />
Other changes include:<br />
<ul>
<li>I've removed a lot of deprecated cruft, e.g. several AuthToken implementations, SASL, S3_PING and GOOGLE_PING (they have better replacements).</li>
<li>Java 11 is now the baseline. The current Alpha1 still runs under Java 8, but I expect this to change, perhaps only with 5.1. But at least, I reserve the right to use Java 11 specific language features, so be warned :-)</li>
</ul>
The full list of 5.0 is here: [2].<br />
<br />
I still have a few JIRAs to resolve before releasing 5.0.0.Final, and then I'll add new functionality (without API changes) in a bunch of minor releases. I've planned 5.1 - 5.3 so far.<br />
<br />
The documentation is here: [3].<br />
<br />
For feedback please use the <a href="http://groups.google.com/forum/#!forum/jgroups-dev">mailing list</a> [4].<br />
<br />
Enjoy!<br />
<br />
<br />
<br />
[1] <a href="http://www.jgroups.org/manual5/index.html#Message">http://www.jgroups.org/manual5/index.html#Message</a><br />
[2] <a href="https://issues.redhat.com/projects/JGRP/versions/12334686">https://issues.redhat.com/projects/JGRP/versions/12334686</a><br />
[3] <a href="http://www.jgroups.org/manual5/index.html">http://www.jgroups.org/manual5/index.html</a><br />
[4] <a href="http://groups.google.com/forum/#!forum/jgroups-dev">http://groups.google.com/forum/#!forum/jgroups-dev</a><br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-8748377774316316612019-12-31T10:01:00.002+01:002019-12-31T10:21:27.872+01:00Spanning JGroups Kubernetes-based clusters across Google and Amazon cloudsIn this (long!) post, I'll provide step-by-step instructions on how to create JGroups clusters in Google Kubernetes Engine (GKE) and Amazon (EKS) clusters, and connect them into one virtual cluster using <a href="http://www.jgroups.org/manual4/index.html#Relay2Advanced">RELAY2</a>.<br />
<br />
Each local cluster is called a <b>site</b>. In this tutorial, we'll call the sites <b>NYC</b> and <b>SFC</b>. We'll start 5 nodes in NYC and 3 in SFC.<br />
<br />
The sample deployments and services are defined in YAML and we're using <b>Kubernetes</b> to create the clusters.<br />
<br />
To try this yourself, you'll need <b>kubectl</b>, <b>eksctl</b> and <b>gcloud</b> installed, and accounts on both EKS and GKE.<br />
<br />
The demo is <a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/demos/RelayDemo.java">RelayDemo</a> [1]. It is a simple chat, started in a pod, and every typed line appears in all pods across all sites, and then every pod sends a response back to the sender, which displays all responses. This way, we know who received our chat message.<br />
<br />
<br />
<h3>
Architecture</h3>
The setup of this tutorial is as follows:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-yrs8If-J8a0/XgoYQyGVWxI/AAAAAAABSl8/bzaVzzzkFtUYvW-CevaQSYkSCtatWVyJgCLcBGAsYHQ/s1600/xsite.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="412" data-original-width="1231" height="212" src="https://1.bp.blogspot.com/-yrs8If-J8a0/XgoYQyGVWxI/AAAAAAABSl8/bzaVzzzkFtUYvW-CevaQSYkSCtatWVyJgCLcBGAsYHQ/s640/xsite.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
On the left, we have nodes A,B,C,D,E in site NYC (Amazon EKS) and on the right, X,Y,Z in SFC (Google GKE).<br />
<br />
A in NYC and X in SFC assume the role of <b>site master</b> (see [2]). This means, they join a separate JGroups cluster, called <b>bridge</b> <b>cluster</b>, which connects the two sites, and relay messages between the sites.<br />
<br />
A site master is not a dedicated node, but any node can assume the role of site master. For example, when A leaves or crashes, B will take over the site master role, join the bridge cluster and relay messages between sites NYC and SFC.<br />
<br />
<b>The problem with Kubernetes / Openshift is that a pod cannot directly connect to a pod in a different cluster, region, or cloud provider</b>. That is, without resorting to specific container network implementations (CNI) implementations.<br />
<br />
To overcome this problem, the above setup uses a <b>GossipRouter</b> and <b>TUNNEL</b> [3]: this way, <b>A and X can communicate across different regions or (in this case) even different cloud providers</b>.<br />
<br />
The way this is done is simple: the configuration of the bridge cluster includes TUNNEL as transport and a list of GossipRouters, in this case the ones in NYC and SFC (more details later).<br />
<br />
A and X connect to both GossipRouters via TCP, under their respective cluster names. So A connects to GR-NYC and GR-SFC and X connects to its local GR, and the remote one in NYC.<br />
<br />
When A wants to send a message to X, it can use either its local GossipRouter, or the one in SFC (by default, JGroups load-balances requests between the GossipRouters). In any case, <b>the ingress TCP connection established by X to a GossipRouter is used to send egress traffic to X</b>.<br />
<br />
This means, we can send messages to any member of the bridge cluster, as long as <b>all GossipRouters are publicly accessible and the members of the bridge cluster can connect to them</b>.<br />
<br />
But now let's get cracking! We'll do the following in the next sections:<br />
<ul>
<li>Set up an EKS cluster (NYC)</li>
<li>Set up a GKE cluster (SFC)</li>
<li>Deploy a GossipRouter service in both sites</li>
<li>Deploy 5 pods in NYC and 3 pods in SFC</li>
<li>Use one of the pods in each site to talk to the other site with RelayDemo</li>
</ul>
<br />
<h3>
Set up the NYC cluster in EKS</h3>
This can be done via the GUI, the AWS CLI or eksctl [4]. For simplicity, I chose the latter.<br />
To create a cluster "nyc" in the us-east-1 region, execute:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">eksctl create cluster --name nyc --region us-east-1 --nodegroup-name nyc-nodegroup --node-type t3.small --nodes 2 --nodes-min 1 --nodes-max 4 --managed</span><br />
<br />
This will take 10-15 minutes.<br />
<br />
The local kubeconfig should now point to the AWS cluster. This can be seen with kubectl config get-contexts. If this is not the case, use the AWS CLI to change this, e.g.:<br />
<pre class="programlisting"><code class="nohighlight hljs"><code></code></code>
<code></code></pre>
<span style="font-family: "courier new" , "courier" , monospace;">aws eks --region use-east-1 update-kubeconfig --name nyc</span><br />
<br />
This make kubectl access the NYC cluster by default.<br />
<br />
Let's now deploy the GossipRouter in NYC:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">kubectl apply -f https://raw.githubusercontent.com/belaban/jgroups-docker/master/yaml/gossiprouter.yaml</span><br />
<br />
The YAML file contains a deployment of the GossipRouter and a LoadBalancer service: [5]. The public address of the GossipRouter service can be seen as follows:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">kubectl get svc gossiprouter<br />NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br />gossiprouter LoadBalancer 10.100.28.38 a6abc71e42b2211ea9c3716e7fa74966-862f92ba6a28fd36.elb.us-east-1.amazonaws.com 8787:31598/TCP,9000:30369/TCP,12001:31936/TCP 2m56s</span><br />
<br />
We can see that the public address is a6abc71e42b2211ea9c3716e7fa74966-862f92ba6a28fd36.elb.us-east-1.amazonaws.com. Write this down somewhere, as we'll need to add it to our TUNNEL configuration later.<br />
<br />
<br />
<h3>
Set up SFC cluster in GKE</h3>
To create a cluster on GKE, execute:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">gcloud container clusters create sfc --num-nodes 2</span> <br />
<br />
This will create a cluster in the default region configured in gcloud.<br />
<br />
Note that this added a new context to the kube config, and switched to it. If not, manually switch to it, e.g.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">kubectl config use-context gke_ispnperftest_us-central1-a_sfc</span><br />
<br />
Now deploy the GossipRouter in SFC (same as above, for NYC):<br />
<span style="font-family: "courier new" , "courier" , monospace;">kubectl apply -f https://raw.githubusercontent.com/belaban/jgroups-docker/master/yaml/gossiprouter.yaml</span><br />
<br />
Now get the public IP address of the GossipRouter:<br />
<span style="font-family: "courier new" , "courier" , monospace;">kubectl get svc gossiprouter<br />NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br />gossiprouter LoadBalancer 10.19.247.254 35.232.92.116 8787:30150/TCP,9000:32534/TCP,12001:32455/TCP 101s</span><br />
<br />
The public IP is 35.232.92.116. Take a note of this, as we'll need it later.<br />
We're now ready to deploy the cluster nodes in NYC and SFC.<br />
<br />
<h3>
Deploy the pods in NYC</h3>
We'll deploy 5 pods in NYC. To do this, we first need to switch the context back to NYC, e.g. by executing<br />
<span style="font-family: "courier new" , "courier" , monospace;">kubectl config use-context jgroups@nyc.us-east-1.eksctl.io</span><br />
<br />
Next, download the 2 YAML files for NYC and SFC locally (we need to make changes):<br />
<span style="font-family: "courier new" , "courier" , monospace;">mkdir tmp ; cd tmp</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">curl https://raw.githubusercontent.com/belaban/jgroups-docker/master/yaml/nyc.yaml > nyc.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">curl https://raw.githubusercontent.com/belaban/jgroups-docker/master/yaml/sfc.yaml > sfc.yaml </span><br />
<br />
Now edit both YAML files and replace the TUNNEL_INITIAL_HOSTS system variable <span style="font-family: "courier new" , "courier" , monospace;">"load-balancer-1[12001],load-balancer-2[12001]"</span> with<br />
<span style="font-family: "courier new" , "courier" , monospace;">"a6abc71e42b2211ea9c3716e7fa74966-862f92ba6a28fd36.elb.us-east-1.amazonaws.com[12001],35.232.92.116[12001]"</span>.<br />
<br />
This points the TUNNEL protocol to the two publicly accessible GossipRouters in NYC and SFC:<br />
<br />
<span style="background-color: #efefef;"><</span><span style="background-color: #efefef; color: navy; font-weight: bold;">TUNNEL</span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef; color: green; font-weight: bold;"></span><br />
<span style="background-color: #efefef; color: green; font-weight: bold;"></span><br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9,0pt;"><span style="background-color: #efefef; color: green; font-weight: bold;"> </span><span style="background-color: #efefef; color: blue; font-weight: bold;">port_range</span><span style="background-color: #efefef; color: green; font-weight: bold;">="${PORT_RANGE:0}"</span><span style="background-color: #efefef; color: green; font-weight: bold;"> </span><span style="background-color: #efefef; color: blue; font-weight: bold;">gossip_router_hosts</span><span style="background-color: #efefef; color: green; font-weight: bold;">="${TUNNEL_INITIAL_HOSTS:127.0.0.1[12001]}"/></span></pre>
<br />
This means that TUNNEL will establish 2 TCP connections, one to the GossipRouter in NYC and the other one to the GossipRouter in SFC.<br />
<br />
Now deploy the NYC pods:<br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl apply -f tmp/nyc.yaml<br />deployment.apps/nyc created<br />service/nyc created</span><br />
<br />
This shows that 1 pod has been created:<br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl get pods -o wide<br />NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES<br />gossiprouter-f65bb6858-jks8q 1/1 Running 0 25m 192.168.36.19 ip-192-168-38-111.ec2.internal <none> <none><br />nyc-5f4964d444-9v5dm 1/1 Running 0 73s 192.168.26.87 ip-192-168-8-51.ec2.internal <none> <none></span><br />
<br />
Next, scale this to 5:<br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl scale --replicas=5 deployment nyc<br />deployment.extensions/nyc scaled</span><br />
<br />
Listing the pods shows 5 'nyc' pods:<br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl get pods<br />NAME READY STATUS RESTARTS AGE<br />gossiprouter-f65bb6858-jks8q 1/1 Running 0 27m<br />nyc-5f4964d444-2ttfp 1/1 Running 0 49s<br />nyc-5f4964d444-4lccs 1/1 Running 0 49s<br />nyc-5f4964d444-8622d 1/1 Running 0 49s<br />nyc-5f4964d444-9v5dm 1/1 Running 0 3m21s<br />nyc-5f4964d444-tm5h5 1/1 Running 0 49s</span><br />
<br />
Let's exec into one of the and make sure that the local cluster formed:<br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl exec nyc-5f4964d444-2ttfp probe.sh<br />#1 (307 bytes):<br />local_addr=nyc-5f4964d444-2ttfp-24388<br />physical_addr=192.168.53.43:7800<br />view=[nyc-5f4964d444-9v5dm-21647|4] (5) [nyc-5f4964d444-9v5dm-21647, nyc-5f4964d444-tm5h5-64872, nyc-5f4964d444-2ttfp-24388, nyc-5f4964d444-8622d-63103, nyc-5f4964d444-4lccs-4487]<br />cluster=RelayDemo<br />version=4.1.9-SNAPSHOT (Mont Ventoux)<br /><br />1 responses (1 matches, 0 non matches)</span><br />
<br />
This shows a view of 5, so the 5 pods did indeed form a cluster.<br />
<br />
<h3>
Deploy the pods in SFC</h3>
Let's now switch the kubeconfig back to SFC (see above) and deploy the SFC cluster:<br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl apply -f tmp/sfc.yaml<br />deployment.apps/sfc created<br />service/sfc created</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl scale --replicas=3 deployment/sfc<br />deployment.extensions/sfc scaled</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl get pods<br />NAME READY STATUS RESTARTS AGE<br />gossiprouter-6cfdc58df5-7jph4 1/1 Running 0 21m<br />sfc-5d6774b647-25tk5 1/1 Running 0 50s<br />sfc-5d6774b647-sgxsk 1/1 Running 0 50s<br />sfc-5d6774b647-sjt9k 1/1 Running 0 88s</span><br />
<br />
<br />
This shows that we have 3 pods in SFC running.<br />
<br />
<h3>
Run the demo</h3>
So, now we can run RelayDemo to see if the virtual cluster across the two clouds is working correctly. To do this, we run a bash in one of the pods:<br />
<span style="font-family: "courier new" , "courier" , monospace;">> kubectl get pods<br />NAME READY STATUS RESTARTS AGE<br />gossiprouter-6cfdc58df5-7jph4 1/1 Running 0 28m<br />sfc-5d6774b647-25tk5 1/1 Running 0 7m50s<br />sfc-5d6774b647-sgxsk 1/1 Running 0 7m50s<br />sfc-5d6774b647-sjt9k 1/1 Running 0 8m28s<br />> kubectl exec -it sfc-5d6774b647-sgxsk bash</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">bash-4.4$ </span><br />
<br />
The RelayDemo can be started with relay.sh:<br />
<span style="font-family: "courier new" , "courier" , monospace;">relay.sh -props sfc.xml -name Temp<br /><br />-------------------------------------------------------------------<br />GMS: address=Temp, cluster=RelayDemo, physical address=10.16.1.6:7801<br />-------------------------------------------------------------------<br />View: [sfc-5d6774b647-sjt9k-37487|9]: sfc-5d6774b647-sjt9k-37487, sfc-5d6774b647-sgxsk-6308, sfc-5d6774b647-25tk5-47315, Temp<br />: </span><br />
<br />
We can see that our cluster member named 'Temp' has joined the cluster.<br />
<br />
When we send a message, we can see that all 3 members of the (local) SFC cluster and the 5 members of the (remote) NYC cluster are replying (we're also getting a reply from self):<br />
<span style="font-family: "courier new" , "courier" , monospace;">hello<br />: << response from sfc-5d6774b647-sgxsk-6308<br /><< response from sfc-5d6774b647-sjt9k-37487<br /><< response from sfc-5d6774b647-25tk5-47315<br /><< hello from Temp<br /><< response from Temp<br /><< response from nyc-5f4964d444-9v5dm-21647:nyc<br /><< response from nyc-5f4964d444-2ttfp-24388:nyc<br /><< response from nyc-5f4964d444-tm5h5-64872:nyc<br /><< response from nyc-5f4964d444-8622d-63103:nyc<br /><< response from nyc-5f4964d444-4lccs-4487:nyc</span><br />
<br />
The topology can be shown by typing 'topo' ('help' lists more commands):<br />
<span style="font-family: "courier new" , "courier" , monospace;">: topo<br /><br />nyc<br /> nyc-5f4964d444-9v5dm-21647 (192.168.26.87:7800) (me) // site master<br /> nyc-5f4964d444-tm5h5-64872 (192.168.30.27:7800)<br /> nyc-5f4964d444-2ttfp-24388 (192.168.53.43:7800)<br /> nyc-5f4964d444-8622d-63103 (192.168.62.83:7800)<br /> nyc-5f4964d444-4lccs-4487 (192.168.40.102:7800)<br /><br />sfc<br /> sfc-5d6774b647-sjt9k-37487 (10.16.1.5:7800) (me) // site master<br /> sfc-5d6774b647-sgxsk-6308 (10.16.1.6:7800)<br /> sfc-5d6774b647-25tk5-47315 (10.16.0.10:7800)<br /> Temp (10.16.1.6:7801)</span><br />
<br />
This shows the members of both sites, plus their (internal) IP addresses and who the site masters are.<br />
<br />
<h3>
Dump the contents of the GossipRouters</h3>
This can be done via a utility program shipped with JGroups:<br />
<span style="font-family: "courier new" , "courier" , monospace;">> java -cp jgroups.jar org.jgroups.tests.RouterStubGet -host 35.232.92.116 -cluster bridge</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">1: null:nyc, name=_nyc-5f4964d444-9v5dm-21647, addr=192.168.26.87:45275, server<br />2: null:sfc, name=_sfc-5d6774b647-sjt9k-37487, addr=10.16.1.5:42812, server</span><br />
<br />
This shows the members of the bridge cluster, which registered with both GossipRouters.<br />
<br />
Alternatively, the other GossipRouter can be used, but it show list the same members.<br />
<br />
<h3>
Add firewall/ingress rules to make the GossipRouter publicly available</h3>
If the GossipRouters cannot be accessed by the above command, then there has to be a firewall/ingress rule to be added to allow ingress traffic to port 12001.<br />
<br />
<h3>
Cross-site replication</h3>
The RelayDemo sample application is very basic and not very useful by itself, but the setup can be used for other types of applications, e.g. <b>replication between data centers</b>.<br />
<br />
If we have in-memory data in NYC, and use SFC as a backup for NYC (and vice versa), then a total loss of the NYC cluster will not lose all the data, but clients can be failed over to SFC and will continue to work with the data.<br />
<br />
This can be done for example by Red Hat Data Grid [6] and cross-site replication; as a matter of fact, all that needs to be done is to change the configuration, as explained in this post!<br />
<br />
As usual, send questions and feedback to the JGroups mailing list.<br />
<br />
Enjoy!<br />
<br />
<br />
<br />
[1] <a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/demos/RelayDemo.java">https://github.com/belaban/JGroups/blob/master/src/org/jgroups/demos/RelayDemo.java</a> <br />
[2] <a href="http://www.jgroups.org/manual4/index.html#Relay2Advanced">http://www.jgroups.org/manual4/index.html#Relay2Advanced</a><br />
[3] <a href="http://www.jgroups.org/manual4/index.html#TUNNEL_Advanced">http://www.jgroups.org/manual4/index.html#TUNNEL_Advanced</a><br />
[4] <a href="https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html">https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html</a><br />
[5] <a href="https://github.com/belaban/jgroups-docker/blob/master/yaml/gossiprouter.yaml">https://github.com/belaban/jgroups-docker/blob/master/yaml/gossiprouter.yaml</a><br />
[6] <a href="https://access.redhat.com/documentation/en-us/red_hat_data_grid/7.3/html/red_hat_data_grid_user_guide/x_site_replication">https://access.redhat.com/documentation/en-us/red_hat_data_grid/7.3/html/red_hat_data_grid_user_guide/x_site_replication</a><br />
<br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-63034281468504616602019-07-03T15:13:00.000+02:002019-07-03T15:13:39.167+02:00Compiling JGroups to native code with Quarkus/GraalVMI'm happy to announce the availability of a JGroups extension for <a href="https://quarkus.io/">Quarkus</a>!<br />
<br />
<br />
<h3>
What?</h3>
<br />
Quarkus is a framework that (among other things) compiles Java code down to native code (using GraalVM [4]), removing code that's not needed at run time.<br />
<br />
Quarkus analyzes the code in a <i>build phase</i>, and removes code that's not used at run time, in order to have a small executable that starts up quickly.<br />
<br />
This means that reflection cannot be used at <i>run time</i>, as all classes that are not used are removed at build time. However, reflection can be used at <i>build time</i>.<br />
<br />
The other limitations that affect JGroups are threads and the creation of sockets. Both cannot be done at build time, but have to be done at run time. (More limitations of JGroups under Quarkus are detailed in [5]).<br />
<br />
So what's the point of a providing a JGroups extension for Quarkus?<br />
<br />
While a JGroups application <i>can</i> be compiled directly to native code (using GraalVM's <span style="font-family: "courier new" , "courier" , monospace;">native-image</span>), it is cumbersome, and the application has to be restructured (see [6] for an example) to accommodate the limitations of native compilation.<br />
<br />
In contrast, the JGroups extension provides a JChannel that can be injected into the application. The channel has been created according to a configuration file and connected (= joined the cluster) by the extension. The extension takes care of doing the reflection, the socket creation and the starting of threads at the right time (build- or run-time), and the user doesn't need to worry about this.<br />
<br />
<h3>
How?</h3>
So let's take a look at a sample application (available at [2]).<br />
<br />
The POM includes the extension: <span style="font-family: "courier new" , "courier" , monospace;">groupId=org.jgroups.quarkus.extension</span> and <span style="font-family: "courier new" , "courier" , monospace;">artifactId=quarkus-jgroups</span>. This provides a JChannel that can be injected. The main class is <span style="font-family: "courier new" , "courier" , monospace;">ChatResource</span>:<br />
<br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="color: olive;">@ApplicationScoped</span><span style="color: olive;"> </span></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="color: olive;">@Path</span>(<span style="color: green; font-weight: bold;">"/chat"</span>)
<span style="color: navy; font-weight: bold;">public class </span>ChatResource <span style="color: navy; font-weight: bold;">extends </span>ReceiverAdapter <span style="color: navy; font-weight: bold;">implements </span>Publisher<String> {
<span style="color: navy; font-weight: bold;">protected final </span>Set<Subscriber<? <span style="color: navy; font-weight: bold;">super </span>String>> <span style="color: #660e7a; font-weight: bold;">subscribers</span>=<span style="color: navy; font-weight: bold;">new </span>HashSet<>();
<span style="color: olive;">@Inject </span>JChannel <span style="color: #660e7a; font-weight: bold;">channel</span>;
<span style="color: navy; font-weight: bold;">protected void </span>init(<span style="color: olive;">@Observes </span>StartupEvent evt) <span style="color: navy; font-weight: bold;">throws </span>Exception {
<span style="color: #660e7a; font-weight: bold;">channel</span>.setReceiver(<span style="color: navy; font-weight: bold;">this</span>);
System.<span style="color: #660e7a; font-style: italic;">out</span>.printf(<span style="color: green; font-weight: bold;">"-- view: %s</span><span style="color: navy; font-weight: bold;">\n</span><span style="color: green; font-weight: bold;">"</span>, <span style="color: #660e7a; font-weight: bold;">channel</span>.getView());
}
<span style="color: navy; font-weight: bold;">protected void </span>destroy(<span style="color: olive;">@Observes </span>ShutdownEvent evt) {
Util.<span style="font-style: italic;">close</span>(<span style="color: #660e7a; font-weight: bold;">channel</span>);
<span style="color: #660e7a; font-weight: bold;">subscribers</span>.forEach(Subscriber::onComplete);
<span style="color: #660e7a; font-weight: bold;">subscribers</span>.clear();
}
<span style="color: olive;">@GET</span></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="color: olive;"> @Produces</span>(MediaType.<span style="color: #660e7a; font-style: italic;">TEXT_PLAIN</span>)
<span style="color: olive;">@Path</span>(<span style="color: green; font-weight: bold;">"/send/{msg}"</span>)
<span style="color: navy; font-weight: bold;">public </span>String sendMessage(<span style="color: olive;">@PathParam</span>(<span style="color: green; font-weight: bold;">"msg"</span>) String msg) <span style="color: navy; font-weight: bold;">throws </span>Exception {
<span style="color: #660e7a; font-weight: bold;">channel</span>.send(<span style="color: navy; font-weight: bold;">null</span>, Objects.<span style="font-style: italic;">requireNonNull</span>(msg).getBytes());
<span style="color: navy; font-weight: bold;">return </span>String.<span style="font-style: italic;">format</span>(<span style="color: green; font-weight: bold;">"message </span><span style="color: navy; font-weight: bold;">\"</span><span style="color: green; font-weight: bold;">%s</span><span style="color: navy; font-weight: bold;">\"</span><span style="color: green; font-weight: bold;"> was sent on channel </span><span style="color: navy; font-weight: bold;">\n</span><span style="color: green; font-weight: bold;">"</span>, msg);
}
<span style="color: olive;">@GET</span></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="color: olive;"> @Produces</span>(MediaType.<span style="color: #660e7a; font-style: italic;">SERVER_SENT_EVENTS</span>)
<span style="color: olive;">@Path</span>(<span style="color: green; font-weight: bold;">"/subscribe"</span>)
<span style="color: navy; font-weight: bold;">public </span>Publisher<String> greeting() {
<span style="color: navy; font-weight: bold;">return this</span>;
}
<span style="color: navy; font-weight: bold;">public void </span>receive(Message msg) {
onNext(msg);
}
<span style="color: navy; font-weight: bold;">public void </span>receive(MessageBatch batch) {
<span style="color: navy; font-weight: bold;">for</span>(Message msg: batch)
onNext(msg);
}
<span style="color: navy; font-weight: bold;">public void </span>viewAccepted(View view) {
System.<span style="color: #660e7a; font-style: italic;">out</span>.printf(<span style="color: green; font-weight: bold;">"-- new view: %s</span><span style="color: navy; font-weight: bold;">\n</span><span style="color: green; font-weight: bold;">"</span>, view);
}
<span style="color: navy; font-weight: bold;">public void </span>subscribe(Subscriber<? <span style="color: navy; font-weight: bold;">super </span>String> s) {
<span style="color: navy; font-weight: bold;">if</span>(s != <span style="color: navy; font-weight: bold;">null</span>)
<span style="color: #660e7a; font-weight: bold;">subscribers</span>.add(s);
}
<span style="color: navy; font-weight: bold;">protected void </span>onNext(Message msg) {
String s=<span style="color: navy; font-weight: bold;">new </span>String(msg.getRawBuffer(), msg.getOffset(), msg.getLength());
System.<span style="color: #660e7a; font-style: italic;">out</span>.printf(<span style="color: green; font-weight: bold;">"-- from %s: %s</span><span style="color: navy; font-weight: bold;">\n</span><span style="color: green; font-weight: bold;">"</span>, msg.src(), s);
<span style="color: #660e7a; font-weight: bold;">subscribers</span>.forEach(sub -> sub.onNext(<span style="color: #660e7a;">s</span>));
}
}</pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"> </pre>
It has a JChannel <span style="font-family: "courier new" , "courier" , monospace;">channel</span> which is injected by Arc (the dependency mechanism used in Quarkus). This channel is fully created and connected when it is injected.<br />
<br />
The <span style="font-family: "courier new" , "courier" , monospace;">receive(Message)</span> and <span style="font-family: "courier new" , "courier" , monospace;">receive(MessageBatch)</span> methods receive messages sent by itself or other members in the cluster. It in turn publishes them via the <span style="font-family: "courier new" , "courier" , monospace;">Publisher</span> interface. All subscribers will therefore receive all messages sent in the cluster.<br />
<br />
The <span style="font-family: "courier new" , "courier" , monospace;">sendMessage()</span> method is invoked when a URL of the form <span style="font-family: "courier new" , "courier" , monospace;">http://localhost:8080/chat/send/mymessage</span> is received. It takes the string parameter ("<span style="font-family: "courier new" , "courier" , monospace;">mymessage</span>") and uses the injected <span style="font-family: "courier new" , "courier" , monospace;">channel</span> to send it to all members of the cluster.<br />
<br />
The URL <span style="font-family: "courier new" , "courier" , monospace;">http://localhost:8080/chat/subscribe</span> (or <span style="font-family: "courier new" , "courier" , monospace;">http://localhost:8080/streaming.html </span>in a web browser) can be used to subscribe to messages being received by the channel.<br />
<br />
<h3>
Demo</h3>
Let's now run a cluster of 2 instances: open 2 shells and type the following commands (output has been edited for brevity):<br />
<br />
Shell1:<br />
<span style="font-family: "courier new" , "courier" , monospace;">[belasmac] /Users/bela/quarkus-jgroups-chat$ mvn compile quarkus:dev <br />...<br />[INFO] --- quarkus-maven-plugin:0.18.0:dev (default-cli) @ quarkus-jgroups-chat ---<br />2019-07-03 14:12:05,025 DEBUG [org.jgr.qua.ext.JChannelTemplate] (main) creating channel based on config config=chat-tcp.xml, bind_addr=, initial_hosts=, cluster=quarkus-jgroups-chat</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: "courier new" , "courier" , monospace;"><br />-------------------------------------------------------------------<br />GMS: address=belasmac-19612, cluster=quarkus-jgroups-chat, physical address=127.0.0.1:7800<br />-------------------------------------------------------------------<br />-- view: [belasmac-19612|0] (1) [belasmac-19612]</span><br />
<br />
Shell2:<br />
<span style="font-family: "courier new" , "courier" , monospace;">[belasmac] /Users/bela/quarkus-jgroups-chat$ mvn compile quarkus:dev -Dquarkus.http.port=8081<br />...<br />[INFO] --- quarkus-maven-plugin:0.18.0:dev (default-cli) @ quarkus-jgroups-chat ---<br />2019-07-03 14:15:02,463 DEBUG [org.jgr.qua.ext.JChannelTemplate] (main) creating channel based on config config=chat-tcp.xml, bind_addr=, initial_hosts=, cluster=quarkus-jgroups-chat<br /><br />-------------------------------------------------------------------<br />GMS: address=belasmac-25898, cluster=quarkus-jgroups-chat, physical address=127.0.0.1:7801<br />-------------------------------------------------------------------<br />-- view: [belasmac-19612|1] (2) [belasmac-19612, belasmac-25898]</span><br />
The system property <span style="font-family: "courier new" , "courier" , monospace;">quarkus.http.port=8081</span> is needed, or else there would be a port collision, as the default port of <span style="font-family: "courier new" , "courier" , monospace;">8080</span> has already been taken by the first application (both apps are started on the same host).<br />
<br />
The output shows that there's a cluster of 2 members.<br />
<br />
We can now post a message by invoking <span style="font-family: "courier new" , "courier" , monospace;">curl http://localhost:8080/chat/send/hello%20world</span> and <span style="font-family: "courier new" , "courier" , monospace;">curl http://localhost:8081/chat/send/message2.</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Both shells show that they received both messages:</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">-- view: [belasmac-19612|1] (2) [belasmac-19612, belasmac-25898]<br />-- from belasmac-19612: hello world<br />-- from belasmac-25898: message2</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Of course, we could also use a web browser to send the HTTP GET requests.</span></span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">When subscribing to the stream of messages in a web browser (<span style="font-family: "courier new" , "courier" , monospace;">http://localhost:8081/streaming.html</span>), this would look as follows:</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span> </span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-mxl2Q6_Abw8/XRygSMjF3HI/AAAAAAABPlo/njk_SHVjjK0N6APrvDQot5x2x5iiVh50QCLcBGAs/s1600/Screenshot%2B2019-07-03%2Bat%2B14.31.31.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="863" data-original-width="854" height="400" src="https://1.bp.blogspot.com/-mxl2Q6_Abw8/XRygSMjF3HI/AAAAAAABPlo/njk_SHVjjK0N6APrvDQot5x2x5iiVh50QCLcBGAs/s400/Screenshot%2B2019-07-03%2Bat%2B14.31.31.png" width="395" /></a></div>
<br />
<br />
Note that both channels bind to the loopback (<span style="font-family: "courier new" , "courier" , monospace;">127.0.0.1</span>) address. This can be changed by changing <span style="font-family: "courier new" , "courier" , monospace;">bind_addr </span>and <span style="font-family: "courier new" , "courier" , monospace;">initial_hosts</span> in <span style="font-family: "courier new" , "courier" , monospace;">application.properties</span>:<br />
<br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="color: navy; font-weight: bold;">quarkus.channel.config</span>=<span style="color: green; font-weight: bold;">chat-tcp.xml</span></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="color: green; font-weight: bold;">
</span><span style="color: navy; font-weight: bold;">quarkus.channel.cluster</span>=<span style="color: green; font-weight: bold;">quarkus-jgroups-chat</span></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="color: green; font-weight: bold;">
</span><span style="color: grey; font-style: italic;"># quarkus.channel.bind_addr=192.168.1.105</span></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="color: grey; font-style: italic;">
</span><span style="color: grey; font-style: italic;"># quarkus.channel.initial_hosts=192.168.1.105[7800]</span><span style="color: grey; font-style: italic;"></span><span style="color: navy; font-weight: bold;"></span></pre>
<br />
Uncomment the 2 properties and set them accordingly.<br />
<br />
Alternatively, we can pass these as system properties, e.g.:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">[belasmac] /Users/bela/quarkus-jgroups-chat$ mvn compile quarkus:dev -Dbind_addr=192.168.1.105 -Dinitial_hosts=192.168.1.105[7800],192.168.1.105[7801]<br />...<br />[INFO] --- quarkus-maven-plugin:0.18.0:dev (default-cli) @ quarkus-jgroups-chat ---<br />2019-07-03 14:38:28,258 DEBUG [org.jgr.qua.ext.JChannelTemplate] (main) creating channel based on config config=chat-tcp.xml, bind_addr=, initial_hosts=, cluster=quarkus-jgroups-chat<br /><br />-------------------------------------------------------------------<br />GMS: address=belasmac-10738, cluster=quarkus-jgroups-chat, physical address=192.168.1.105:7800<br />-------------------------------------------------------------------<br />-- view: [belasmac-10738|0] (1) [belasmac-10738]</span><br />
<br />
<h3>
Native compilation</h3>
To compile the application to native code, the <span style="font-family: "courier new" , "courier" , monospace;">mvn package -Pnative</span> command has to be executed:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">[belasmac] /Users/bela/quarkus-jgroups-chat$ mvn package -Pnative<br />[INFO] Building jar: /Users/bela/quarkus-jgroups-chat/target/quarkus-jgroups-chat-1.0.0-SNAPSHOT.jar<br />[INFO] <br />[INFO] --- quarkus-maven-plugin:0.18.0:build (default) @ quarkus-jgroups-chat ---<br />[INFO] [io.quarkus.deployment.QuarkusAugmentor] Beginning quarkus augmentation<br />[INFO] [org.jboss.threads] JBoss Threads version 3.0.0.Beta4<br />[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 1343ms<br />[INFO] [io.quarkus.creator.phase.runnerjar.RunnerJarPhase] Building jar: /Users/bela/quarkus-jgroups-chat/target/quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner.jar<br />[INFO] <br />[INFO] --- quarkus-maven-plugin:0.18.0:native-image (default) @ quarkus-jgroups-chat ---<br />[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] Running Quarkus native-image plugin on OpenJDK 64-Bit Server VM<br />[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] /Users/bela/graalvm/Contents/Home/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportUnsupportedElementsAtRuntime -H:+ReportExceptionStackTraces -H:+PrintAnalysisCallTree -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-SpawnIsolates -H:+JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] classlist: 6,857.25 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] (cap): 4,290.72 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] setup: 6,430.30 ms<br />14:43:05,540 INFO [org.jbo.threads] JBoss Threads version 3.0.0.Beta4<br />14:43:06,468 INFO [org.xnio] XNIO version 3.7.2.Final<br />14:43:06,528 INFO [org.xni.nio] XNIO NIO Implementation Version 3.7.2.Final<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] (typeflow): 17,331.26 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] (objects): 24,511.12 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] (features): 1,194.16 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] analysis: 44,204.65 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] (clinit): 579.00 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] universe: 1,715.40 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] (parse): 3,315.80 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] (inline): 4,563.11 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] (compile): 24,906.58 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] compile: 34,907.28 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] image: 4,557.78 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] write: 2,531.16 ms<br />[quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner:93574] [total]: 109,858.54 ms<br />[INFO] ------------------------------------------------------------------------<br />[INFO] BUILD SUCCESS<br />[INFO] ------------------------------------------------------------------------<br />[INFO] Total time: 01:58 min<br />[INFO] Finished at: 2019-07-03T14:44:40+02:00<br /> </span><br />
<br />
This uses GraalVM's <span style="font-family: "courier new" , "courier" , monospace;">native-image</span> to generate a native executable. After a while, the resulting executable is in the <span style="font-family: "courier new" , "courier" , monospace;">./target </span>directory:<br />
<br />
It's size is only 27MB and we can see that it is a MacOS native executable:<br />
<span style="font-family: "courier new" , "courier" , monospace;">[belasmac] /Users/bela/quarkus-jgroups-chat/target$ ls -lh quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner<br />-rwxr-xr-x 1 bela staff 27M Jul 3 14:44 quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner<br />[belasmac] /Users/bela/quarkus-jgroups-chat/target$ file quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner<br />quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner: Mach-O 64-bit executable x86_64 </span><br />
<br />
To run it:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">[belasmac] /Users/bela/quarkus-jgroups-chat/target$ ./quarkus-jgroups-chat-1.0.0-SNAPSHOT-runner <br /><br />-------------------------------------------------------------------<br />GMS: address=belasmac-55106, cluster=quarkus-jgroups-chat, physical address=127.0.0.1:7800<br />-------------------------------------------------------------------<br />-- view: [belasmac-55106|0] (1) [belasmac-55106]<br /> </span><br />
<br />
When you run this yourself, you will notice that quick startup time of the second and subsequent members. Why not the first member? The first member has to wait for <span style="font-family: "courier new" , "courier" , monospace;">GMS.join_timeout</span> millis (defined in <span style="font-family: "courier new" , "courier" , monospace;">chat-tcp.xml</span>) to see if it discovers any other members, and so it always runs into this timeout.<br />
<br />
To change <span style="font-family: "courier new" , "courier" , monospace;">bind_addr</span> and <span style="font-family: "courier new" , "courier" , monospace;">initial_hosts</span>, <span style="font-family: "courier new" , "courier" , monospace;">application.properties</span> has to be changed <b>before</b> compiling to native code.<br />
<br />
<br />
<br />
<h3>
Caveats </h3>
The quarkus-jgroups extension depends on JGroups-4.1.2-SNAPSHOT, which it may not find unless a snapshot repository has been added to the POM (or <span style="font-family: "courier new" , "courier" , monospace;">settings.xml</span>). Alternatively, <span style="font-family: "courier new" , "courier" , monospace;">git clone https://github.com/belaban/JGroups.git ; cd JGroups ; mvn install</span> will generate and install this artifact in your local maven repo.<br />
<br />
Currently, only TCP is supported (<span style="font-family: "courier new" , "courier" , monospace;">chat-tcp.xml</span>). UDP will be supported once MulticastSockets are properly supported by GraalVM (see [5] for details).<br />
<br />
For some obscure reason, <br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="background-color: #efefef;"><</span><span style="background-color: #efefef; color: navy; font-weight: bold;">enableJni</span><span style="background-color: #efefef;">></span>true<span style="background-color: #efefef;"></</span><span style="background-color: #efefef; color: navy; font-weight: bold;">enableJni</span><span style="background-color: #efefef;">></span></pre>
<br />
had to be enabled in the POM, or else native compilation would fail. I hope to remove this once I understand why...<br />
<br />
<h3>
Outlook</h3>
This was a relatively quick port of JGroups to native code. For feedback and questions please use the JGroups mailing list.<br />
<br />
The following things are on my todo list for further development:<br />
<ul>
<li>Provide more JGroups classes via extensions, e.g. RpcDispatcher (to make remote method calls)</li>
<li>Provide docker images with native executables</li>
<li>Implement support for UDP</li>
<li>Trim down the size of the executable even more</li>
</ul>
<br />
Enjoy!<br />
<br />
<br />
<br />
[1] <a href="https://github.com/jgroups-extras/quarkus-jgroups">https://github.com/jgroups-extras/quarkus-jgroups</a><br />
[2] <a href="https://github.com/jgroups-extras/quarkus-jgroups-chat">https://github.com/jgroups-extras/quarkus-jgroups-chat</a><br />
[3] <a href="https://quarkus.io/">https://quarkus.io</a><br />
[4] <a href="https://www.graalvm.org/">https://www.graalvm.org</a><br />
[5] <a href="https://github.com/belaban/JGroups/blob/master/doc/design/PortingToGraalVM.txt">https://github.com/belaban/JGroups/blob/master/doc/design/PortingToGraalVM.txt</a><br />
[6] <a href="https://github.com/belaban/JGroups/blob/master/tests/perf/org/jgroups/tests/perf/ProgrammaticUPerf2.java">https://github.com/belaban/JGroups/blob/master/tests/perf/org/jgroups/tests/perf/ProgrammaticUPerf2.java</a> <br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-91026841455831870732019-06-05T13:12:00.000+02:002019-06-05T13:17:25.355+02:00Network sniffingOftentimes, <a href="http://jgroups.org/">JGroups</a>/<a href="https://www.redhat.com/en/technologies/jboss-middleware/data-grid">Datagrid</a> users capture traffic from the network for analysis. Using tools such as <a href="https://www.wireshark.org/">wireshark</a> or <a href="https://www.wireshark.org/docs/man-pages/tshark.html">tshark</a>, they can look at UDP packets or TCP streams.<br />
<br />
There used to be a <a href="https://developer.jboss.org/wiki/JGroupsWiresharkPlugin">wireshark plugin</a>, written by Richard Achmatowicz, but since it was written in C, every time the wire format changed, the C code had to be changed, too. It is therefore not maintained any longer.<br />
<br />
However, there's a class in JGroups that can be used to read messages from the wire: <a href="https://github.com/belaban/JGroups/blob/master/tests/other/org/jgroups/tests/ParseMessages.java">ParseMessages</a>. Since it uses the the same code that's reading messages off the wire, it can always parse messages in the version it's shipped with. It is therefore resistant to wire format changes.<br />
<br />
In 4.1.0, I changed ParseMessages to be more useful:<br />
<ul>
<li>Reading of TCP streams is now supported</li>
<li>It can read packets from stdin (ideal for piping from tshark)</li>
<li>Handling of binary data (e.g. from a PCAP capture) is supported</li>
<li>Views are parsed and displayed (e.g. in VIEW or JOIN response messages)</li>
<li>Logical names can be displayed: instead of <span style="font-family: "courier new" , "courier" , monospace;">{node-2543, node-2543}</span> instead of <span style="font-family: "courier new" , "courier" , monospace;">{3673e687-fafb-63e0-2ff1-67c0a8a6f8eb,312aa7da-f3d5-5999-1f5c-227f6e43728e}</span></li>
</ul>
To demonstrate how to use this, I made 4 short videos:<br />
<ol>
<li><a href="https://www.youtube.com/watch?v=vSD2QZNSUuE">Capture UDP IPv4 traffic with tshark</a></li>
<li><a href="https://www.youtube.com/watch?v=zEyTDEetj48">Capture TCP IPv6 traffic with tshark</a></li>
<li><a href="https://www.youtube.com/watch?v=mlhj9EPJ31c">Capture with tcpdump and wireshark</a></li>
</ol>
The documentation is <a href="http://www.jgroups.org/manual4/index.html#_analyzing_wire_format_packets">here</a>.<br />
<br />
Happy network sniffing!<br />
<br />
<br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-65567510159451762032019-05-21T17:25:00.002+02:002019-05-21T17:25:36.310+02:004.1.0 releasedI'm happy to announce that I just released JGroups 4.1.0!<br />
<br />
Why the bump to a new minor version?<br />
<br />
I had to make some API changes and - as I'm trying to avoid that in micro releases - I decided to bump the version to 4.1.0. The changes involve removing a few (rather exotic) JChannel constructors, but chances are you've never used any of them anyway. The other change is a signature change in Streamable, where I now throw IOExceptions and ClassNotFoundExceptions instead of simple Exceptions.<br />
<br />
Here's a list of the major changes:<br />
<br />
<h3>
<a href="http://graalvm.org/">GraalVM</a> / <a href="http://www.quarkus.io/">Quarkus</a> support</h3>
<ul>
<li>JGroups can now be compiled into a native executable using GraalVM's <span style="font-family: "Courier New", Courier, monospace;">native-image</span>. This is a very cool feature; I've used <a href="https://github.com/belaban/JGroups/blob/master/tests/perf/org/jgroups/tests/perf/ProgrammaticUPerf2.java">ProgrammaticUPerf2</a> to start a member in ~<b>1 millisecond</b>!</li>
<li>This means JGroups can now be used by other applications to create native binaries. Not yet very polished, and I'll write a Quarkus extension next, but usable by folks who know GraalVM...</li>
<li>I'll blog about the port to GraalVM and the Quarkus extension (once it's ready) next, so stay tuned!</li>
<li><a href="https://issues.jboss.org/browse/JGRP-2332">https://issues.jboss.org/browse/JGRP-2332</a></li>
<li><a href="https://issues.jboss.org/browse/JGRP-2341">https://issues.jboss.org/browse/JGRP-2341</a></li>
</ul>
<h3>
Parsing of network packets</h3>
<ul>
<li>When using network sniffers, such as wireshark, tshark or tcpdump, the
resulting PCAP files can be parsed by JGroups, displaying the contents
of network packets as JGroups messages</li>
<li>This can be done <b>offline</b> (e.g. reading a previously recorded PCAP file) or <b>online</b> (e.g. by piping the output of tshark into JGroups).</li>
<li>I'll write a separate blog post about this shortly</li>
<li><a href="https://issues.jboss.org/browse/JGRP-2311">https://issues.jboss.org/browse/JGRP-2311</a></li>
<li>Documentation: <a href="http://www.jgroups.org/manual4/index.html#_analyzing_wire_format_packets">http://www.jgroups.org/manual4/index.html#_analyzing_wire_format_packets</a> </li>
</ul>
<h3>
Diagnostics handler without reflection</h3>
<ul>
<li>This is related to the GraalVM port: the default DiagnosticsHandler (used by probe.sh) uses reflection, which is not allowed in GraalVM</li>
<li>This additional DiagnosticsHandler can be used instead of the default one when creating a native binary of an application. The advantage is that probe.sh still works, even in a native binary.</li>
<li><a href="https://issues.jboss.org/browse/JGRP-2337">https://issues.jboss.org/browse/JGRP-2337</a></li>
</ul>
<h3>
RpcDispatcher without reflection</h3>
<ul>
<li>This is similar to the above bullet item: since reflection is not allowed in the GraalVM at runtime, reflective calls by RpcDispatcher would have to be defined in a config file passed to <span style="font-family: "Courier New", Courier, monospace;">native-image</span>, when creating a native binary. </li>
<li>I therefore added a way to invoke methods, without the need to use reflection</li>
<li>For details see <a href="https://issues.jboss.org/browse/JGRP-2338">https://issues.jboss.org/browse/JGRP-2338</a> or the documentation at <a href="http://www.jgroups.org/manual4/index.html#_methodlookup_and_methoddispatcher">http://www.jgroups.org/manual4/index.html#_methodlookup_and_methoddispatcher</a></li>
</ul>
<h3>
Probe: support when running under TCP</h3>
<ul>
<li>Also required when running on GraalVM: since JGroups currently only supports TCP (MulticastSockets don't currently work on GraalVM), probe.sh needs to be given the address of *one* member, to fetch information about all members</li>
<li><a href="https://issues.jboss.org/browse/JGRP-2336">https://issues.jboss.org/browse/JGRP-2336</a></li>
</ul>
<h3>
Change in how IPv4/IPv6 addresses are picked</h3>
<ul>
<li>The new algorithm centers around <span style="font-family: "Courier New", Courier, monospace;">bind_addr</span> defined in the transport (UDP, TCP, TCP_NIO2); the value of this address determines how other addresses (such as loopback, site_local, global, localhost or default values) are resolved.</li>
<li>Example: if <span style="font-family: "Courier New", Courier, monospace;">bind_addr=::1</span>, then all other addresses that are not explicitly defined will be IPv6. If <span style="font-family: "Courier New", Courier, monospace;">bind_addr=127.0.0.1</span>, the all other addresses will be IPv4 addresses.</li>
<li>The advantage of this is that you can run a JGroups stack using IPv4 and another one using IPv6 <i>in the same process</i>!</li>
<li><a href="https://issues.jboss.org/browse/JGRP-2343">https://issues.jboss.org/browse/JGRP-2343</a></li>
</ul>
A complete list of JIRA issues is at <a href="https://issues.jboss.org/projects/JGRP/versions/12341388">https://issues.jboss.org/projects/JGRP/versions/12341388</a>. <br />
Please post questions/issues to the mailing list at <a href="https://groups.google.com/forum/#!forum/jgroups-dev">https://groups.google.com/forum/#!forum/jgroups-dev</a>.<br />
<br />
Enjoy!<br />
Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-19360540490522073742019-03-14T14:19:00.002+01:002019-03-14T14:20:15.222+01:004.0.19FYI: I just released 4.0.19.Final. <br />
<br />
This changes the way ASYM_ENCRYPT disseminates the private shared group key to members, going from a pull- to a push-based approach [1] [2].<br />
<br />
In combination with JGRP-2293 [3], this should help a lot in environments like Kubernetes or Openshift where pods with JGroups nodes are started/stopped dynamically, and where encryption is required.<br />
<br />
The design is at [3]. <br />
<br />
Check it out and let me know if you run into issues.<br />
Cheers,<br />
<br />
[1] <a href="https://issues.jboss.org/browse/JGRP-2297">https://issues.jboss.org/browse/JGRP-2297</a><br />
[2] <a href="https://issues.jboss.org/browse/JGRP-2293">https://issues.jboss.org/browse/JGRP-2293</a><br />
[3] <a href="https://github.com/belaban/JGroups/blob/master/doc/design/ASYM_ENCRYPT.txt">https://github.com/belaban/JGroups/blob/master/doc/design/ASYM_ENCRYPT.txt</a>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-91598255520374665562019-02-13T12:13:00.002+01:002019-02-13T12:14:17.607+01:00and 4.0.17Sorry for the short release cycle between 4.0.16 and 4.0.17, but there was an important feature in 4.0.17, which I wanted to release as soon as possible: <a href="https://issues.jboss.org/browse/JGRP-2293">JGRP-2293</a>.<br />
<br />
This has better support for concurrent graceful leaves of multiple members (also coordinators), which is important in cloud environments, where pods are started and stopped dynamically by Kubernetes.<br />
<br />
Not having this enhancement would lead to members leaving gracefully, but that would sometimes not be recognized and failure detection would have to kick in to exclude those members and installing the correct views. This would slow down things, and the installing of new views would be goverened by the timeouts in the failure detection protocols (FD_ALL, FD_SOCK). On top of this, in some edge cases, MERGE3 would have to kick in and fix views, further slowing things down.<br />
<br />
There's a unit test [1] which tests the various aspects of concurrent leaving, e.g. all coordinators leaving sequentially, multiple members leaving concurrently etc <br />
<br />
I recommend installing this version as soon as possible, especially if you run in the cloud. Questions / problems etc -> [2]<br />
<br />
Cheers,<br />
<br />
[1] https://github.com/belaban/JGroups/blob/master/tests/junit-functional/org/jgroups/tests/LeaveTest.java<br />
<br />
[2] https://groups.google.com/forum/#!forum/jgroups-devBela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-65491765033643280762019-01-21T09:31:00.000+01:002019-01-21T09:31:34.403+01:00JGroups 4.0.16I just released 4.0.16. The most important features/bug fixes are (for a comprehensive list see [1]):<br />
<br />
<ul>
<li>Better mechanism to detect whether IPv4 or IPv6 addresses need to be used. This eliminates the need to use <span style="font-family: "Courier New", Courier, monospace;">java.net.preferIPv4Stack</span> (unless forcing the use of IPv4 addresses)</li>
<ul>
<li><a href="https://issues.jboss.org/browse/JGRP-2305">https://issues.jboss.org/browse/JGRP-2305</a> </li>
</ul>
<li>Due to a regression, MULTI_PING didn't work any longer. This was fixed in <a href="https://issues.jboss.org/browse/JGRP-2319">https://issues.jboss.org/browse/JGRP-2319</a></li>
<li>Bug fixes in DNS_PING (used for running in Kubernetes/Openshift clouds): </li>
<ul>
<li><a href="https://issues.jboss.org/browse/JGRP-2296">https://issues.jboss.org/browse/JGRP-2296</a></li>
<li><a href="https://issues.jboss.org/browse/JGRP-2300">https://issues.jboss.org/browse/JGRP-2300</a></li>
<li><a href="https://issues.jboss.org/browse/JGRP-2314">https://issues.jboss.org/browse/JGRP-2314</a></li>
<li><a href="https://issues.jboss.org/browse/JGRP-2316">https://issues.jboss.org/browse/JGRP-2316</a></li>
<li><a href="https://issues.jboss.org/browse/JGRP-2309">https://issues.jboss.org/browse/JGRP-2309</a></li>
</ul>
<li>ASYM_ENCRYPT</li>
<ul>
<li>Pooled ciphers were not correctly re-initialized: <a href="https://issues.jboss.org/browse/JGRP-2279">https://issues.jboss.org/browse/JGRP-2279</a></li>
<li>Concurrent access to pooled ciphers can lead to decryption errors: <a href="https://issues.jboss.org/browse/JGRP-2315">https://issues.jboss.org/browse/JGRP-2315</a> </li>
</ul>
<li>FILE_PING (and all subclasses, e.g. S3_PING) had an issue with merging: <a href="https://issues.jboss.org/browse/JGRP-2288">https://issues.jboss.org/browse/JGRP-2288</a></li>
<li>slf4j doesn't work: <a href="https://issues.jboss.org/browse/JGRP-2307">https://issues.jboss.org/browse/JGRP-2307</a></li>
<li>LazyThreadFactory leaks threads: <a href="https://issues.jboss.org/browse/JGRP-2312">https://issues.jboss.org/browse/JGRP-2312</a></li>
<li>TransferQueueBundler (default): view changes lead to dropping of messages to non-members; leading to issues such as delaying the leaving of a member: <a href="https://issues.jboss.org/browse/JGRP-2324">https://issues.jboss.org/browse/JGRP-2324</a></li>
</ul>
Enjoy! <br />
<br />
[1] <a href="https://issues.jboss.org/projects/JGRP/versions/12339241">https://issues.jboss.org/projects/JGRP/versions/12339241</a>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-34773273668439909782018-01-03T16:51:00.001+01:002018-01-03T16:51:08.196+01:00Injecting a split brain into a JGroups clusterThanks to a contribution by <a href="https://github.com/valdar">Andrea Tarocchi</a>, we can now inject arbitrary views into a JGroups cluster, generating split brain scenarios.<br />
<br />
This is done by a new protocol <a href="https://github.com/belaban/JGroups/blob/master/src/org/jgroups/protocols/INJECT_VIEW.java">INJECT_VIEW</a>, which can be placed anywhere in the stack, e.g. like this:<br />
<br />
...<br />
<span style="font-family: "Courier New", Courier, monospace;"> <MERGE3 max_interval="30000"<br /> min_interval="10000"/><br /> <FD_SOCK/></span><br />
<span style="font-family: "Courier New", Courier, monospace;"> <FD_SOCK/><br /> <FD_ALL/><br /> <VERIFY_SUSPECT timeout="500" /><br /> <b><INJECT_VIEW/></b><br /> <BARRIER /></span><br />
<span style="font-family: "Courier New", Courier, monospace;"> ... </span><br />
<br />
The protocol exposes a method injectView(), which can be called by probe.sh. Let's say we have a cluster {A,B,C}: we can now inject view {A} into A, and {B,C} into B and C, as follows:<br />
<br />
<span style="font-family: "Courier New", Courier, monospace;">probe.sh op=INJECT_VIEW.injectView["A=A;B=B/C;C=B/C"]</span><br />
<br />
(Note that '/' is used to separate node names; this is needed, because ',' is used to seperate arguments by default in probe) .<br />
<br />
The result is that node A is now its own singleton cluster (not seeing B and C), and B and C only see each other (but not A).<br />
<br />
Note that the member names are the<i> logical names</i> given to individual nodes.<br />
<br />
Once the split has been injected, MERGE3 should remedy it by merging A, B and C back into a single cluster, after between 10-30 seconds (in the above configuration).<br />
<br />
Thanks to Andrea for creating this nifty tool to experiment with cluster splits!Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-79701655840344831192017-08-28T12:33:00.000+02:002017-08-28T12:33:17.230+02:00JGroups workshops in Rome and Berlin in NovemberI'm happy to announce that I've revamped the JGroups workshop and I'll be teaching 2 workshops in November: Rome Nov 7-10 and Berlin Nov 21-24.<br />
<br />
The new workshop is 4 days (TUE-FRI). I've updated the workshop to use the latest version of JGroups (4.0.x), removed a few sections and added sections on troubleshooting/diagnosis and handling of network partitions (split brain).<br />
<br />
Should be a fun and hands-on workshop!<br />
<br />
To register and for the table of contents, navigate to [1].<br />
<br />
To get the early-bird discount, use code EARLYBIRD. <br />
<br />
Cheers,<br />
<br />
<br />
[1] <a href="http://www.jgroups.org/workshops.html">http://www.jgroups.org/workshops.html</a>Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-56351586088706594482017-06-26T11:02:00.002+02:002017-06-26T11:02:43.794+02:00Non-blocking JGroupsI'm happy to announce that I just released JGroups 4.0.4.Final!<br />
<br />
It's prominent feature is <b>non-blocking flow control</b> [1,2].<br />
<br />
Flow control makes sure that a fast sender cannot overwhelm a slow receiver in the long run (short spikes are tolerated) by adjusting the send rate to the receive rate. This is done by giving each sender credits (number of bytes) that it is allowed to send until it has to block.<br />
<br />
A receiver sends new credits to the sender once it has processed a message.<br />
<br />
When no credits are available anymore, the sender blocks until it gets new credits from the receiver(s). This is done in UFC (unicast flow control) and MFC (multicast flow control).<br />
<br />
Flow control and TCP have been the only protocols that could block (UDP and TCP_NIO2 are non-blocking).<br />
<br />
Non-blocking flow control now adds protocols UFC_NB and MFC_NB. These can replace their blocking counterparts.<br />
<br />
Instead of blocking sender threads, the <b>non-blocking flow control protocols queue messages when not enough credits are available to send them</b>, allowing the sender threads to return immediately.<br />
<br />
When fresh credits are received, the queued messages will be sent.<br />
<br />
The queues are bound to prevent heap exhaustion; setting attribute max_queue_size (bytes) will queue messages up to max_queue_size bytes and then block subsequent attempts to queue until more space is available. Of course, setting max_queue_size to a large value will effectively make queues unbounded.<br />
<br />
Using MFC_NB / UFC_NB with a transport of UDP or TCP_NIO2, which also never block, provides a completely non-blocking stack, where sends never block a JChannel.send(Message). If RpcDispatcher is used, there are non-blocking methods to invoke (even synchronous) RPCs, ie. the ones which return a CompletableFuture.<br />
<br />
<br />
Other features of 4.0.4 include a new message bundler and a few bug fixes (e.g. the internal thread pool was not shut down). For the list of all JIRAs see [3].<br />
<br />
Cheers,<br />
<br />
<br />
[1] <a href="https://issues.jboss.org/browse/JGRP-2172">https://issues.jboss.org/browse/JGRP-2172</a><br />
<br />
[2] <a href="http://www.jgroups.org/manual4/index.html#NonBlockingFlowControl">http://www.jgroups.org/manual4/index.html#NonBlockingFlowControl</a><br />
<br />
[3] <a href="https://issues.jboss.org/projects/JGRP/versions/12334674">https://issues.jboss.org/projects/JGRP/versions/12334674</a><br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-37407631347780280532017-05-08T17:11:00.002+02:002017-05-08T17:11:37.845+02:00Running an Infinispan cluster with Kubernetes on Google Container Engine (GKE)In this post, I'm going to show the steps needed to get a 10 node <a href="http://www.infinispan.org/">Infinispan</a> cluster up and running on <a href="https://cloud.google.com/container-engine/">Google Container Engine (GKE)</a>.<br />
<br />
The test we'll be running is <a href="https://github.com/belaban/IspnPerfTest">IspnPerfTest</a> and the corresponding docker image is <a href="https://hub.docker.com/r/belaban/ispn_perf_test/">belaban/ispn_perf_test</a> on dockerhub.<br />
<br />
<b>All that's needed to run this youselves is a <a href="https://cloud.google.com/compute/">Google Compute Engine</a> account</b>, so head on over there now and create one if you want to reproduce this demo! :-)<br />
<br />
<b>Alternatively, the cluster could be run locally in <a href="https://github.com/kubernetes/minikube">minikube</a></b>, but for this post I chose GKE instead.<br />
<br />
Ready? Then let's get cracking...<br />
<br />
First, let's create a 10-node cluster in GKE. The screen shot below shows the form that needs to be filled out to create a 10 node cluster in GKE. This results in 10 nodes getting created in Google Compute Engine (GCE):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-k3A-Ef0tCUY/WRBtYxPQTJI/AAAAAAAApZo/_jDrgDrhRQktSMEQxvcKhaWnU_t0iasugCLcB/s1600/Screen%2BShot%2B2017-05-08%2Bat%2B15.06.08.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://3.bp.blogspot.com/-k3A-Ef0tCUY/WRBtYxPQTJI/AAAAAAAApZo/_jDrgDrhRQktSMEQxvcKhaWnU_t0iasugCLcB/s1600/Screen%2BShot%2B2017-05-08%2Bat%2B15.06.08.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
As shown, we'll use 10 v16-cpu instances with 14GB of memory each.<br />
<br />
Press "Create" and the cluster is being created:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-AOaLH-TiZAo/WRBuF-Sx5GI/AAAAAAAApZw/avGlE2MnLzoekuLwyVPQEKOmChmECss0QCLcB/s1600/Screen%2BShot%2B2017-05-08%2Bat%2B15.09.01.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-AOaLH-TiZAo/WRBuF-Sx5GI/AAAAAAAApZw/avGlE2MnLzoekuLwyVPQEKOmChmECss0QCLcB/s1600/Screen%2BShot%2B2017-05-08%2Bat%2B15.09.01.png" /></a></div>
<br />
If you logged into your Google Compute Engine console, it would show the 10 nodes that are getting created.<br />
<br />
When the cluster has been created, click on "Connect" and execute the "gcloud" command that's shown as a result:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"><span class="p6n-snippet-overflow-hidden p6n-snippet-shell-container p6n-snippet-container p6n-snippet-focused" tabindex="0" type="shell"><span>gcloud container clusters get-credentials ispn-cluster \
--zone us-central1-a --project ispnperftest</span></span></span><br />
<br />
We can now see the 10 GCE nodes:<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">[belasmac] /Users/bela/kubetest$ kubectl get nodes<br />NAME STATUS AGE VERSION<br />gke-ispn-cluster-default-pool-59ed0e14-1zdb Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-33pk Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-3t95 Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-5sn9 Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-9lmz Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-j646 Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-k797 Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-q80q Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-r96s Ready 4m v1.5.7<br />gke-ispn-cluster-default-pool-59ed0e14-zhdj Ready 4m v1.5.7</span></span><br />
<br />
Next we'll run an interactive instance of IspnPerfTest and 3 non-interactive (no need for a TTY) instances, forming a cluster of 4. First, we start the interactive instance. Note that it might take a while until Kubernetes has downloaded image <span style="font-family: "Courier New",Courier,monospace;">belaban/ispn_perf_test</span> from dockerhub. When done, the following is shown:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">[belasmac] /Users/bela/kubetest$ kubectl run infinispan --rm=true -it --image=belaban/ispn_perf_test kube.sh<br />If you don't see a command prompt, try pressing enter.<br /><br />-------------------------------------------------------------------<br />GMS: address=infinispan-749052960-pl066-27417, cluster=default, physical address=10.40.0.4:7800<br />-------------------------------------------------------------------<br /><br />-------------------------------------------------------------------<br />GMS: address=infinispan-749052960-pl066-18029, cluster=cfg, physical address=10.40.0.4:7900<br />-------------------------------------------------------------------<br />created 100,000 keys: [1-100,000]<br />[1] Start test [2] View [3] Cache size [4] Threads (100) <br />[5] Keys (100,000) [6] Time (secs) (60) [7] Value size (1.00KB) [8] Validate<br />[p] Populate cache [c] Clear cache [v] Versions<br />[r] Read percentage (1.00) <br />[d] Details (true) [i] Invokers (false) [l] dump local cache<br />[q] Quit [X] Quit all</span></span><br />
<br />
Now we'll start 3 more instances, the definition of which is taken from a Yaml file (ispn.yaml):<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">## Creates a number of pods on kubernetes running IspnPerfTest<br />## Run with: kubectl create -f ispn.yaml<br />apiVersion: extensions/v1beta1<br />kind: Deployment<br />metadata:<br /> name: ispn<br /> namespace: default<br />spec:<br /> replicas: 3<br /> template:<br /> metadata:<br /> labels:<br /> run: ispn-perf-test<br /> spec:<br /> hostNetwork: false<br /> containers:<br /> - args:<br /> - kube.sh<br /> - -nohup<br /> name: ispn-perf-test<br /> image: belaban/ispn_perf_test<br /> # imagePullPolicy: IfNotPresent</span></span><br />
<br />
The YAML file creates 3 instances (<span style="font-family: "Courier New",Courier,monospace;">replicas: 3</span>):<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">belasmac] /Users/bela/kubetest$ kubectl create -f ispn.yaml<br />deployment "ispn" created</span></span><br />
<br />
Running "kubectl get pods -o wide" shows:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">[belasmac] /Users/bela/kubetest$ kubectl get pods -o wide<br />NAME READY STATUS RESTARTS AGE IP NODE<br />infinispan-749052960-pl066 1/1 Running 0 4m 10.40.0.4 gke-ispn-cluster-default-pool-59ed0e14-5sn9<br />ispn-1255975377-hm785 1/1 Running 0 46s 10.40.2.4 gke-ispn-cluster-default-pool-59ed0e14-r96s<br />ispn-1255975377-jx70d 1/1 Running 0 46s 10.40.4.3 gke-ispn-cluster-default-pool-59ed0e14-1zdb<br />ispn-1255975377-xf9r8 1/1 Running 0 46s 10.40.5.3 gke-ispn-cluster-default-pool-59ed0e14-3t95</span></span><br />
<br />
This shows 1 <span style="font-family: "Courier New",Courier,monospace;">infinispan</span> instance (interactive terminal) and 3 <span style="font-family: "Courier New",Courier,monospace;">ispn</span> instances (non-interactive).<br />
<br />
We can now exec into one of the instances are run probe.sh to verify that a cluster of 4 has formed:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;">[belasmac] /Users/bela/kubetest$ kubectl exec -it ispn-1255975377-hm785 bash<br />bash-4.3$ probe.sh -addr localhost <br />-- sending probe request to /10.40.5.3:7500<br />-- sending probe request to /10.40.4.3:7500<br />-- sending probe request to /10.40.2.4:7500<br />-- sending probe request to /10.40.0.4:7500<br /><br />#1 (287 bytes):<br />local_addr=ispn-1255975377-xf9r8-60537<br />physical_addr=10.40.5.3:7800<br />view=[infinispan-749052960-pl066-27417|3] (4) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319]<br />cluster=default<br />version=4.0.3-SNAPSHOT (Schiener Berg)<br /><br />#2 (287 bytes):<br />local_addr=ispn-1255975377-hm785-39319<br />physical_addr=10.40.2.4:7800<br />view=[infinispan-749052960-pl066-27417|3] (4) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319]<br />cluster=default<br />version=4.0.3-SNAPSHOT (Schiener Berg)<br /><br />#3 (284 bytes):<br />local_addr=ispn-1255975377-jx70d-16<br />physical_addr=10.40.4.3:7800<br />view=[infinispan-749052960-pl066-27417|3] (4) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319]<br />cluster=default<br />version=4.0.3-SNAPSHOT (Schiener Berg)<br /><br />#4 (292 bytes):<br />local_addr=infinispan-749052960-pl066-27417<br />physical_addr=10.40.0.4:7800<br />view=[infinispan-749052960-pl066-27417|3] (4) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319]<br />cluster=default<br />version=4.0.3-SNAPSHOT (Schiener Berg)<br /><br />4 responses (4 matches, 0 non matches)</span></span><br />
<br />
The 'view' item shows the same view (list of cluster members) across all 4 nodes, so the cluster has formed successfully. Also, if we look at our interactive instance, pressing [2] shows the cluster has 4 members:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">-- local: infinispan-749052960-pl066-18029<br />-- view: [infinispan-749052960-pl066-18029|3] (4) [infinispan-749052960-pl066-18029, ispn-1255975377-xf9r8-34843, ispn-1255975377-jx70d-20952, ispn-1255975377-hm785-48520]</span></span><br />
<br />
This means we have view number 4 (0-3) and 4 cluster members (the number in parens).<br />
<br />
Next, let's scale the cluster to 10 members. To do this, we'll tell Kubernetes to scale the <span style="font-family: "Courier New",Courier,monospace;">ispn</span> deployment to 9 instances (from 3):<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">[belasmac] /Users/bela/IspnPerfTest$ kubectl scale deployment ispn --replicas=9<br />deployment "ispn" scaled</span></span><br />
<br />
After a few seconds, the interactive terminal shows a new view containing 10 members:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">** view: [infinispan-749052960-pl066-27417|9] (10) [infinispan-749052960-pl066-27417, ispn-1255975377-xf9r8-60537, ispn-1255975377-jx70d-16, ispn-1255975377-hm785-39319, ispn-1255975377-6191p-9724, ispn-1255975377-1g2kx-5547, ispn-1255975377-333rl-13052, ispn-1255975377-57zgl-28575, ispn-1255975377-j8ckh-35528, ispn-1255975377-lgvmt-32173]</span></span><br />
<br />
We also see a minor inconvenience when looking at the pods:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">[belasmac] /Users/bela/jgroups-docker$ lubectl get pods -o wide<br />NAME READY STATUS RESTARTS AGE IP NODE<br />infinispan-749052960-pl066 1/1 Running 0 13m 10.40.0.4 gke-ispn-cluster-default-pool-59ed0e14-5sn9<br />ispn-1255975377-1g2kx 1/1 Running 0 1m 10.40.7.4 gke-ispn-cluster-default-pool-59ed0e14-k797<br />ispn-1255975377-333rl 1/1 Running 0 1m 10.40.9.3 gke-ispn-cluster-default-pool-59ed0e14-9lmz<br />ispn-1255975377-57zgl 1/1 Running 0 1m 10.40.1.4 gke-ispn-cluster-default-pool-59ed0e14-q80q<br />ispn-1255975377-6191p 1/1 Running 0 1m 10.40.0.5 gke-ispn-cluster-default-pool-59ed0e14-5sn9<br />ispn-1255975377-hm785 1/1 Running 0 10m 10.40.2.4 gke-ispn-cluster-default-pool-59ed0e14-r96s<br />ispn-1255975377-j8ckh 1/1 Running 0 1m 10.40.6.4 gke-ispn-cluster-default-pool-59ed0e14-j646<br />ispn-1255975377-jx70d 1/1 Running 0 10m 10.40.4.3 gke-ispn-cluster-default-pool-59ed0e14-1zdb<br />ispn-1255975377-lgvmt 1/1 Running 0 1m 10.40.8.3 gke-ispn-cluster-default-pool-59ed0e14-33pk<br />ispn-1255975377-xf9r8 1/1 Running 0 10m 10.40.5.3 gke-ispn-cluster-default-pool-59ed0e14-3t95</span></span><br />
<br />
The <span style="font-family: "Courier New",Courier,monospace;">infinispan</span> pod and one of the <span style="font-family: "Courier New",Courier,monospace;">ispn</span> pods have been created on the same GCE node <span style="font-family: "Courier New",Courier,monospace;">gke-ispn-cluster-default-pool-59ed0e14-5sn9</span>. The reason is that they are different deployments, and so GKE deploys them in an unrelated manner. Had all pods been created in the same depoyment, Kubernetes would have assigned pods to nodes in a round-robin fashion.<br />
<br />
This could be fixed by using labels, but I didn't want to complicate the demo. Note that running more that one pod per GCE node will harm performance slightly...<br />
<br />
Now we can run the performance test by populating the cluster (grid) with key/value pairs ([p]) and then running the test ([1]). This inserts 100'000 key/value pairs into the grid and executes read operations on every node for 1 minute (for details on IspnPerfTest consult the Github URL given earlier):<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"> Running test for 60 seconds:<br />1: 47,742 reqs/sec (286,351 reads 0 writes)<br />2: 56,854 reqs/sec (682,203 reads 0 writes)<br />3: 60,628 reqs/sec (1,091,264 reads 0 writes)<br />4: 62,922 reqs/sec (1,510,092 reads 0 writes)<br />5: 64,413 reqs/sec (1,932,427 reads 0 writes)<br />6: 65,050 reqs/sec (2,341,846 reads 0 writes)<br />7: 65,517 reqs/sec (2,751,828 reads 0 writes)<br />8: 66,172 reqs/sec (3,176,344 reads 0 writes)<br />9: 66,588 reqs/sec (3,595,839 reads 0 writes)<br />10: 67,183 reqs/sec (4,031,168 reads 0 writes)<br /><br />done (in 60020 ms)<br /><br /><br />all: get 1 / 1,486.77 / 364,799.00, put: 0 / 0.00 / 0.00<br /><br />======================= Results: ===========================<br />ispn-1255975377-1g2kx-51998: 100,707.90 reqs/sec (6,045,193 GETs, 0 PUTs), avg RTT (us) = 988.79 get, 0.00 put<br />ispn-1255975377-xf9r8-34843: 95,986.15 reqs/sec (5,760,705 GETs, 0 PUTs), avg RTT (us) = 1,036.78 get, 0.00 put<br />ispn-1255975377-jx70d-20952: 103,935.58 reqs/sec (6,239,149 GETs, 0 PUTs), avg RTT (us) = 961.14 get, 0.00 put<br />ispn-1255975377-j8ckh-11479: 100,869.08 reqs/sec (6,054,263 GETs, 0 PUTs), avg RTT (us) = 987.95 get, 0.00 put<br />ispn-1255975377-lgvmt-26968: 104,007.33 reqs/sec (6,243,144 GETs, 0 PUTs), avg RTT (us) = 960.05 get, 0.00 put<br />ispn-1255975377-6191p-15331: 69,004.31 reqs/sec (4,142,053 GETs, 0 PUTs), avg RTT (us) = 1,442.04 get, 0.00 put<br />ispn-1255975377-57zgl-58007: 92,282.75 reqs/sec (5,538,903 GETs, 0 PUTs), avg RTT (us) = 1,078.14 get, 0.00 put<br />ispn-1255975377-333rl-8583: 99,130.95 reqs/sec (5,949,542 GETs, 0 PUTs), avg RTT (us) = 1,004.08 get, 0.00 put<br />infinispan-749052960-pl066-18029: 67,166.91 reqs/sec (4,031,358 GETs, 0 PUTs), avg RTT (us) = 1,486.77 get, 0.00 put<br />ispn-1255975377-hm785-48520: 79,616.70 reqs/sec (4,778,196 GETs, 0 PUTs), avg RTT (us) = 1,254.87 get, 0.00 put<br /><br /><br />Throughput: 91,271 reqs/sec/node (91.27MB/sec) 912,601 reqs/sec/cluster<br />Roundtrip: gets min/avg/max = 0/1,092.14/371,711.00 us (50.0=938 90.0=1,869 95.0=2,419 99.0=5,711 99.9=20,319 [percentile at mean: 62.50]),<br /> puts n/a</span></span><br />
<br />
As suspected<span style="font-size: x-small;"><span style="font-size: small;"><span style="font-family: inherit;">, instances </span></span></span><span style="font-size: x-small;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">infinispan-749052960-pl066-18029</span></span> and </span></span></span><span style="font-size: x-small;"><span style="font-size: small;"><span style="font-family: inherit;"><span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">ispn-1255975377-6191p-15331</span></span> show lower performance than the other nodes, as they are co-located on the same GCE node.</span></span></span><br />
<br />
<br />
The Kubernetes integration in JGroups is done by <a href="https://github.com/jgroups-extras/jgroups-kubernetes/tree/master">KUBE_PING</a> which interacts with the Kubernetes master (API server) to fetch the IP addresses of the pods that have been started by Kubernetes.<br />
<br />
The KUBE_PING protocol is new, so direct problems, issues, configuration questions etc to the JGroups mailing list.<br />
<br />
<span style="font-size: x-small;"><span style="font-size: small;"><span style="font-family: inherit;"> </span></span><span style="font-family: "Courier New",Courier,monospace;"><br /></span></span><br />
<br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-39912222827146646062017-02-21T13:11:00.001+01:002017-02-21T13:11:50.349+01:00JGroups 4.0.0.FinalI'm happy to announce that JGroups 4.0.0.Final is out!<br />
<br />
With 120+ issues, the focus of this version is <b>API changes</b>, the switch to <b>Java 8</b> and thus the use of new language features (streams, lambdas) and <b>optimizations</b>.<br />
<br />
<br />
<br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="font-size: large;">API changes</span>
</pre>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-1605">https://issues.jboss.org/browse/JGRP-1605</a>]</span><br />
<br />
<h2>
<span style="font-size: small;">Fluent configuration</span></h2>
<span style="font-weight: normal;"><span style="font-size: small;">Example: </span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;">JChannel ch=new JChannel("config.xml").name("A").connect("cluster");</span></span><br />
<br />
<h2>
<span style="font-size: small;">Removed deprecated classes and methods</span></h2>
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Removed support for plain string-based channel configuration</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2019">https://issues.jboss.org/browse/JGRP-2019</a>]</span><br />
<span style="font-size: small;"><br /></span>
<b><span style="font-weight: normal;"><span style="font-size: small;">Use of Java 8 features</span></span></b><br />
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2007">https://issues.jboss.org/browse/JGRP-2007</a>]</span><br />
<span style="font-size: small;"><br />E.g. replace Condition with Predicate etc</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Removed classes</span></h2>
<ul>
<li><span style="font-size: small;">Shared transport [<a href="https://issues.jboss.org/browse/JGRP-1844">https://issues.jboss.org/browse/JGRP-1844</a>]</span></li>
<li><span style="font-size: small;">TCP_NIO</span></li>
<li><span style="font-size: small;">UNICAST, UNICAST2</span></li>
<li><span style="font-size: small;">NAKACK</span></li>
<li><span style="font-size: small;">PEER_LOCK</span></li>
<li><span style="font-size: small;">FD_PING</span></li>
<li><span style="font-size: small;">MERGE2</span></li>
<li><span style="font-size: small;">FC</span></li>
<li><span style="font-size: small;">SCOPE</span></li>
<li><span style="font-size: small;">MuxRpcDispatcher</span></li>
<li><span style="font-size: small;">ENCRYPT</span></li>
</ul>
<span style="font-size: small;"><br /></span>
<b><span style="font-size: small;">MessageDispatcher / RpcDispatcher changes</span></b><br />
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-1620">https://issues.jboss.org/browse/JGRP-1620</a>]</span><br />
<br />
<span style="font-size: small;">Use of CompletableFuture instead of NotifyingFuture</span><br />
<span style="font-size: small;"><br /><b>TCP: remove send queues</b></span><br />
<br />
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-1994">https://issues.jboss.org/browse/JGRP-1994</a>]</span><br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="font-size: large;">New features</span>
</pre>
<h2>
<span style="font-size: small;">Deliver message batches</span><span style="font-size: small;"> </span></h2>
<span style="font-weight: normal;"><span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2003">https://issues.jboss.org/browse/JGRP-2003</a>]</span></span><br />
<span style="font-size: small;"><br />Receiver (ReceiverAdapter) now has an additional callback receive(MessageBatch batch). This allows JGroups to pass an entire batch of messages to the application rather than passing them up one by one.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Refactored ENCRYPT into SYM_ENCRYPT and ASYM_ENCRYPT</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2021">https://issues.jboss.org/browse/JGRP-2021</a>]</span><br />
<span style="font-size: small;"><br />Plus fixed security issues in the refactored code. Removed ENCRYPT.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Measure round-trip times for RPCs via probe</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2049">https://issues.jboss.org/browse/JGRP-2049</a>]</span><br />
<span style="font-size: small;"><br />Keys 'rpcs' and 'rpcs-details' dump information about average RTTs between individual cluster members</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Change message bundler at runtime</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2058">https://issues.jboss.org/browse/JGRP-2058</a>]</span><br />
<span style="font-size: small;"><br />Message bundlers can be changed at runtime via probe. This is useful to see the effect of different bundlers on performance, even in the same test run.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Probe</span></h2>
<ul>
<li><span style="font-size: small;">Sort attributes for better legibility: <a href="https://issues.jboss.org/browse/JGRP-2066">https://issues.jboss.org/browse/JGRP-2066</a></span></li>
<li><span style="font-size: small;">Removal of protocols via regular expressions</span></li>
</ul>
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">DELIVERY_TIME: new protocol to measure delivery time in the application</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2101">https://issues.jboss.org/browse/JGRP-2101</a>]</span><br />
<span style="font-size: small;"><br />Exposes stats via JMX and probe</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">RELAY2: sticky site masters</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2112">https://issues.jboss.org/browse/JGRP-2112</a>]</span><br />
<span style="font-size: small;"><br />When we have multiple site masters, messages from the same member should always be handled by the same site master. This prevents reordering of messages at the receiver.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Multiple elements in bind_addr</span></h2>
<br />
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2113">https://issues.jboss.org/browse/JGRP-2113</a>]</span><br />
<span style="font-size: small;"><br />E.g.</span><br />
<span style="font-size: small;"><br /><span style="font-family: "courier new" , "courier" , monospace;"><TCP</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"> bind_addr="match-interface:eth2,10.5.5.5,match-interface:en*,127.0.0.1"</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"> ...</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"> /></span></span><br />
<span style="font-size: small;"><br />This tries to bind to eth2 first, then to 10.5.5.5, then to an interface that starts with en0, and</span><br />
<span style="font-size: small;">finally to loopback.</span><br />
<br />
<span style="font-size: small;">Useful when running in an environment where the IP addresses and/or interfaces are not known before.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Multiple receiver threads in UDP</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2146">https://issues.jboss.org/browse/JGRP-2146</a>]</span><br />
<span style="font-size: small;"><br />Multiple threads can receive (and process) messages from a datagram socket, preventing queue buildups.</span><br />
<span style="font-size: small;"><br /></span>
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">FRAG3</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2154">https://issues.jboss.org/browse/JGRP-2154</a>]</span><br />
<span style="font-size: small;"><br /></span>
<br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="font-size: large;">Optimizations</span>
</pre>
<h2>
<span style="font-size: small;">RpcDispatcher: don't copy the first anycast</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2010">https://issues.jboss.org/browse/JGRP-2010</a>]</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Reduction of memory size of classes</span></h2>
<ul>
<li><span style="font-size: small;">Rsp (32 -> 24 bytes): <a href="https://issues.jboss.org/browse/JGRP-2011">https://issues.jboss.org/browse/JGRP-2011</a></span></li>
<li><span style="font-size: small;">Request: <a href="https://issues.jboss.org/browse/JGRP-2012">https://issues.jboss.org/browse/JGRP-2012</a></span></li>
</ul>
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Remove one buffer copy in COMPRESS</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2017">https://issues.jboss.org/browse/JGRP-2017</a>]</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Replace Java serialization with JGroups marshalling</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2033">https://issues.jboss.org/browse/JGRP-2033</a>]</span><br />
<span style="font-size: small;"><br />Some internal classes still used Java serialization, which opens up security holes</span><br />
<span style="font-size: small;">(google 'java serialization vulnerability').</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Faster marshalling / unmarshalling of messages</span></h2>
<ul>
<li><span style="font-size: small;">Writing of headers: [<a href="https://issues.jboss.org/browse/JGRP-2042">https://issues.jboss.org/browse/JGRP-2042</a>]</span></li>
<li><span style="font-size: small;">Reading of headers: [<a href="https://issues.jboss.org/browse/JGRP-2043">https://issues.jboss.org/browse/JGRP-2043</a>]</span></li>
</ul>
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">TCP: reduce blocking</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2053">https://issues.jboss.org/browse/JGRP-2053</a>]</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Message bundler improvements</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2057">https://issues.jboss.org/browse/JGRP-2057</a>]</span><br />
<span style="font-size: small;"><br />E.g. removal of SingletonAddress: not needed anymore as shared transports have been removed, too.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Protocol: addition of up(Message) and down(Message) callbacks</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2067">https://issues.jboss.org/browse/JGRP-2067</a>]</span><br />
<span style="font-size: small;"><br />This massively reduces the number of Event creations.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">TransferQueueBundler: remove multiple messages</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2076">https://issues.jboss.org/browse/JGRP-2076</a>]</span><br />
<span style="font-size: small;"><br />Instead of removing messages one-by-one, the remover thread now removes as many messages as are in the queue (contention) into a local queue (no contention) and then creates and sends message batches off of the local queue.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Single thread pool</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2099">https://issues.jboss.org/browse/JGRP-2099</a>]</span><br />
<span style="font-size: small;"><br />There's only a single thread pool for all typs of messages, reducing the maintenance overhead of 4 thread pools and the configuration required.</span><br />
<br />
<span style="font-size: small;">The internal thread pool is still available (size to the number of cores), but not configurable.</span><br />
<span style="font-size: small;">A ForkJoinPool can be used instead of the thread pool (which can be disabled as well).</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">Timer: tasks can indicate that they will not block</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2100">https://issues.jboss.org/browse/JGRP-2100</a>]</span><br />
<span style="font-size: small;"><br />If a task calls execute() with an argument blocking==false, the task will be executed by the timer's main thread, and not be passed on to the timer's thread pool. This reduces the number of threads needed and therefore the number of context switches.</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">Headers are resized unnecessarily</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2120">https://issues.jboss.org/browse/JGRP-2120</a>]</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">ByteArrayDataOutputStream: expand more conservatively</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2124">https://issues.jboss.org/browse/JGRP-2124</a>]</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">Reading ints and longs creates unnecessary buffers</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2125">https://issues.jboss.org/browse/JGRP-2125</a>]</span><br />
<span style="font-size: small;"><br />Ints and longs are read into a byte[] array first, then parsed. This was changed to read the values and add them to the resulting ints or longs.</span><br />
<br />
<span style="font-size: small;">Should reduce overall memory allocation as ints and longs are used a lot in headers.</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">Table.removeMany() creates unneeded temp list</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2126">https://issues.jboss.org/browse/JGRP-2126</a>]</span><br />
<span style="font-size: small;"><br />This method is used a lot in NAKACK2 and UNICAST3. The change was to read messages directly into the resulting MessageBatch instead of a temp list and from there into the batch.</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">Reduce in-memory size of UnicastHeader3</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2127">https://issues.jboss.org/browse/JGRP-2127</a>]</span><br />
<span style="font-size: small;"><br />Reduced size from 40 -> 32 bytes.</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">Cache result of log.isTraceEnabled()</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2130">https://issues.jboss.org/browse/JGRP-2130</a>]</span><br />
<span style="font-size: small;"><br />This was done mainly for protocols where log.isTraceEnabled() was used a lot, such as TP, NAKACK2 or UNICAST3.</span><br />
<br />
<span style="font-size: small;">Note that the log level can still be changed at runtime.</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">Added MessageProcessingPolicy to define assigning of threads to messages or batches</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2143">https://issues.jboss.org/browse/JGRP-2143</a>]</span><br />
<span style="font-size: small;"><br />This only applies only to regular (not OOB or internal) messages. Make sure that only one message per member is processed at a given time by the thread pool.</span><br />
<br />
<span style="font-size: small;">This reduces the number of threads needed.</span><br />
<span style="font-size: small;"><br /></span>
<h2>
<span style="font-size: small;">UNICAST3 / NAKACK2: more efficient adding and removing of messages / batches to/from tables</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2150">https://issues.jboss.org/browse/JGRP-2150</a>]</span><br />
<span style="font-size: small;"><br />Simpler algorithm and removal of one lock (= less contention)</span><br />
<pre style="background-color: white; color: black; font-family: "Monaco";"></pre>
<br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="font-size: large;">Bug fixes</span>
</pre>
<h2>
<span style="font-size: small;">GMS sometimes ignores view bundling timeout</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2028">https://issues.jboss.org/browse/JGRP-2028</a>]</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">UFC and MFC headers get mixed up</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2072">https://issues.jboss.org/browse/JGRP-2072</a>]</span><br />
<span style="font-size: small;"><br />Although indepent protocols, the protocol ID was assigned by the superclass, so replenish and credit messages would get mixed up, leading to stuttering in the sending of credits.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">Flow control: replenish credits after message delivery</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2084">https://issues.jboss.org/browse/JGRP-2084</a>]</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">MERGE3: merge is never triggered</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2092">https://issues.jboss.org/browse/JGRP-2092</a>]</span><br />
<span style="font-size: small;"><br />This is an edge case that was not covered before: every subgroup coordinator has some other member as coord:</span><br />
<span style="font-size: small;">A: BAC</span><br />
<span style="font-size: small;">B: CAB</span><br />
<span style="font-size: small;">C: ABC</span><br />
<span style="font-size: small;"><br /></span>
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">MPING: restart fails</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2116">https://issues.jboss.org/browse/JGRP-2116</a>]</span><br />
<span style="font-size: small;"><br /></span>
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">UNICAST3 drops all messages until it receives one with first==true</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2131">https://issues.jboss.org/browse/JGRP-2131</a>]</span><br />
<span style="font-size: small;"><br />This caused a bug in ASYM_ENCRYPT.</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">ASYM_ENCRYPT: message batches are not handled correctly</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2149">https://issues.jboss.org/browse/JGRP-2149</a>]</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">SYM_ENCRYPT: allow for other keystores besides JCEKS</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2151">https://issues.jboss.org/browse/JGRP-2151</a>]</span><br />
<span style="font-size: small;"><br />E.g. pcks#12 or jks</span><br />
<span style="font-size: small;"><br /></span>
<br />
<h2>
<span style="font-size: small;">ASYM_ENCRYPT encrypts an empty buffer into a null buffer</span></h2>
<span style="font-size: small;">[<a href="https://issues.jboss.org/browse/JGRP-2153">https://issues.jboss.org/browse/JGRP-2153</a>]</span><br />
<span style="font-size: small;"><br />This caused an NPE.</span><br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"></pre>
<br />
<span style="font-size: large;">Downloads</span><br />
<br />
<span style="font-size: large;"><span style="font-size: small;">On Sourceforge: <a href="https://sourceforge.net/projects/javagroups/files/JGroups/4.0.0.Final">https://sourceforge.net/projects/javagroups/files/JGroups/4.0.0.Final</a>,</span></span><br />
<span style="font-size: large;"><span style="font-size: small;">or via Maven:</span> </span><br />
<br />
<span style="background-color: #efefef;"><</span><span style="background-color: #efefef; color: navy; font-weight: bold;">groupId</span><span style="background-color: #efefef;">></span>org.jgroups<span style="background-color: #efefef;"></</span><span style="background-color: #efefef; color: navy; font-weight: bold;">groupId</span><span style="background-color: #efefef;">></span>
<span style="background-color: #efefef;"> </span><br />
<span style="background-color: #efefef;"><</span><span style="background-color: #efefef; color: navy; font-weight: bold;">artifactId</span><span style="background-color: #efefef;">></span>jgroups<span style="background-color: #efefef;"></</span><span style="background-color: #efefef; color: navy; font-weight: bold;">artifactId</span><span style="background-color: #efefef;">></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef;"></span><br />
<span style="background-color: #efefef;"><</span><span style="background-color: #efefef; color: navy; font-weight: bold;">version</span><span style="background-color: #efefef;">></span>4.0.0.Final<span style="background-color: #efefef;"></</span><span style="background-color: #efefef; color: navy; font-weight: bold;">version</span><span style="background-color: #efefef;">></span><span style="background-color: #efefef;"> </span><br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="font-size: large;"> </span></pre>
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="font-size: large;">Manual</span>
</pre>
<span style="font-size: small;">The manual is at <a href="http://www.jgroups.org/manual4/index.html">http://www.jgroups.org/manual4/index.html</a>.</span><br />
<br />
<span style="font-size: small;">The complete list of features and bug fixes can be found at </span><br />
<span style="font-size: small;"><a href="http://jira.jboss.com/jira/browse/JGRP">http://jira.jboss.com/jira/browse/JGRP</a>.</span><br />
<br />
<span style="font-size: small;">Bela Ban, Kreuzlingen, Switzerland</span><br />
<span style="font-size: small;">March 2017</span><br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="font-size: small;">
</span></pre>
Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0tag:blogger.com,1999:blog-19835054.post-44330849650888293152016-09-08T10:03:00.000+02:002016-09-08T10:03:49.737+02:00Removing thread pools in JGroups 4.0JGroups 3.x has 4 thread pools:<br />
<ul>
<li>Regular thread pool: used for regular messages (default has a queue)</li>
<li>OOB thread pool: used for OOB messages (no queue)</li>
<li>Internal thread pool: used for JGroups internal messages only. The main raison d'etre for this pool was that internal messages such as heartbeats or credits should never get queued up behind other messages, and get processed immediately.</li>
<li>Timer thread pool: all tasks in a timer need to be executed by a thread pool as they can potentially block</li>
</ul>
Over time, I found out that - with most configurations - I had the queue in the regular thread pool disabled, as I wanted the pool to dynamically create new threads (up to max-threads) when required, and terminate them again after some idle time.<br />
<br />
Hence the idea to club regular and OOB thread pools into one.<br />
<br />
When I further thought about this, I realized that incoming messages could also be handled by a queue-less thread pool: by handling RejectedExecutionException thrown when the pool is full and simply spawning a new thread to process the internal message, so it wouldn't get dropped.<br />
<br />
The same goes for timer tasks: a timer task (e.g. a message retransmission task) cannot get dropped, or retransmission would cease. However, by using the same mechanism as for internal messages, namely spawning a new thread when the thread pool is full, this can be solved.<br />
<br />
Therefore, in 4.0 there's only a single thread pool handling regular, OOB and internal messages, plus timer tasks.<br />
<br />
The new thread pool has no queue, or else it would not throw a RejectedExecutionException when a task is added, but simply queue it, which is not what we want for internal messages or timer tasks.<br />
<br />
It also has a default rejection policy of "abort" which cannot be changed (only by substituting the thread pool with a custom pool).<br />
<br />
This dramatically reduces configuration complexity: from 4 to 1 pools, and the new thread pool only exposes min-threads, max-threads, idle-time and enabled as configuration options.<br />
<br />
Here's an example of a 3.x configuration:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> timer_type="new3"<br /> timer.min_threads="2"<br /> timer.max_threads="4"<br /> timer.keep_alive_time="3000"<br /> timer.queue_max_size="500"<br /><br /> thread_pool.enabled="true"<br /> thread_pool.min_threads="2"<br /> thread_pool.max_threads="8"<br /> thread_pool.keep_alive_time="5000"<br /> thread_pool.queue_enabled="true"<br /> thread_pool.queue_max_size="10000"<br /> thread_pool.rejection_policy="discard"<br /><br /> internal_thread_pool.enabled="true"<br /> internal_thread_pool.min_threads="1"<br /> internal_thread_pool.max_threads="4"<br /> internal_thread_pool.keep_alive_time="5000"<br /> internal_thread_pool.queue_enabled="false"<br /> internal_thread_pool.rejection_policy="discard"<br /><br /> oob_thread_pool.enabled="true"<br /> oob_thread_pool.min_threads="1"<br /> oob_thread_pool.max_threads="8"<br /> oob_thread_pool.keep_alive_time="5000"<br /> oob_thread_pool.queue_enabled="false"<br /> oob_thread_pool.queue_max_size="100"<br /> oob_thread_pool.rejection_policy="discard"</span><br />
<br />and here's the 4.0 configuration:<br />
<br />
<span style="background-color: #efefef; color: green; font-weight: bold;"></span><span style="font-family: "Courier New",Courier,monospace;"> thread_pool.enabled="true"<br /> thread_pool.min_threads="2"<br /> thread_pool.max_threads="8"<br /> thread_pool.keep_alive_time="5000"<br /><span style="background-color: #efefef; color: green; font-weight: bold;"></span></span><br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="font-family: "Courier New",Courier,monospace;"><span style="background-color: #efefef; color: green; font-weight: bold;"></span></span><span style="background-color: #efefef; color: blue; font-weight: bold;">
</span></pre>
<br />
Nice, isn't it?<br />
<br />
Cheers,Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com3tag:blogger.com,1999:blog-19835054.post-25097502109287114802016-08-23T13:17:00.000+02:002016-08-23T13:17:04.842+02:00JGroups 4.0.0.Alpha1 releasedHowdy folks,<br />
<br />
I just released a first alpha of JGroups 4.0.0 to SourceForge and maven. There are 12+ issues left, and I expect a final release in October.<br />
<br />
The major version number was incremented because there are a lot of API changes and removal of deprecated classes/methods.<br />
<br />
I'd be happy to get feedback on the API; there's still time to make a few API changes until the final release.<br />
<br />
<h3>
Major API changes</h3>
<ul>
<li>Deprecated classes and methods were removed (TCP_NIO, UNICAST, UNICAST2, NAKACK, PEER_LOCK, MERGE, MERGE2, FC, SCOPE etc)</li>
<li>The shared transport was removed (<a href="https://issues.jboss.org/browse/JGRP-1844">https://issues.jboss.org/browse/JGRP-1844</a>). Use fork channels instead.</li>
<li>MuxRpcDispatcher was finally removed (<a href="https://issues.jboss.org/browse/JGRP-2008">https://issues.jboss.org/browse/JGRP-2008</a>). YEAH!!!</li>
<li>RpcDispatcher now uses CompletableFuture (<a href="https://issues.jboss.org/browse/JGRP-1605">https://issues.jboss.org/browse/JGRP-1605</a>)</li>
<li>Changed code to use new Java 8 features (e.g. streams): <a href="https://issues.jboss.org/browse/JGRP-2007">https://issues.jboss.org/browse/JGRP-2007</a></li>
<li>Removed the Channel interface and merged it into JChannel: <a href="https://issues.jboss.org/browse/JGRP-2018">https://issues.jboss.org/browse/JGRP-2018</a> </li>
</ul>
<br />
<br />
<h3>
New features </h3>
<h4>
</h4>
<h4>
Deliver message batches</h4>
Ability to handle message batches in addition to individual messages: receive(MessageBatch) was added to ReceiverAdapter: <a href="https://issues.jboss.org/browse/JGRP-2003">https://issues.jboss.org/browse/JGRP-2003</a><br />
<br />
<h4>
Encryption and authentication</h4>
ENCRYPT was removed and replaced by SYM_ENCRYPT (keystore-based) and ASYM_ENCRYPT (asymmetric key exchange): <a href="https://issues.jboss.org/browse/JGRP-2021">https://issues.jboss.org/browse/JGRP-2021</a>.<br />
<br />
ENCRYPT had a couple of security flaws, and since the code hasn't been touched for almost a decade, I took this chance and completely refactored it into the 2 protocols above.<br />
<br />
The code is now much more readable and maintainable.<br />
<br />
<br />
<h3>
Optimizations</h3>
<h4>
</h4>
<h4>
Removal of Java serialization</h4>
JGroups marshalling falls back to Java serialization if a class is not primitive or implements Streamable. This was completely removed for JGroups-internal classes: <a href="https://issues.jboss.org/browse/JGRP-2033">https://issues.jboss.org/browse/JGRP-2033</a><br />
<br />
<h4>
MessageBundler improvements</h4>
The performance of existing bundlers has been improved, and new bundlers (RingBufferBundler, NoBundler) have been added. Since the shared transport feature was removed, SingletonAddress used as key into a bundler's hashmap could also be removed.<br />
<br />
For instance, TransferQueueBundler now removes chunks of message from the queue, rather than one-by-one: <a href="https://issues.jboss.org/browse/JGRP-2076">https://issues.jboss.org/browse/JGRP-2076</a>.<br />
<br />
I also added the ability to switch bundlers at runtime (via (e.g.) probe.sh op=TCP.bundler["rb"]): <a href="https://issues.jboss.org/browse/JGRP-2057">https://issues.jboss.org/browse/JGRP-2057</a> / <a href="https://issues.jboss.org/browse/JGRP-2058">https://issues.jboss.org/browse/JGRP-2058</a><br />
<br />
<br />
<h4>
Reduction of event creation</h4>
For every message sent up or down, a new Event (wrapping the message) had to be created. Adding 2 separate up(Message) and down(Message) callbacks removed that unneeded creation: <a href="https://issues.jboss.org/browse/JGRP-2067">https://issues.jboss.org/browse/JGRP-2067</a><br />
<br />
<br />
<h4>
Faster header marshalling and creation</h4>
Headers used to do a hashmap lookup to get their magic-id when getting marshalled. Now they carry the magic ID directly in the header. This saves a hashmap lookup per header (usually we have 3-4 headers per message): <a href="https://issues.jboss.org/browse/JGRP-2042">https://issues.jboss.org/browse/JGRP-2042</a>.<br />
<br />
Header creation was getting the class using a lookup with the magic ID, and then creating an instance. This was replaced by each header class implementing a create() method, resulting in header creation getting from 300ns down to 25ns.<br />
<a href="https://issues.jboss.org/browse/JGRP-2043">https://issues.jboss.org/browse/JGRP-2043</a>.<br />
<br />
<br />
<h3>
Bug fixes</h3>
<br />
<h4>
MFC and UFC headers getting mixed up</h4>
This is a regression that led to premature exhaustion of credits, slowing things down: <br />
<a href="https://issues.jboss.org/browse/JGRP-2072">https://issues.jboss.org/browse/JGRP-2072</a> <br />
<br />
<br />
<h3>
Download</h3>
<br />
<pre style="background-color: white; color: black; font-family: 'Monaco'; font-size: 9.0pt;"><span style="background-color: #efefef;"><</span><span style="background-color: #efefef; color: navy; font-weight: bold;">groupId</span><span style="background-color: #efefef;">></span>org.jgroups<span style="background-color: #efefef;"></</span><span style="background-color: #efefef; color: navy; font-weight: bold;">groupId</span><span style="background-color: #efefef;">></span>
<span style="background-color: #efefef;"><</span><span style="background-color: #efefef; color: navy; font-weight: bold;">artifactId</span><span style="background-color: #efefef;">></span>jgroups<span style="background-color: #efefef;"></</span><span style="background-color: #efefef; color: navy; font-weight: bold;">artifactId</span><span style="background-color: #efefef;">></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef;"></span><span style="background-color: #efefef; color: navy; font-weight: bold;"></span><span style="background-color: #efefef;"></span>
<span style="background-color: #efefef;"><</span><span style="background-color: #efefef; color: navy; font-weight: bold;">version</span><span style="background-color: #efefef;">></span>4.0.0-Alpha1<span style="background-color: #efefef;"></</span><span style="background-color: #efefef; color: navy; font-weight: bold;">version</span><span style="background-color: #efefef;">></span></pre>
<br />
<br />
The manual has not been changed yet, so browse the source code or javadocs for details.<br />
<br />
Enjoy! and feedback via the mailing list, please!<br />
<br />
Cheers,<br />
<br />
Bela Ban<br />
<br />
<br />Bela Banhttp://www.blogger.com/profile/03832183909163653875noreply@blogger.com0