You might wonder if I have nothing better to do (like biking in the French Alps) :-)
But actually, there have been significant changes since CR1, so please read on !
CR2 only contains 3 JIRA issues:
#1 is a partial backport of NAKACK from head (2.8) to the 2.6 branch. This version doesn't acquire locks for incoming messages anymore, but uses a CAS (compare-and-swap) operation to decide whether to process a message, or not.
What used to happen when a message from P is received is that we grabbed the receiver window for P and added the message. Then we grabbed the lock associated with P's window and - once acquired - removed as many messages as possible and passed them up to the application sequentially. Sequential order is always respected unless a message is tagged as OOB (out-of-band).
So here's what happened: say we received 10 multicast messages from B and 3 from A. Both A's and B's messages would be delivered in parallel with respect to each other, but sequentially for a given sender. So A's message #34 would always get delivered before #35 before #36 and so on...
However, say we have to process 10 messages from B: 1 2 3 4 5 6 7 8 9 10:
- Every message would get into NAKACK on a separate thread
- All the 10 messages would get added into B's receiver window
- The thread with message #3 would grab the lock
- All other threads would block, trying to acquire the lock
- The thread with the lock would remove #1 and pass it up the stack, then #2, then #3 and so on, until it passed #10 up the stack to the application
- Now it releases the lock
- All other 9 threads now compete for the lock, but every single thread will return because there are no more messages in the receiver window
For example, if our total thread pool only had 10 threads, and 1 of them was processing messages and 9 were blocked on lock acquisition, if a message from a different sender came in (which could be delivered in parallel to B's messages), then no thread would be available !
So the simple but effective change was to replace the lock on the receive window with a CAS: when a thread tries to remove messages, it simply set the CAS from false to true. If it succeed, it goes into the removal loop and sets the CAS back to false when done. Else, the thread simply returns because it knows that someone else will be processing the message it just added.
Result: we've returned 9 threads to the thread pool, ready to serve other messages, without even locking !
The net affect is faster performance and smaller thread pools. As a rule of thumb, a thread pool's max threads can now be around the number of cluster nodes: if every node sends messages, we only need 1 thread per sender to process all of the sender's messages...
#2 has 2 changes: same as above (locks replaced by CAS) and the changes outlined in the design document. The latter changes simplify UNICAST a lot and also handle the cases of asymmetrical connection closings. This was also back-ported from head (2.8)
#3 UNICAST contention issues
We used to have 2 big fat locks in UNICAST, which severely impacted performance on high unicast message volumes. The bottleneck was detected as part of our EAP testing for JBoss.
This has been fixed and is getting forward-ported to CVS head.
I guess the 3 changes are worth trying out 2.6.13.CR2; in some cases this should make a real difference in performance !
Enjoy,
