RadarGun and performance testing in OpenShift

RadarGun has a brand new feature that allows for performance testing of distributed caches in OpenShift. There's a helper script called "openshift" and a new OpenShift plugin. Together they make it possible to test both library and client-server mode. In library mode the cache and the loader thread run in the same JVM process. In client-server mode the clients and servers run in
separate processes (and in this case even separate Pods).

I'll describe the process of running performance tests in OpenShift with Infinispan server and HotRod clients in client-server mode.

The process is as follows:

1.   Checkout the RadarGun master branch from https://github.com/radargun/radargun

2.   Build RadarGun from the top-level directory using Maven. This will produce the  target/distribution/RadarGun-3.0.0-SNAPSHOT directory that will be used in the resulting Docker image.

    mvn clean install

3.   Export the DOCKER_REGISTRY address where RadarGun images will be pushed for consumption by OpenShift deployments.

    export DOCKER_REGISTRY=<OpenShift's docker registry address>
 
    The registry address can be usually obtained by opening the "About" link in the top-right corner of your OpenShift project's web UI.
 
4.   Go to the openshift sub-directory. Pass the login command with the correct parameters specific to the OpenShift provider:

    ./openshift -L "oc login https://api.rh-us-xxx-1.openshift.com --token=l5gGjuKOPAWsdf6564sdfeOI7qhmQCGhJLvyj4oa4" login
 
    Note: The login command can be usually obtained by opening "Command line tools" link in the top-right corner of your OpenShift project's web UI.

5.   Create your project! This is named "myproject" by default. Customization is only possible by changing the openshift script.

    ./openshift newproject

6.   Build the RadarGun image and push it to the remote Docker registry.

    ./openshift build
 
    This command will take the target/distribution/RadarGun-3.0.0-SNAPSHOT directory from previous steps and create a Docker image based on it.

7.   Create a subdirectory named "configs" with the RadarGun benchmark, config files required by individual RadarGun plugins, and config file for Infinispan server. This directory will be mounted in master and slave Pods in OpenShift as /opt/radargun-configs. And as you'll see the Infinispan server template will also mount this directory under ${INFINISPAN_SERVER_HOME}/standalone/configuration/custom so that the configuration file that you place in this directory can be used by the server.

8.   Create the RadarGun deployment.

    ./openshift -cd "configs/" -cf "benchmark-openshift-hotrod.xml" -s 2 deploy
 
    The example uses a RadarGun config file named "benchmark-openshift-hotrod.xml" . It needs to be placed in the config/ sub-directory before running the command. The parameter -s determines the number of RadarGun slaves. Each slave will run in a separate Pod in OpenShift. The RadarGun master will run in its own Pod. In summary, this command will spin up one Pod for master and 2 Pods for slaves.
 
    Here's a snippet from the RadarGun benchmark. Comments in the file will shed some light on it.
         
    A full config with an artificial test scenario is here.
 
    Overall, this benchmark expects two RadarGun slaves to be running. One of the slaves will then deploy the template which results in two additional Pods for Infinispan servers. The other slave then holds the HotRod clients which will be sending requests to the server.

Here's a screenshot from OpenShift GUI at this point:
 

 
    The OpenShift template for Infinispan server from the benchmark specifies several objects that will be created in OpenShift:

• HotRod service available on port 11222
• REST service available on port 8080
• JGroups Ping service which is later reference by JGroups DNS_PING protocol 
• Two Infinispan server instances based on jboss/infinispan-server Docker image 

Below is a short snippet that shows a trick to pass the correct address of the DNS server to JGroups. JGroups then uses this DNS to find the address of the aforementioned JGroups Ping service which forwards discovery messages from server nodes. This way Infinispan/JGroups get to know the other cluster members.
   
   
     Full Infinispan Server template here.
   
   
     The following is a short snippet that shows the DNS_PING protocol configuration in JGroups subsystem of Infinispan servers. Other parts of the config file are not specific to OpenShift.
   
   
     Full Infinispan Server config available here.

9.    Alright, now when the tests have finished, we should be able to download test results and logs.

    ./openshift results

And we're done! We can now analyze the results which have been stored in radargun-data sub-directory.