Arquillian in-container tests with WildFly, Infinispan Server and remote OpenShift

The Arquillian Cube project provides a nice API and everything you need for running standalone tests against a local OpenShift instance (e.g. started by oc cluster up). However, running in-container tests against a remote OpenShift instance requires a few tricks. This is a tutorial on how to execute in-container tests on WildFly 11 running in a remote OpenShift instance. Throughout this tutorial we utilise our own instance of OpenShift, however we could also have used the OpenShift Online Starter edition which is available for free.

• First we must login to the remote OpenShift cluster as recommended by the provider. For example:

          oc login <openshift address> --token=<token>

Next, it is necessary to note the remote OpenShift's Docker registry. This registry is required so that we can push our created images to the remote registry, which enables other services deployed in OpenShift to consume them. In our case it will be registry.online-int.openshift.com.

•  Create an OpenShift project:

     oc new-project myproject

• Switch to the project:

           oc project myproject

• Build your image that you'd like to test in remote OpenShift. I've used Infinispan server Docker image which is extended in our project. 

           sudo docker build -t infinispan-server-dev ./infinispan-server

    Our Dockerfile looks like this. It doesn't really modify any behaviour, it just shows that we can create our own image based on another image and push it to the remote OpenShift's Docker registry. Should you be interested in how to build a Docker image from a snapshot of Infinispan server, refer to an older article: Bleeding edge on Docker
 

	FROM jboss/infinispan-server
	#Do something here ...

view raw Dockerfile hosted with ❤ by GitHub

• Build the remote container's image where tests will be run. In our case WildFly with a few modifications. 

           sudo docker build -t wildfly11-testrunner ./wildfly11-testrunner    

  The Dockerfile for a modified WildFly server looks like this:

	FROM jboss/wildfly
	EXPOSE 9990
	RUN /opt/jboss/wildfly/bin/add-user.sh admin Admin#70365 --silent
	CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]

view raw Dockerfile hosted with ❤ by GitHub

The Dockerfile above exposes port 9990, this is required so that Arquillian is to be able to perform a remote deployment via the management port. Furthermore, we also expose the management interface on address 0.0.0.0 so that the port is forwarded to the localhost. Finally, we add a user who will later connect and login via Arquillian.

• Login to remote docker registry:

          sudo docker login -u $(shell oc whoami) -p $(shell oc whoami -t) registry.online-int.openshift.com

• Tag your images and push them to the remote docker registry.

   The images must be pushed under this schema: <registry_address>/<project_name>/<image_name>

          sudo docker tag infinispan-server-dev registry.online-int.openshift.com/myproject/infinispan-server-dev

    sudo docker push registry.online-int.openshift.com/myproject/infinispan-server-dev

    sudo docker tag wildfly11-testrunner registry.online-int.openshift.com/myproject/wildfly11-testrunner

    sudo docker push registry.online-int.openshift.com/myproject/wildfly11-testrunner

• Enable image stream lookups for our images so that we can use them from our application.

          oc set image-lookup infinispan-server-dev
    oc set image-lookup wildfly11-testrunner

• Create an application that will be using our infinispan-server-dev image:

         oc new-app infinispan-server-dev -e "APP_USER=user" -e "APP_PASS=changeme"

• Run the functional tests from Maven:

          mvn -Dkubernetes.auth.token=
kkF_eC3ZXcsddf5RUeZFXw4Ce9rC_uVVZt875mLsQ 
clean test -f functional-tests/pom.xml

    This maven command specifies a property named kubernetes.auth.token. This token is used to create an OpenShiftClient instance within the WildFly container.
 
    The arquillian.xml file follows.

	<?xml version="1.0" encoding="UTF-8"?>
	<arquillian xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns="http://jboss.org/schema/arquillian"
	xsi:schemaLocation="http://jboss.org/schema/arquillian http://jboss.org/schema/arquillian/arquillian_1_0.xsd">
	<extension qualifier="openshift">
	<property name="namespace.use.current">true</property>
	<!-- Pod running remote tests. -->
	<property name="definitionsFile">target/classes/wildfly11-testrunner.json</property>
	<property name="wait.for.service.list">infinispan-server-dev</property>
	<property name="proxiedContainerPorts">testrunner:9990,testrunner:8080</property>
	<!-- Fetch the logs from Openshift and pods, and save them into target/surefire-reports -->
	<property name="logs.copy">true</property>
	</extension>
	<container qualifier="testrunner" mode="suite" default="true">
	<configuration>
	<property name="username">admin</property>
	<property name="password">Admin#70365</property>
	</configuration>
	</container>
	</arquillian>

view raw arquillian.xml hosted with ❤ by GitHub

Notice the definitionsFile property which specifies the WildFly server's pod. We previously pushed the image and created the image stream that is now available in OpenShift. And the wildfly11-testrunner consumes the WildFly image by pointing to the image stream called "wildfly11-testrunner:latest"
 
    Here's the json definition

The test class is less interesting. It basically needs a method annotated with @Deployment. This method tells Arquillian which classes/libraries to bundle withing the archive which is  deployed in a remote container (in this case, WildFly 11).
     Another interesting part is that Arquillian Cube's annotations such as @Named, or @RouteURL don't work in-container as of now. Similar to injecting OpenShiftClient via @ArquillianResource. So we have to create our own instances manually.

	@RunWith(ArquillianConditionalRunner.class)
	@RequiresOpenshift
	public class HotRodServiceTest {
	OpenShiftClient client = OpenShiftClientCreator.getClient();
	OpenShiftHandle handle = new OpenShiftHandle(client);
	ReadinessCheck readinessCheck = new ReadinessCheck();
	@Deployment
	public static Archive<?> deploymentApp() {
	return ShrinkWrap
	.create(WebArchive.class, "test.war")
	.addAsLibraries(DeploymentHelper.testLibs())
	.addPackage(ReadinessCheck.class.getPackage());
	}
	@Before
	public void before() throws MalformedURLException {
	readinessCheck.waitUntilAllPodsAreReady(client);
	}
	@Test
	public void testPutGet() throws IOException {
	URL serviceUrl = handle.getServiceWithName("infinispan-server-dev");
	RemoteCacheManager manager = getRemoteCacheManager(serviceUrl);
	RemoteCache<String, String> cache = manager.getCache();
	cache.put("k", "v");
	assertThat(cache.get("k")).isEqualTo("v");
	}
	private RemoteCacheManager getRemoteCacheManager(URL urlToService) {
	Configuration cachingServiceClientConfiguration = new ConfigurationBuilder()
	.addServer()
	.host(urlToService.getHost())
	.port(urlToService.getPort())
	.build();
	return new RemoteCacheManager(cachingServiceClientConfiguration);
	}
	}

view raw test.java hosted with ❤ by GitHub

There you go. The full working example is available at https://github.com/mgencur/arquillian-remote-openshift-example . The project has a convenient Makefile which can do all the steps by simply invoking `make test-remote`. The readme file provides more info how to run the tests.