Infinispan Arquillian Container 1.0.0.CR1 released

Infinispan Arquillian Container is an extension to Arquillian that provides several ways to interacting with Infinispan, either with a standalone Infinispan server or just with Infinispan libraries. This extension can also communicate with JBoss Data Grid server via JMX.

It was released as Maven artifacts in JBoss Maven Repository. It is located at http://repository.jboss.org/nexus/content/groups/public-jboss/ . More information on how to set up and use the repo can be found at https://community.jboss.org/wiki/MavenGettingStarted-Users

What does this Arquillian extension offer to you? Let me describe all aspects of this extension one by one.

Developing tests with standalone Infinispan server

When testing, you might want to automatically start the Infinispan server before the test and stop it afterwards. This can be achieved by configuring infinispan-arquillian-container via Arquillian's configuration file. The following is a subset of attributes that can be specified and thus passed to the Infinispan server during startup: masterThreads, workerThreads, cacheConfig, jmxPort, ... The complete list can be found in bit.ly/R7j4d1 (all private fields).


NOTE: Examples are not a part of the release, only libraries are. In order to check out examples provided with the project, one has to clone project's repository: https://github.com/mgencur/infinispan-arquillian-container Examples are located in the respective sub-directory.

The configuration file then looks similar to the following:

Complete example: bit.ly/RkrpEE

When we tell Arquillian to work with Infinispan server, we can inject RemoteInfinispanServer object into our test. Such an object provides various information about the running Infinispan server. For example, we can retrieve a hostname and HotRod port and use these pieces of information to create a RemoteCacheManager instance. Besides that users are allowed to retrieve information available via JMX from the server like cluster size, number of entries in the cache, number of cache hits and many more.

Complete example: http://bit.ly/OaCw8q

Vital dependencies required for the test to run are:

org.infinispan.arquillian.container:infinispan-arquillian-container-managed:jar:1.0.0.CR1:test
org.infinispan.arquillian.container:infinispan-arquillian-impl:jar:1.0.0.CR1:test

Not only with standalone Infinispan server can Infinispan Arquillian extension work.

Developing tests with JBoss Data Grid (JDG) server

This time, the properties in Arquillian's configuration file are different and correspond to properties of JBoss Application Server 7. The most important property is again the path to the server (jbossHome).

Are you interested in what the test look like? It looks completely the same as tests for standalone Infinispan server, you just have a few more attributes available. JDG server usually starts all three endpoints (HotRod, Memcached, REST) at the same time while for the Infinispan server you have to specify which end point should be started. Furthermore, Infinispan server does not have the REST endpoint available out-of-the-box.

As a result, you can call the following methods with JDG in one single test.

server1.getMemcachedEndpoint().getPort();
server1.getRESTEndpoint().getContextPath();
server1.getHotRodEndpoint().getPort();

The difference is, of course in dependencies. Instead of a handler for standalone Infinispan server, one has to use a handler for JBoss AS 7. The dependencies then look like this:

org.jboss.as:jboss-as-arquillian-container-managed:jar:7.1.2.Final:test
org.infinispan.arquillian.container:infinispan-arquillian-impl:jar:1.0.0.CR1:test

Testing Infinispan libraries

Sometimes we don't want to use a standalone server. Sometimes we want to test just Infinispan in its basic form - Java libraries. Infinispan has been under development for years and during that time, lots of tests were developed. With tests come utility methods. Infinispan Arquillian Container enables you to leverage these utility methods and call them via an instance of DatagridManager. This instance can be easily injected into a test, no matter which test framework (TestNG, JUnit) you use.

DatagridManager class can be found at http://bit.ly/Q0a7ki

Can you see the advantage? No? Let me point out some useful methods available in the manager.

List<Cache<K, V>> createClusteredCaches(int numMembersInCluster, String cacheName, ConfigurationBuilder builder)

- creates a cluster of caches with certain name and pre-defined configuration

void waitForClusterToForm()

- helps to wait until the cluster is up and running

Cache<A, B> cache(int index)

- retrieves a cache from certain node according to the index

Cache<A, B> cache(int managerIndex, String cacheName)

- retrieves a cache with that name

void killMember(int cacheIndex)

- kills a cache with cacheIndex index

AdvancedCache advancedCache(int i)

- retrieves an advanced cache from node i

Trancation tx(int i)

- retrieves a transaction from node i

TransactionManager tm(int i)

- retrieves a transaction manager from node i

...and much more.


The following test can be found among other examples in the GIT repository.

Required dependencies:

org.infinispan:infinispan-core:jar:5.1.5.FINAL:test  -  users should replace this version with the one they want to test
org.infinispan.arquillian.container:infinispan-arquillian-impl:jar:1.0.0.CR1:test

Infinispan Arquillian Container was tested with Infinispan 5.1.5.FINAL and JDG 6.0.0.GA. Nevertheless, it should work smoothly also with other not-very-distinct versions. I'll be updating the project to work with newer versions of both Infinispan and JBoss Data Grid.
 

Fine-grained replication in Infinispan

Sometimes we have a large object, possibly with lots of attributes or holding some binary data, and we would like to tell Infinispan to replicate only certain part of the object across the cluster. Typically, we wanna replicate only that part which we've just updated. This is where DeltaAware and Delta interfaces come to play. By providing implementations of these interfaces we can define fine-grained replication. When we put some effort into such such an enhancements, we would also like to speed up object marshalling and unmarshalling. Therefore, we're going to define our own externalizers - to avoid slow default Java serialization.

The following code snippets are gathered in a complete example at https://github.com/mgencur/infinispan-examples/tree/master/partial-state-transfer This project contains a readme file with instructions on how to build and run the example. It is based on clustered-cache quickstart in Infinispan.

Implementing DeltaAware interface

So let's look at our main object. For the purpose of this excercise, I defined a Bicycle class that consists of many components like frame, fork, rearShock, etc. This object is stored in a cache as a value under certain (not important) key. It might happen in our scenario that we update only certain components of the bike and in such case we want to replicate just those component changes.

Important methods here are (description taken from javadocs):

commit() - Indicates that all deltas collected to date has been extracted (via a
                 call to delta()) and can be discarded. Often used as an optimization if
                 the delta isn't really needed, but the cleaning and resetting of       
                 internal state is desirable.

delta() - Extracts changes made to implementations, in an efficient format that
             can easily and cheaply be serialized and deserialized.  This method will
             only be called once for each changeset as it is assumed that any
             implementation's internal changelog is wiped and reset after generating
             and submitting the delta to the caller.
         
We also need to define setters and getters for our members. Setter methods are, among other things, responsible for registering changes to the changelog that will be later used to reconstruct the object's state. The externalizer for this class is only needed when cache stores are used. For the sake of simplicity, I don't mention it here.

Implementing Delta interface

Actual object that will be replicated across the cluster is the implementation of Delta interface. Let's look at the class. First, we need a field that will hold the changes - changeLog. Second, we need to define a merge() method. This method must be implemented so that Infinispan knows how to merge an existing object with incoming changes. The parameter of this method represents an object that is already stored in a cache, incoming changes will be applied to this object. We're using a reflection here to apply the changes to the actual object but it is not necessary. We could easily call setter methods. The advantage of using reflection is that we can set those fields in a loop.

Another piece is a registerComponentChange() method. This is called by an object of the Bicycle class - to record changes to that object. The name of this method is not important.

Defining our own externalizer

Alright, so what remains is the externalizer definition for the Delta implementation. We implement AdvancedExternalizer interface and say that only changeLog object should be marshalled and unmarshalled when transfering data over the wire. A complete (almost) implementation of Delta interface is the following.

Tell Infinispan about the extra externalizer

We also need to configure Infinispan to use our special externalizer to marshall/unmarshall our objects. We can do it e.g. programatically by calling .addAdvancedExternalizer() on the serialization configuration builder.

You can see we're also configuring transactions here. This is not necessary, though. We just aim to provide a richer example, removing transactional behavior is trully easy.

And here comes the "usage" part. Enclose cache calls by a transaction, retrieve a bicycle object from the cache, do some changes and commit them.

That's it. What is eventually transferred over the wire is just the changeLog object. The actual bicycle object is reconstructed from incomming updates.

If all of this seem to be too complex to you, I have good news. Infinispan provides one implementation of DeltaAware interface whish is called AtomicHashMap (package org.infinispan.atomic). If this map is used as a value in key/value pairs stored in the cache, only puts/gets/removes performed to this map during a transaction are replicated to other nodes. Classes like Bicycle and BicycleDelta are not need then. Even registering the externalizer for AtomicHashMap is not needed, this is done automatically during registration of internal externalizers. However, one might want a class emulating a real-world object, not just a map. That's the case when your own implementations of DeltaAware and Delta interfaces are the only way.

How to configure Infinispan with transactions, backed by relational DB on JBoss AS 7 vs. Tomcat 7

Migrating  projects from one container to another is often problematic. Not as much with Infinispan. This article is about configuring Infinispan, using Transaction Manager for demarcating transaction boundaries, while  keeping the data both in a memory and relational database - stored via  JDBC cache store. I'll demonstrate all the features on code snippets. 

A complete application is located at https://github.com/mgencur/infinispan-examples and is called carmart-tx-jdbc. It's a web application based on JSF 2, Seam 3 and Infinispan 5.1.4.FINAL, is fully working, tested with JBoss  Application Server 7.1.1.Final and Tomcat 7.0.27. There  is one prerequisite, though. It needs an installed and working MySQL database in your system. The database name should be carmartdb, accessible by a user with carmart/carmart username/password.
 
First, look at what we need to configure for JBoss Application Server 7. 

Configuring transactions and JDBC cache store on JBoss AS 7

Infinispan will be configured via new fluent API using builders, hence the call to  .build() method at the end. We need to configure aspects related to  transactions and cache loaders. The configuration API for cache loaders  is likely going to be changed in not-so-far future. It should be fluent  and more intuitive, generally easier to use than current one. 

I purposely do not show XML configuration. Configuration examples can be found at https://github.com/infinispan/infinispan/blob/master/core/src/main/resources/config-samples/sample.xml. In order to configure transactions and cache loaders, look for tags called  <transaction> and <loaders> and modify that sample file according to below configuration. Tag names and attribute names are very similar for both XML and Java configuration. If that is not enough, there is always a schema in Infinispan distribution.

The configuration of Infinispan is as follows: 

GlobalConfiguration glob = new GlobalConfigurationBuilder()
   .nonClusteredDefault().build();
 
Configuration loc = new ConfigurationBuilder()
   .clustering().cacheMode(CacheMode.LOCAL) 
   .transaction().transactionMode(TransactionMode.TRANSACTIONAL)
   .autoCommit(false)
   .transactionManagerLookup(new GenericTransactionManagerLookup())
   .loaders().passivation(false).preload(false).shared(false)
   .addCacheLoader().cacheLoader(new JdbcStringBasedCacheStore())
   .fetchPersistentState(false).purgeOnStartup(true)
   .addProperty("stringsTableNamePrefix", "carmart_table")
   .addProperty("idColumnName", "ID_COLUMN")
   .addProperty("dataColumnName", "DATA_COLUMN")
   .addProperty("timestampColumnName", "TIMESTAMP_COLUMN")
    //for different DB, use different type
   .addProperty("timestampColumnType", "BIGINT")
   .addProperty("connectionFactoryClass", 
        "org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory")
   .addProperty("connectionUrl", "jdbc:mysql://localhost:3306/carmartdb")
   .addProperty("driverClass", "com.mysql.jdbc.Driver")
    //for different DB, use different type
   .addProperty("idColumnType", "VARCHAR(255)") 
    //for different DB, use different type
   .addProperty("dataColumnType", "VARBINARY(1000)")
   .addProperty("dropTableOnExit", "false")
   .addProperty("createTableOnStart", "true")
   .addProperty("databaseType", "MYSQL")
   .addProperty("datasourceJndiLocation", "java:jboss/datasources/ExampleDS")
   .build();

BasicCacheContainer manager = new DefaultCacheManager(glob, loc, true); 
.... = manager.getCache()
 
Lines marked with red are different in other containers/configurations, as you'll see in a minute. The code above implies that we need to specify proper TransactionManagerLookup implementation which is, in this case, GenericTransactionManagerLookup. We  also need to say: "Hey, I wanna use ManagedConnectionFactory as a connectionFactoryClass". OK, here we go.
I should, as well, explain how to configure a datasource properly, right? In JBoss AS 7, this is configured as a subsystem in
 $JBOSS_HOME/standalone/configuration/standalone.xml:

<subsystem xmlns="urn:jboss:domain:datasources:1.0">
    <datasources>
        <datasource jndi-name="java:jboss/datasources/ExampleDS"    
                    pool-name="ExampleDS" enabled="true" 
                    use-java-context="true">
            <connection-url>jdbc:mysql://localhost:3306/carmartdb</connection-url>
            <driver>mysql-connector-java-5.1.17-bin.jar</driver>
            <security>
                <user-name>carmart</user-name>
                <password>carmart</password>
            </security>
        </datasource>
    </datasources> 
</subsystem>

The usage of transactions is very simple as we can obtain a transaction object by injection.

@Inject 
private javax.transaction.UserTransaction utx;
     
try {
    utx.begin();
    cache.put(...) //store/load keys to/from the cache
    utx.commit(); 
} catch (Exception e) {
    if (utx != null) {
        try {
            utx.rollback();
        } catch (Exception e1) {}
    } 
}
   
Sources: https://github.com/mgencur/infinispan-examples/blob/master/carmart-tx-jdbc/src/jbossas/java/org/infinispan/examples/carmart/session/CarManager.java

Quite easy, isn't it ...if you know how to do it. The only problem is that it does not work (at least not completely) :-) If you deploy the app, you find out that when storing a key-value pair in  the cache, an exception is thrown. This exception indicates that the operation with DB (and JDBC cache store) failed. The exception says:

Error while processing a commit in a two-phase transaction:

org.infinispan.CacheException:

org.infinispan.loaders.CacheLoaderException:

This might be related to https://jira.jboss.org/browse/ISPN-604

A complete stack trace looks similar to https://gist.github.com/2777348
There's still an open issue in JIRA (ISPN-604) and it is being worked on. 

Configuring transactions and JDBC cache store on JBoss AS 7 - c3p0

But how do we cope with this inconvenience for now... By not using a managed datasource but rather a third party library called c3p0 (JDBC3  Connection and Statement Pooling, more information at http://www.mchange.com/projects/c3p0/index.html) Infinispan allows you to use this library for connecting to the database. If you really want to use it, you need to choose a different connectionFactoryClass which is, in this case, PooledConnectionFactory.

Infinispan configuration looks like this:

GlobalConfiguration glob = new GlobalConfigurationBuilder()
    .nonClusteredDefault().build(); 

Configuration loc = new ConfigurationBuilder()
    .clustering().cacheMode(CacheMode.LOCAL)
      .transaction().transactionMode(TransactionMode.TRANSACTIONAL)
    .autoCommit(false)
    .transactionManagerLookup(new GenericTransactionManagerLookup())
    .loaders().passivation(false).preload(false).shared(false)
    .addCacheLoader().cacheLoader(new JdbcStringBasedCacheStore())
    .fetchPersistentState(false).purgeOnStartup(true)
    .addProperty("stringsTableNamePrefix", "carmart_table")
    .addProperty("idColumnName", "ID_COLUMN")
    .addProperty("dataColumnName", "DATA_COLUMN")
    .addProperty("timestampColumnName", "TIMESTAMP_COLUMN")
    .addProperty("timestampColumnType", "BIGINT")
    .addProperty("connectionFactoryClass",
        "org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory")
    .addProperty("connectionUrl", "jdbc:mysql://localhost:3306/carmartdb")
    .addProperty("userName", "carmart")   //we do not have a managed datasource 
     -> specify credentials here
    .addProperty("password", "carmart")
    .addProperty("driverClass", "com.mysql.jdbc.Driver")
    .addProperty("idColumnType", "VARCHAR(255)")
    .addProperty("dataColumnType", "VARBINARY(1000)")
    .addProperty("dropTableOnExit", "false")
    .addProperty("createTableOnStart", "true")
    .addProperty("databaseType", "MYSQL")
    //.addProperty("datasourceJndiLocation", "java:jboss/datasources/ExampleDS") 
    //oh, yes, we do not use JNDI now     
    .build();
 Transactions are accessible in the same way as in the previous use case. Now let's look at configuration for Tomcat servlet container. 

Configuring transactions and JDBC cache store on Tomcat 7

Tomcat does not have any Transaction Manager in it so we have to bundle one with the application. For the purpose of this exercise, we choose JBoss Transactions (http://www.jboss.org/jbosstm). See dependencies at the end.

Cache manager and cache configuration is in this form:

GlobalConfiguration glob = new GlobalConfigurationBuilder()
    .nonClusteredDefault().build();

Configuration loc = new ConfigurationBuilder()
    .clustering().cacheMode(CacheMode.LOCAL) 
    .transaction().transactionMode(TransactionMode.TRANSACTIONAL)
    .autoCommit(false)
    .transactionManagerLookup(new JBossStandaloneJTAManagerLookup()) 
    .loaders().passivation(false).preload(false).shared(false)
    .addCacheLoader().cacheLoader(new JdbcStringBasedCacheStore())
    .fetchPersistentState(false).purgeOnStartup(true)
    .addProperty("stringsTableNamePrefix", "carmart_table")
    .addProperty("idColumnName", "ID_COLUMN")
    .addProperty("dataColumnName", "DATA_COLUMN")
    .addProperty("timestampColumnName", "TIMESTAMP_COLUMN")
    .addProperty("timestampColumnType", "BIGINT")
    .addProperty("connectionFactoryClass", 
        "org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory")
    .addProperty("connectionUrl", "jdbc:mysql://localhost:3306/carmartdb")
    .addProperty("userName", "carmart")
    .addProperty("driverClass", "com.mysql.jdbc.Driver")
    .addProperty("idColumnType", "VARCHAR(255)")
    .addProperty("dataColumnType", "VARBINARY(1000)")
    .addProperty("dropTableOnExit", "false")
    .addProperty("createTableOnStart", "true")
    .addProperty("databaseType", "MYSQL")
    .addProperty("datasourceJndiLocation", "java:comp/env/jdbc/ExampleDB")
    .build();
       
For Tomcat, we need to specify a different transactionManagerLookup implementation and datasourceJndiLocation. Tomcat simply places objects  under a bit different JNDI locations. The datasource is defined in context.xml file which has to be on classpath. This file might look like this:

<Context>
    <Resource name="jdbc/ExampleDB"
        auth="Container"
        type="javax.sql.DataSource"
        factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
        maxActive="100"
        minIdle="30"
        maxWait="10000"
        jmxEnabled="true"
        username="carmart"
        password="carmart"
        driverClassName="com.mysql.jdbc.Driver"
        url="jdbc:mysql://localhost:3306/carmartdb"/> 
</Context>

How do we get the transaction manager in the application then? Lets obtain  it directly from a cache. 

Infinispan knows how to find the manager and we need to know how to obtain it from Infinispan.

javax.transaction.TransactionManager tm = 
    ((CacheImpl) myCache).getAdvancedCache().getTransactionManager(); 
try {
    tm.begin();
    myCache.put(...)
    tm.commit(); 
} catch (Exception e) {
    if (tm != null) {
        try {
            tm.rollback();
        } catch (Exception e1) {}
    } 
}
   
Sources: https://github.com/mgencur/infinispan-examples/blob/master/carmart-tx-jdbc/src/tomcat/java/org/infinispan/examples/carmart/session/CarManager.java The transaction manager provides standard methods for transactions, such as begin(), commit() and rollback(). 

Now is the time for dependencies.

So...which dependencies do we always need when using Infinispan with JDBC cache stores and transactions? These are infinspan-core, infinispan-cachestore-jdbc and javax.transaction.jta. The scope for jta dependency, as defined in Maven, is different for JBossAS and Tomcat.

Common dependencies for JBossAS and Tomcat

    <dependency>
        <groupId>org.infinispan</groupId>
        <artifactId>infinispan-core</artifactId>
        <version>5.1.4.FINAL</version>
    </dependency>
   
    <dependency>
        <groupId>org.infinispan</groupId>
        <artifactId>infinispan-cachestore-jdbc</artifactId>
        <version>5.1.4.FINAL</version>
    </dependency>

Of course, our application needs a few more dependencies but these are not directly related to Infinispan. Let's ignore them in this article. JBoss AS 7 provides managed datasource that is accessible from Infinispan. The only specific dependency (related to transactions or Infinispan) is JTA.

Dependencies specific to JBossAS - using managed Datasource (managed by the server)

    <dependency>
        <groupId>javax.transaction</groupId>
        <artifactId>jta</artifactId>
        <version>1.1</version>
        <scope>provided</scope>
    </dependency>

Dependencies specific to JBossAS - using c3p0

    <dependency>
        <groupId>javax.transaction</groupId>
        <artifactId>jta</artifactId>
        <version>1.1</version>
        <scope>provided</scope>
    </dependency>
   
    <dependency>
        <groupId>c3p0</groupId>
        <artifactId>c3p0</artifactId>
        <version>0.9.1.2</version>
    </dependency>
   
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.17</version>
    </dependency>
   
Yes, you need to bundle also MySQL connector. On the other hand, for Tomcat use case and JBossAS with managed datasource, this jar file needs do be deployed to the server separately. For Tomcat, do this simply by copying the jar file to $TOMCAT_HOME/lib.  For JBoss AS 7, copy the jar file into $JBOSS_HOME/standalone/deployments.

Dependencies specific to Tomcat - using JBoss Transactions

    <dependency>
        <groupId>javax.transaction</groupId>
        <artifactId>jta</artifactId>
        <version>1.1</version>
        <scope>runtime</scope>
    </dependency>
   
    <dependency>
        <groupId>org.jboss.jbossts</groupId>
        <artifactId>jbossjta</artifactId>
        <version>4.16.4.Final</version>
    </dependency>

That's it. I hope you've found this article helpful. Any feedback is welcome, especially the positive one :-) If you find any problem with the  application, feel free to comment here or participate in Infinispan forums (http://www.jboss.org/infinispan/forums).