This performance benchmark report will cover three separate areas of performance related to JBoss Remoting. Section 3 (All transports - speed results will cover basic performance when making synchronous invocations from a single client to a server and will include results for (1) all of the JBoss Remoting transports, (2) two raw Java transports, and (3) two Spring Remoting transports.^[1] Section 4 (All JBoss Remoting transports - all invocation types) will cover performance differences among the various JBoss Remoting transports when making different types of client to server invocations (i.e. synchronous and asynchronous). Section 5 (All transports - scalability) will cover how well the JBoss Remoting transports, raw Java transports, and Spring Remoting transports scale as more clients are added.

This report is NOT intended to indicate how fast different transports are in absolute terms, but to compare speed and scalability of transports relative to one another. The tests were run on laptops and the results might not be indicative of performance levels achievable in a production environment. However, since all tests in each group were run on the same hardware, using the same environment and test scenario, the results should be sufficient for side by side comparisons. All the test result data are included at the end of this report along with instructions on how to reproduce the test runs in any other environment.

Since there is a limited amount of space within the graphs for full descriptions of the transports and test scenarios, the abbreviations are explained here.

In the first performance test a single client continuously makes synchronous invocations on a single server. This test shows how quickly the transport can make remote invocations when not under load (basically showing raw data transfer speed). Three test runs were executed for each transport, and the averages are given in the following graph.

The graph shows that the JBoss Remoting socket transport with JBoss Serialization is able to make the largest number of requests per second, followed by the raw rmi and raw socket transports (which use the Java API directly and not a remoting framework).

The second performance test also consists of a single client making invocations on a single server, but the test runs span two dimensions: transport and invocation type. The transports exercised in this case are the JBoss Remoting transports, and the invocation types are the following:

The following graph indicates how long it took each JBoss Remoting transport to make 100,000 client to server invocations for each invocation type. The lower the time, the faster the combination of transport and invocation type.

The results show that the JBoss Remoting socket transport with JBoss Serialization was the fastest transport over all the different invocation types.

The final performance test is designed to indicate how well the various transports scale in response to increased load from additional clients. Each client uses a single thread to make 100,000 synchronous invocations on the server. The following graphs indicate the total number of invocations per second processed by the server as it services up to thirty clients. The numbers are averaged over three runs. For the sake of readability the results are partitioned by serialization type. ^[2][3]

Several phenomena may be observed in these graphs.

All the performance tests run for this report have the same basic structure and most share the same base code (the most notable exception being the Spring Remoting http transport, which requires a web container for server side deployment). All tests have one or more clients, each of which starts by creating a callback server and then sending the total number of invocations it intends to make. The server will then create a call tracker that will manage the test invocations as they come in from that client, ensuring that all intended invocations are received without duplicates. The server will also create a client to call back on the callback server created within that client instance.

Once the client and server have been fully initialized with their initial configuration data, the client will begin its loop to make test invocations. The number of test invocations is configurable, but for this report, the clients each make 100,000 test invocations. The payload size for each invocation is configurable as well, but for this report will be approximately 1024 bytes (plus size of payload wrapper for each transport). The time it takes the client to make all its test invocations is the main result captured for this report. When the call tracker on the server side has received all the invocations from its associated client, it will call back on that client (actually the client's callback server) and send the number of test invocations it received and number of duplicate test invocations, if any. This will indicate test run completion from the client's perspective.

All the transport tests (except the Spring Remoting http transport) are executed using the JBoss JRunit test framework. This means that each test run can be executed within ant as a junit task, which will call on a JRunit test driver which will spawn one or more client test instances and a server test instance. All configuration information (as seen below) can be modified within the JBoss Remoting build.xml file and tests can be executed via running the JBoss Remoting ant script. All the performance test source code can be found within the org.jboss.test.remoting.performance package of the JBoss Remoting project. Everything needed to run all the transport tests are included within the JBoss Remoting project (except a web container, which is needed for the Spring Remoting http transport tests).

The test result data including all the junit result files (in xml format) and the MS Excel spreadsheet used to capture raw result data, calculate averages, and generate graphs can be found within test_results.zip.