Page 1 of 2   next page »

We have measured performance information to distinguish the differences between the Windows and Linux platforms. Given comparable hardware we found the performance differences almost trivial.

When the server was pressed to capacity, our Windows installation was forced to turn away some traffic with minimal alteration in service performance, while our Linux installation elected to service nearly all the connections at the cost of introducing latency. However, before hitting capacity, our Linux server appeared on average to be capable of servicing connections at a slightly faster rate than our Windows server.

Sun's release of its J2EE specifications has been followed by enthusiastic developers crafting powerful applications that can be seen on the Web in businesses of every size. The result is one very hot topic continually focused on by many developer groups, but mature enough to be well known in all aspects of the industry. Some of the first questions that must be asked with any new project remain: what is the best starting point, and how should this application be deployed? Here, we'd like to provide some guidance as to what trade-offs can be expected before that decision is made.

Earlier editions of this report presented some comparative results between different servlet containers. The response and feedback have been tremendous, and there's been no shortage of suggestions for comparing function X against Y and Z. However, with J2EE delivering platform-independence, many people (including myself) have gotten interested in determining just how significant a difference there is between the platforms. So we'll look at one of the most popular servlet containers, Apache Tomcat, operating on the two most common platforms in general use, Windows and Linux.

For simplicity's sake, this article will be broken into two parts. Here, we'll discuss a simple scenario that leads the servers quickly into a performance limitation. In Part 2, we'll examine the performance of the platforms after this limitation is lifted.

Testing Goals
Usage Scenario
We're interested in evaluating the servlet performance of a standalone server. It is expected to be applicable to small and medium-sized projects that deploy applications in departmental or small corporate environments. It's not intended to represent server performance in a clustered or distributed environment. Nor is it expected to reflect the performance of a server that's been highly optimized for maximum performance on a particular application.

Difficulties of Benchmarking
Our choice of benchmarks is merely a smoothed-out average of samples that we have observed in the wild. Each application stresses different aspects, or users have different expectations. Some applications will run better on some servers than others. Well aware of the danger of misleading benchmarks, we don't expect to reflect the performance of any particular application or class of applications that might be deployed in the real world. We merely hope to provide the data that helps the reader weigh the performance trade-offs of the platforms.

Remember too that numerical values have a margin of error. When we compare timing measurements that are nearly equal, the faster and slower responses can easily exchange places with the slightest fluctuation. Upsets can derive from practically any cause imaginable. One source, for instance, might be whether software configurations include or exclude background/idle processes. Minor revision changes in loaded applications/libraries can have the same effect. On the hardware side, the next faster or slower processor can affect the processor-to-IO performance ratio, which can upset subtleties in how the platform manages internal semantics.

Raw Numbers vs Relative Performance
Due to variations in applications and hardware configurations, we can't tell you how many users your particular application will be able to handle in your environment (which is why we sell a product for exactly that purpose). We've got numbers such as hits/sec and average page duration, but raw numbers are of little value by themselves. Only when statistics are compared against other servers do they provide valuable insight on which to make performance judgments.

It may seem obvious to some, but it's important to reinforce this point: If the data indicates that Server X can serve 200 hits/sec with an average duration of under two seconds, it doesn't correlate to the number of hits/sec that your application will be capable of. The data presented here for one server is only useful for comparison against another under the identical test conditions of our test lab. Also note that the hardware used for the server in these tests is not a state-of-the-art configuration.

Reproducible Results
No matter how the testing is done, someone will cry "Foul Play!" A primary goal of this effort is a test that's easily duplicated by anyone else who wants to try. No doubt the exact numbers won't be duplicable due to differences in hardware and measurement technique, but the relative differences between the servers should be reproducible.

Configuration
Philosophy
Ideally we'd like to give the community some insight into the relative differences between the platforms so one might be able to infer some information about his own application. So we've intentionally used any default values provided except where noted. We feel that while performance tuning will inevitably become a part of an app that's not meeting expectations, our testing strategy provides an effective baseline guide.

By using default settings, our test results can be interpreted more broadly than had we tuned our application server specifically for our test app. And then too there's practically an infinite number of permutations of options related to server and servlet container's functionality. Evaluating permutations of even a few of these settings can make our results too long to digest. Third, if the same functionality can be obtained by setting twice as many options (hence taking twice as much time to configure), the question of testing fairness could be argued.

Server Software
The two operating systems used in this test were Microsoft Windows Server 2003 and CentOS 4.2 x86_64. Windows Update was used to install the most recent service packs as of November 16, 2005. Our copy of Linux was installed via CDs burned from the publicly available ISO images. The YUM Updater was then used to update all installed packages to the latest stable versions as of November 15, 2005.

Server Hardware
Two servers were used, each with identical hardware. Each was a Dell PowerEdge SC1420 (see Table 1).

Testing Laboratory
Each workstation used in the test was connected directly to a Dell PowerConnect 2324 switch. This switch offers two GbE ports to which the servers were connected as well as 24 100MbE ports.

During each test the server not being tested acted as the test controller. The load controller used five additional workstations for additional load generation to minimize timing discrepancies. Each load-generating workstation was connected directly to the same switch through one of the 100MbE ports.

The Tests
User Scenarios
Despite the fact that any performance test is only an approximation of real-world use, it was important to choose user scenarios that are similar to the kind of use typical of production deployments. The server statistics for our own Web site provided the inspiration. After a detailed analysis of the paths and pages traversed on our Web site, a few common Web sites (Yahoo, Amazon, etc.) were also analyzed (see Table 2).

User Distribution
Based on the distribution observed in our server statistics, the distribution of the scenarios was chosen (below, middle). During the simulation, each virtual (simulated) user will execute a single scenario repeatedly until the end of the test. After compensating for the differences in the length of each scenario, the final user distribution could be calculated (see Table 3).

Bandwidth Distribution
Public Web sites and intranet applications see distinctively different distributions of user bandwidth. For public Web sites 56k to 4Mbit connections are typical. For intranet applications 10Mbit-100Mbit is common (this bandwidth is shared). So a maximum 10Mbit bandwidth per user was selected for simulation purposes. The bandwidth was limited for each virtual user by the testing tool.

Note that with a bandwidth per user of 10Mbit on a 100Mbit network, no more than 10 users could be supported using their full bandwidth (assuming 100% network efficiency, which Ethernet can't achieve). However, all the scenarios contain significant "think time" (time between pages) that allows more than 1,000 users to use the 100Mbit bandwidth. Each test was stopped before any indications were given that the throughput capacity had been reach to ensure that our measurements were an accurate gauge of the server's performance.

Construction of Test Cases
The test case required a servlet that could return Web pages of a specified length referencing a specified number of external resources (images were used in this simulation). The servlet used provided the required functionality hard-coded in the servlet. Its source code is publicly available in the ContainerBenchmark.java file. Once the servlet and necessary resources were installed (via a WAR file), the test tool was used to record the scenarios interactively using the Web browser. In this case Opera 8.01 was used, but which browser is used to record the scenarios should not affect the test. Each scenario was recorded over the duration listed above. The testing tool was configured to simulate an approximately equal share of "think time" between each page, lingering slightly longer on the last page before restarting the test. (See Figure 1-2-3)

Testing Procedure
Test case recordings, virtual user simulation, and data gathering were all managed by the testing tool in this case Web Performance Trainer 2.8, Build 629.

Servlet Container Preparation
In our simple test environment, all operations were carried out while logged in with full administrative privileges. No further security restrictions were manually applied to the servlet's operating environment.

The first required component to be installed was the JRE. Sun's JRE 1.5.0 - update 5 was used and installed on each platform with the executable installer.

After the JVM was installed, Tomcat 5.5.12 was installed. The Windows download included another executable installer that automated the process. The Linux installation was done by simply unpacking the compressed archive then setting the necessary CATALINA_HOME and JAVA_HOME variables from the command line before using the shell scripts included with the code to start the container.

Tomcat 5.5.12 supports linkages to the APR library for acceleration, but for simplicity's sake, both servers were installed and configured with this option disabled. The testing was done by connecting directly to Tomcat and not through any additional software connector or proxy.

Once the servlet container was running successfully, the WAR file with our test application was copied to Tomcat's Web apps folder and auto-deployed.

Record the Test
Once the first servlet container was installed, it was possible to record one case of a user using his Web browser to step through each page of the test. Once the recorded, the test was configured as mentioned above.

Page 1 of 2   next page »