Welcome!

Linux Containers Authors: Liz McMillan, Elizabeth White, Yeshim Deniz, Zakia Bouachraoui, Carmen Gonzalez

Related Topics: Linux Containers

Linux Containers: Article

Comparing Apache Tomcat Performance Across Platforms

Part 1: Performance and distinct error handling under memory load

We have measured performance information to distinguish the differences between the Windows and Linux platforms. Given comparable hardware we found the performance differences almost trivial.

When the server was pressed to capacity, our Windows installation was forced to turn away some traffic with minimal alteration in service performance, while our Linux installation elected to service nearly all the connections at the cost of introducing latency. However, before hitting capacity, our Linux server appeared on average to be capable of servicing connections at a slightly faster rate than our Windows server.

Sun's release of its J2EE specifications has been followed by enthusiastic developers crafting powerful applications that can be seen on the Web in businesses of every size. The result is one very hot topic continually focused on by many developer groups, but mature enough to be well known in all aspects of the industry. Some of the first questions that must be asked with any new project remain: what is the best starting point, and how should this application be deployed? Here, we'd like to provide some guidance as to what trade-offs can be expected before that decision is made.

Earlier editions of this report presented some comparative results between different servlet containers. The response and feedback have been tremendous, and there's been no shortage of suggestions for comparing function X against Y and Z. However, with J2EE delivering platform-independence, many people (including myself) have gotten interested in determining just how significant a difference there is between the platforms. So we'll look at one of the most popular servlet containers, Apache Tomcat, operating on the two most common platforms in general use, Windows and Linux.

For simplicity's sake, this article will be broken into two parts. Here, we'll discuss a simple scenario that leads the servers quickly into a performance limitation. In Part 2, we'll examine the performance of the platforms after this limitation is lifted.

Testing Goals
Usage Scenario
We're interested in evaluating the servlet performance of a standalone server. It is expected to be applicable to small and medium-sized projects that deploy applications in departmental or small corporate environments. It's not intended to represent server performance in a clustered or distributed environment. Nor is it expected to reflect the performance of a server that's been highly optimized for maximum performance on a particular application.

Difficulties of Benchmarking
Our choice of benchmarks is merely a smoothed-out average of samples that we have observed in the wild. Each application stresses different aspects, or users have different expectations. Some applications will run better on some servers than others. Well aware of the danger of misleading benchmarks, we don't expect to reflect the performance of any particular application or class of applications that might be deployed in the real world. We merely hope to provide the data that helps the reader weigh the performance trade-offs of the platforms.

Remember too that numerical values have a margin of error. When we compare timing measurements that are nearly equal, the faster and slower responses can easily exchange places with the slightest fluctuation. Upsets can derive from practically any cause imaginable. One source, for instance, might be whether software configurations include or exclude background/idle processes. Minor revision changes in loaded applications/libraries can have the same effect. On the hardware side, the next faster or slower processor can affect the processor-to-IO performance ratio, which can upset subtleties in how the platform manages internal semantics.

Raw Numbers vs Relative Performance
Due to variations in applications and hardware configurations, we can't tell you how many users your particular application will be able to handle in your environment (which is why we sell a product for exactly that purpose). We've got numbers such as hits/sec and average page duration, but raw numbers are of little value by themselves. Only when statistics are compared against other servers do they provide valuable insight on which to make performance judgments.

It may seem obvious to some, but it's important to reinforce this point: If the data indicates that Server X can serve 200 hits/sec with an average duration of under two seconds, it doesn't correlate to the number of hits/sec that your application will be capable of. The data presented here for one server is only useful for comparison against another under the identical test conditions of our test lab. Also note that the hardware used for the server in these tests is not a state-of-the-art configuration.

Reproducible Results
No matter how the testing is done, someone will cry "Foul Play!" A primary goal of this effort is a test that's easily duplicated by anyone else who wants to try. No doubt the exact numbers won't be duplicable due to differences in hardware and measurement technique, but the relative differences between the servers should be reproducible.

Configuration
Philosophy
Ideally we'd like to give the community some insight into the relative differences between the platforms so one might be able to infer some information about his own application. So we've intentionally used any default values provided except where noted. We feel that while performance tuning will inevitably become a part of an app that's not meeting expectations, our testing strategy provides an effective baseline guide.

By using default settings, our test results can be interpreted more broadly than had we tuned our application server specifically for our test app. And then too there's practically an infinite number of permutations of options related to server and servlet container's functionality. Evaluating permutations of even a few of these settings can make our results too long to digest. Third, if the same functionality can be obtained by setting twice as many options (hence taking twice as much time to configure), the question of testing fairness could be argued.

Server Software
The two operating systems used in this test were Microsoft Windows Server 2003 and CentOS 4.2 x86_64. Windows Update was used to install the most recent service packs as of November 16, 2005. Our copy of Linux was installed via CDs burned from the publicly available ISO images. The YUM Updater was then used to update all installed packages to the latest stable versions as of November 15, 2005.

Server Hardware
Two servers were used, each with identical hardware. Each was a Dell PowerEdge SC1420 (see Table 1).

Testing Laboratory
Each workstation used in the test was connected directly to a Dell PowerConnect 2324 switch. This switch offers two GbE ports to which the servers were connected as well as 24 100MbE ports.

During each test the server not being tested acted as the test controller. The load controller used five additional workstations for additional load generation to minimize timing discrepancies. Each load-generating workstation was connected directly to the same switch through one of the 100MbE ports.

The Tests
User Scenarios
Despite the fact that any performance test is only an approximation of real-world use, it was important to choose user scenarios that are similar to the kind of use typical of production deployments. The server statistics for our own Web site provided the inspiration. After a detailed analysis of the paths and pages traversed on our Web site, a few common Web sites (Yahoo, Amazon, etc.) were also analyzed (see Table 2).

User Distribution
Based on the distribution observed in our server statistics, the distribution of the scenarios was chosen (below, middle). During the simulation, each virtual (simulated) user will execute a single scenario repeatedly until the end of the test. After compensating for the differences in the length of each scenario, the final user distribution could be calculated (see Table 3).

Bandwidth Distribution
Public Web sites and intranet applications see distinctively different distributions of user bandwidth. For public Web sites 56k to 4Mbit connections are typical. For intranet applications 10Mbit-100Mbit is common (this bandwidth is shared). So a maximum 10Mbit bandwidth per user was selected for simulation purposes. The bandwidth was limited for each virtual user by the testing tool.

Note that with a bandwidth per user of 10Mbit on a 100Mbit network, no more than 10 users could be supported using their full bandwidth (assuming 100% network efficiency, which Ethernet can't achieve). However, all the scenarios contain significant "think time" (time between pages) that allows more than 1,000 users to use the 100Mbit bandwidth. Each test was stopped before any indications were given that the throughput capacity had been reach to ensure that our measurements were an accurate gauge of the server's performance.

Construction of Test Cases
The test case required a servlet that could return Web pages of a specified length referencing a specified number of external resources (images were used in this simulation). The servlet used provided the required functionality hard-coded in the servlet. Its source code is publicly available in the ContainerBenchmark.java file. Once the servlet and necessary resources were installed (via a WAR file), the test tool was used to record the scenarios interactively using the Web browser. In this case Opera 8.01 was used, but which browser is used to record the scenarios should not affect the test. Each scenario was recorded over the duration listed above. The testing tool was configured to simulate an approximately equal share of "think time" between each page, lingering slightly longer on the last page before restarting the test. (See Figure 1-2-3)

Testing Procedure
Test case recordings, virtual user simulation, and data gathering were all managed by the testing tool in this case Web Performance Trainer 2.8, Build 629.

Servlet Container Preparation
In our simple test environment, all operations were carried out while logged in with full administrative privileges. No further security restrictions were manually applied to the servlet's operating environment.

The first required component to be installed was the JRE. Sun's JRE 1.5.0 - update 5 was used and installed on each platform with the executable installer.

After the JVM was installed, Tomcat 5.5.12 was installed. The Windows download included another executable installer that automated the process. The Linux installation was done by simply unpacking the compressed archive then setting the necessary CATALINA_HOME and JAVA_HOME variables from the command line before using the shell scripts included with the code to start the container.

Tomcat 5.5.12 supports linkages to the APR library for acceleration, but for simplicity's sake, both servers were installed and configured with this option disabled. The testing was done by connecting directly to Tomcat and not through any additional software connector or proxy.

Once the servlet container was running successfully, the WAR file with our test application was copied to Tomcat's Web apps folder and auto-deployed.

Record the Test
Once the first servlet container was installed, it was possible to record one case of a user using his Web browser to step through each page of the test. Once the recorded, the test was configured as mentioned above.


More Stories By Frank Ziglar

Frank C. Ziglar works at Web Performance, Inc.

Comments (2)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


IoT & Smart Cities Stories
As you know, enterprise IT conversation over the past year have often centered upon the open-source Kubernetes container orchestration system. In fact, Kubernetes has emerged as the key technology -- and even primary platform -- of cloud migrations for a wide variety of organizations. Kubernetes is critical to forward-looking enterprises that continue to push their IT infrastructures toward maximum functionality, scalability, and flexibility. As they do so, IT professionals are also embr...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
AI and machine learning disruption for Enterprises started happening in the areas such as IT operations management (ITOPs) and Cloud management and SaaS apps. In 2019 CIOs will see disruptive solutions for Cloud & Devops, AI/ML driven IT Ops and Cloud Ops. Customers want AI-driven multi-cloud operations for monitoring, detection, prevention of disruptions. Disruptions cause revenue loss, unhappy users, impacts brand reputation etc.
As you know, enterprise IT conversation over the past year have often centered upon the open-source Kubernetes container orchestration system. In fact, Kubernetes has emerged as the key technology -- and even primary platform -- of cloud migrations for a wide variety of organizations. Kubernetes is critical to forward-looking enterprises that continue to push their IT infrastructures toward maximum functionality, scalability, and flexibility.
The Japan External Trade Organization (JETRO) is a non-profit organization that provides business support services to companies expanding to Japan. With the support of JETRO's dedicated staff, clients can incorporate their business; receive visa, immigration, and HR support; find dedicated office space; identify local government subsidies; get tailored market studies; and more.
Today's workforce is trading their cubicles and corporate desktops in favor of an any-location, any-device work style. And as digital natives make up more and more of the modern workforce, the appetite for user-friendly, cloud-based services grows. The center of work is shifting to the user and to the cloud. But managing a proliferation of SaaS, web, and mobile apps running on any number of clouds and devices is unwieldy and increases security risks. Steve Wilson, Citrix Vice President of Cloud,...
When Enterprises started adopting Hadoop-based Big Data environments over the last ten years, they were mainly on-premise deployments. Organizations would spin up and manage large Hadoop clusters, where they would funnel exabytes or petabytes of unstructured data.However, over the last few years the economics of maintaining this enormous infrastructure compared with the elastic scalability of viable cloud options has changed this equation. The growth of cloud storage, cloud-managed big data e...
Artificial intelligence, machine learning, neural networks. We're in the midst of a wave of excitement around AI such as hasn't been seen for a few decades. But those previous periods of inflated expectations led to troughs of disappointment. This time is (mostly) different. Applications of AI such as predictive analytics are already decreasing costs and improving reliability of industrial machinery. Pattern recognition can equal or exceed the ability of human experts in some domains. It's devel...