Welcome!

Linux Authors: Elizabeth White, Liz McMillan, Jason Bloomberg, Roger Strukhoff, Pat Romanski

Related Topics: Open Source, Linux

Open Source: Article

Strengthening Open Source's Weakest Link: Software Testing

Broader, more collaborative open testing can yield more meaningful results for business and developers alike

Murugan Pal (pictured), founder and CTO of SpikeSource, writes: Pop quiz: If one open source user tests 30 percent of an application, and another tests 20 percent, how much of the application has been tested?

The answer is probably closer to 30 percent than 50 percent, since both users probably focused on common functions like start-up, shutdown, and data access. The problem gets amplified if the application is built for n-tier deployment based on service-oriented architecture. newspapers and broadcasters. The service handles between 150,000 and 500,000 pages of content per affiliate per day, supporting 11,000 concurrent users. MySQL, a free open source database, has been the backbone of AP Hosted News since 2002.

Everyone knows that the cornerstone of open source software is the free availability of its source code, which lets developers and users around the world contribute to it and improve it. The software naturally becomes stronger as it accumulates improvements and sheds imperfections. The quality improves based on more usage and reviews.

But the model breaks down when it comes to making sure the software actually works in real-world deployment scenarios. The power of participation has been confined almost entirely to the development phase of the software life cycle. Testing remains open source's weakest link as it is difficult to reproduce all intended usages.

While code repositories and other shared resources help developers revise and build upon the efforts of their peers, the testing of that software has remained an uncoordinated, isolated affair. Instead of learning from and enhancing each other's tests, users and developers test the same functions and routines, and have no way to easily share their results with each other.

Because most testers are testing only on the platform they happen to be using, most test results aren't widely applicable. Results that show how well a piece of software works on a particular platform might not say much about how well it works with a different operating system, or how well it interacts with other software components. That's a major shortcoming, since open source software components are mostly used as part of a stack with other components.

A Moving Target
The constantly changing nature of many open source programs makes meaningful results even harder to come by. To get results that would be accurate and meaningful to a broad section of the open source community, a user would have to constantly retest it on an ever-growing number of platforms (which are also changing).

As a result, some of the biggest challenges of open source software have remained intact. "Dependency hell," (Jar Wars and DLL Hell) in which each piece of software relies on a specific version of another piece of software, continues to be a constant time drain for many IT departments. Not only are the dependencies difficult to resolve, some times you end up with redundant footprints of the same libraries embedded in the integrated runtime (e.g., Log4J in Tomcat, Struts, etc.).

What if the open source development model - the "architecture of participation," to use Tim O'Reilly's phrase - could be extended to software testing? If users could easily access, build upon, and contribute to a growing body of open source tests, testing could become an extension of the participatory development process. Fixes could be validated faster, and functionality and backward compatibility across different versions of integrated software components would be easier. Even enterprise customers can participate in this model by validating their tests on integrated hardware and software runtime environments.

Participatory testing would also help certify the interoperability of the exponentially increasing combinations of component choices. Most businesses use open source software not in isolation, but in stacks of interoperating components. Tests should be able to tell you exactly how well those components work together.

Just as the participatory development community relies on open resources and information repositories for source code, the participatory testing community should have open resources for testing -including open tests, test manifests, test results, and interfaces/protocols to share test results and tests.

That's where SpikeSource want to be a catalyst in promoting participatory testing. We provide all of those resources, as well as an environment for open source software users and developers to share, obtain, and exchange federated information about open source testing.

Businesses can upload an application to SpikeSource's open testing tool, which continuously pulls code from open source repositories and builds its own repository of different versions of different components and operating systems. SpikeSource automatically constructs systems out of those repositories, provisions them on virtualized runtime environments, tests them, and records the results.

Interoperability Is Key
For most businesses, how well a component works with others is just as important as its independent functionality. For example, if there's a change to the Apache servlet engine Tomcat, a test should show how those changes impact other components like JK2, ConnectorJ or MySQL.

The ultimate goal is to make open source software more scalable and predictable, so that business can use it in conjunction with or as an alternative to proprietary software. Our goal at SpikeSource is to help further the enterprise adoption of open source software. Through testing, it becomes more reliable, easy, and safe for enterprises to deploy.

More Stories By Murugan Pal

Murugan Pal is founder and CTO of SpikeSource.

Comments (3) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Jonathan Bruce's Web Log 09/19/05 01:06:38 PM EDT

Trackback Added: Open source components fall short of the quality metrics?; Sys-Con ran an article on September 16th, by Murugan Pal, CTO of SpikeSource.com Strengthening Open Source's Weakest Link: Software Testing — Pop quiz: If one open source user tests 30 percent of an application, and another tests 20 percent, how...

David Tomlinson 09/12/05 06:38:50 PM EDT

First, the writer tells us that the biggest problem is testing. Then that the problem with testing is that everyone tests the same thing. They he tells us that everybody tests on their own platform, so my test results will mean nothing to you. Then he talks about jar wars and dependency hell. Well, which is it? Is the biggest problem with open source that the writer's company doesn't have enough of our business? Have I ever seen a more obvious commercial for a vendor? Of course. Do I want to see more? Hell, no!

The only reason for writing this article is to get his company's name out there and get a few hits on their website. I gotta admit they got a hit from me, but it's the last one.

Pointless article.

Enterprise Open Source Magazine 09/12/05 09:52:44 AM EDT

Strengthening Open Source's Weakest Link. Pop quiz: If one open source user tests 30 percent of an application, and another tests 20 percent, how much of the application has been tested? The answer is probably closer to 30 percent than 50 percent, since both users probably focused on common functions like start-up, shutdown, and data access.