One of the problems with open source software is that it isn't controlled or monitored. It's all very well for us Linux types to sit here and say, "Open source software is better," but how do we quantify and qualify that statement with hard evidence that will convince our customers, users, and most important, management?

There's a company out there that is testing open source software and then comparing the results with those from commercial software from their clients. Reasoning Inc. (www.reasoning.com) provides automated software inspection (ASI) services that study the code in detail and determine potential faults and failures. When used as part of your development cycle ASI can help identify problems that can be fixed early on in the development process.

In February 2003 they studied the Linux TCP/IP stack as compared to commercial TCP/IP stack solutions, and it showed that open source development of the stack showed fewer defects than the commercial model. In July 2003 Reasoning published the results of comparing an early prerelease version of Apache 2.1 to commercial solutions at a similar stage in the development life cycle. What was significant in both investigations is that both the open source and commercial solutions showed similar defects at this early stage.

MySQL Investigations
In December 2003 it was MySQL's turn. MySQL is the cornerstone of any LAMP (Linux, Apache, MySQL, Perl/Python/PHP) project and forms a considerable component of many applications both internal and Web-based. Reasoning compared MySQL 4.0.16 with the results from their tests on commercial RDBMS software.

MySQL is made up of over 235,000 lines of source code - excluding all header files and comments - spread over 517 files. Reasoning's tests are split up into a number of different "defect classes," including:

• Memory leaks
• NULL pointer dereferences (NPDs)
• Bad de-allocations
• Out of bounds array accesses
• Uninitialized variables

The Results
The result of all the tests was a defect count of 21 - that's a defect density (measured in Thousand of Lines of Code [KLOC]) of 0.09. This rate is somewhat below the average for the industry, which hits an average of 0.57 defects per KLOC. That's across a total of more than 35 million lines of code (see Figure 1).

The spread of defects in commercial software actually ranges from below 0.36 defects per KLOC up to over 0.71 defects per KLOC. These figures are based on the 200 most recent inspections, so you can see the significance of such a low count. You can get a better idea of the comparative defects between the industry average and MySQL by looking at the chart.

When the MySQL development team was provided with these results they immediately fixed thirteen of the faults, and identified eight that would be unlikely to manifest themselves during normal usage.

What It Means
So what does all this really mean?

First it means we have some decent data to back up what most of us already know - MySQL is a good, stable product that's good enough not only to support the development of an RDBMS-based project but also to support it in a production environment.

More significantly, the results also show that open source software generally is less bug-ridden than commercial equivalents. In fact, if you look at the test results in combination with the past studies by Reasoning you can pick out three main points:

1. All projects, open source or commercial, start off with a similar level of bugs.
2. Open source projects generally have their bugs fixed faster, and fixes are incorporated into the project much earlier in the project life cycle.
3. Open source projects end up with fewer bugs in the released product as compared to their commercial equivalents.

The reasons for this probably come from the massive community effort to help find and fix the bugs. Your average commercial development will have 10-20 people, or perhaps as many as 100 working on a single project. In an open source environment the number of key developers will be similar, but the number of ancillary developers who provide bug fixes is much higher, and the number who test and find the problems can be higher still.

In fact, Reasoning suggests that the reason the differences are so significant may be entirely based around the development and release model of open source software. In particular they mention that open source projects often obtain more bug reports - and fixes if the reporter is able - because it's so easy for users to identify the problem area and fix it. Also, the way in which many developers get to cast an eye over the code, and therefore pick up on other people's bugs and typos, also helps to improve the quality of the software.

Based on their findings, and the experience of those of us at the front lines of the development and use of open source software, most of us would probably agree.

The upshot? The quality of open source software is better - at least from a bug and defect perspective.

You can read the full details on all of Reasoning's reports at www.reasoning.com/downloads.html.