Welcome!

Linux Containers Authors: Derek Weeks, Elizabeth White, Liz McMillan, Gordon Haff, Carmen Gonzalez

Related Topics: Linux Containers

Linux Containers: Article

The kernel of pain

Let's call a spade a spade: For large servers, the 2.4 kernel has been a disaster.

(LinuxWorld) -- Let's start from the beginning. In July 2001, I was responsible for upgrading a customer's server from Red Hat 6.2 to Mandrake 8.0. The machine was built from scratch, and Mandrake was installed onto a freshly formatted RAID 5 array. We then migrated the Red Hat 6.2 applications to the new machine.

After a little configuration, the machine seemed to run fine. We successfully migrated the entire system in less than five hours. Considering this was a large-scale server, that was quite a feat and was certainly welcomed by our paying customer.

However, after about a month into deployment I started noticing strange problems with the machine. Intermittent lockups were the most common. The lockups appeared physical, and the machine was unrecoverable without a reboot.

While performing research on the problem, I learned there was a serious sync() bug in the 2.4 kernel. This bug exists in all kernel 2.4 versions until 2.4.6. The solution seemed simple: I upgrade the kernel.

About a week later, the machine locks up cold -- again. We considered it a fluke and rebooted. The very next day the machine locked up -- again. We do further research and find that the original 2.4 VM (Virtual Memory) implementation was causing problems. In my frustration and embarrassment, I would be inclined to call it bad design, but I don't know enough about the intricacies of the Linux kernel to say whether it was.

The VM problem was so horribly bad that the kernel team decided to rip out the older implementation and implement a completely new design. These problems continued as the kernel versions worked their way up through 2.4.11, which has a serious symlink bug that could lead to corrupted inodes. As of 2.4.13, things finally seemed to be cleaned up a bit. The kernel seemed to show more stability. Then we hit kernel 2.4.15.

Linux version 2.4.15 contained a bug that was arguably worse than the VM bug. Essentially, if you unmounted a file system via reboot -- or any another common method -- you would get filesystem corruption. A fix, called kernel 2.4.16, was released 24 hours later.

Kernel 2.4.16 now appeared to be the kernel of choice. It seemed as if it was possible that after almost a year of "stable" status that the 2.4 kernel would be usable in a production environment.

We still aren't there yet

Alas, the mire of trouble within the 2.4 series kernels continues. As of kernel 2.4.16, there is a serious bug in the OOM that can cause system lockups. The lock-up bug in 2.4.16 has supposedly been fixed in 2.4.17pre4aa1.

The current kernel release is 2.4.17, and one would hope that it is stable, but a brief review of the changelog will show that the kernel team is still working on fine-tuning the new VM design, and the vast amount of changes that have been made are already making me weary of it.

As I reviewed the archives of late December, I found that the per-user limit support in the 2.4 series kernels is broken. With the limit support broken, any user -- privileged or not -- has the potential to suck up all of the machines resources, effectively causing an intramural DoS (Denial of Service) attack. They could do this accidentally, and it would cause a great deal of grief for any system administrator.

So, what does all of this mean for me? It means that after five months of battling the new, better-than-fresh-butter, enterprise-ready 2.4 kernel, I am moving my customer back to the stodgy, conservative, more-enterprise-ready-than-2.4-has-been-since-its-release-almost-a-year-ago, 2.2 kernel-based Red Hat 6.2.

The 2.2 kernels may not handle large SMP machines as well, they may not handle large amounts of memory well (only 2 gigabytes), and they may have a practical limit of 2 gigabytes on a single file, but the 2.2. kernels don't crash or cause phone calls at 5:00 AM. Moreover, the 2.2 kernels don't make customers unhappy that they chose Linux as their server solution.

What does this mean for you?

What does all of this mean for you? That is your decision. You just read mine.

I hope Red Hat, SuSE, and Mandrake are taking a long hard look at the 2.4 process and formulating long-term plans to circumvent problems like this. I know, for example, that Red Hat has its own stress testing for the kernel, and that the Red Hat-shipped kernel is a fork of the standard Linux kernel. This fork is a good thing, because it means that Red Hat is able to apply patches that, in theory, make its kernel more stable.

On the desktop that I write this article, I am running Red Hat 7.2 with the 2.4.9-enterprise kernel. (It's a long story that involves this machine's AMD Duron processor.) I have yet to have any lockups on the Red Hat kernel since I upgraded to 2.4.9. I can say that Red Hat 7.2 seems reasonable and usable (at least as a desktop machine) but I am unsure if any 2.4 kernel-based system would be considered acceptable in a production server environment today.

More Stories By Joshua Drake

Joshua Drake is the co-founder of Command Prompt, Inc., a PostgreSQL and Linux custom development company. He is also the current author of the Linux Networking HOWTO, Linux PPP HOWTO, and Linux Consultants HOWTO. His most demanding project at this time is a new PostgreSQL book for O'Reilly, 'Practical PostgreSQL'

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
"ReadyTalk is an audio and web video conferencing provider. We've really come to embrace WebRTC as the platform for our future of technology," explained Dan Cunningham, CTO of ReadyTalk, in this SYS-CON.tv interview at WebRTC Summit at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
In 2014, Amazon announced a new form of compute called Lambda. We didn't know it at the time, but this represented a fundamental shift in what we expect from cloud computing. Now, all of the major cloud computing vendors want to take part in this disruptive technology. In his session at 20th Cloud Expo, John Jelinek IV, a web developer at Linux Academy, will discuss why major players like AWS, Microsoft Azure, IBM Bluemix, and Google Cloud Platform are all trying to sidestep VMs and containers...
IoT is at the core or many Digital Transformation initiatives with the goal of re-inventing a company's business model. We all agree that collecting relevant IoT data will result in massive amounts of data needing to be stored. However, with the rapid development of IoT devices and ongoing business model transformation, we are not able to predict the volume and growth of IoT data. And with the lack of IoT history, traditional methods of IT and infrastructure planning based on the past do not app...
The many IoT deployments around the world are busy integrating smart devices and sensors into their enterprise IT infrastructures. Yet all of this technology – and there are an amazing number of choices – is of no use without the software to gather, communicate, and analyze the new data flows. Without software, there is no IT. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, Dave McCarthy, Director of Products at Bsquare Corporation; Alan Williamson, Principal ...
WebRTC has had a real tough three or four years, and so have those working with it. Only a few short years ago, the development world were excited about WebRTC and proclaiming how awesome it was. You might have played with the technology a couple of years ago, only to find the extra infrastructure requirements were painful to implement and poorly documented. This probably left a bitter taste in your mouth, especially when things went wrong.
SYS-CON Media announced today that @WebRTCSummit Blog, the largest WebRTC resource in the world, has been launched. @WebRTCSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. @WebRTCSummit Blog can be bookmarked ▸ Here @WebRTCSummit conference site can be bookmarked ▸ Here
A critical component of any IoT project is what to do with all the data being generated. This data needs to be captured, processed, structured, and stored in a way to facilitate different kinds of queries. Traditional data warehouse and analytical systems are mature technologies that can be used to handle certain kinds of queries, but they are not always well suited to many problems, particularly when there is a need for real-time insights.
Providing secure, mobile access to sensitive data sets is a critical element in realizing the full potential of cloud computing. However, large data caches remain inaccessible to edge devices for reasons of security, size, format or limited viewing capabilities. Medical imaging, computer aided design and seismic interpretation are just a few examples of industries facing this challenge. Rather than fighting for incremental gains by pulling these datasets to edge devices, we need to embrace the i...
Web Real-Time Communication APIs have quickly revolutionized what browsers are capable of. In addition to video and audio streams, we can now bi-directionally send arbitrary data over WebRTC's PeerConnection Data Channels. With the advent of Progressive Web Apps and new hardware APIs such as WebBluetooh and WebUSB, we can finally enable users to stitch together the Internet of Things directly from their browsers while communicating privately and securely in a decentralized way.
Fifty billion connected devices and still no winning protocols standards. HTTP, WebSockets, MQTT, and CoAP seem to be leading in the IoT protocol race at the moment but many more protocols are getting introduced on a regular basis. Each protocol has its pros and cons depending on the nature of the communications. Does there really need to be only one protocol to rule them all? Of course not. In his session at @ThingsExpo, Chris Matthieu, co-founder and CTO of Octoblu, walked through how Octob...
The Internet of Things can drive efficiency for airlines and airports. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Sudip Majumder, senior director of development at Oracle, discussed the technical details of the connected airline baggage and related social media solutions. These IoT applications will enhance travelers' journey experience and drive efficiency for the airlines and the airports.
SYS-CON Events announced today that Catchpoint, a leading digital experience intelligence company, has been named “Silver Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Catchpoint Systems is a leading Digital Performance Analytics company that provides unparalleled insight into your customer-critical services to help you consistently deliver an amazing customer experience. Designed for digital business, C...
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
In his General Session at 17th Cloud Expo, Bruce Swann, Senior Product Marketing Manager for Adobe Campaign, explored the key ingredients of cross-channel marketing in a digital world. Learn how the Adobe Marketing Cloud can help marketers embrace opportunities for personalized, relevant and real-time customer engagement across offline (direct mail, point of sale, call center) and digital (email, website, SMS, mobile apps, social networks, connected objects).
Things are changing so quickly in IoT that it would take a wizard to predict which ecosystem will gain the most traction. In order for IoT to reach its potential, smart devices must be able to work together. Today, there are a slew of interoperability standards being promoted by big names to make this happen: HomeKit, Brillo and Alljoyn. In his session at @ThingsExpo, Adam Justice, vice president and general manager of Grid Connect, will review what happens when smart devices don’t work togethe...
"Tintri was started in 2008 with the express purpose of building a storage appliance that is ideal for virtualized environments. We support a lot of different hypervisor platforms from VMware to OpenStack to Hyper-V," explained Dan Florea, Director of Product Management at Tintri, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
"There's a growing demand from users for things to be faster. When you think about all the transactions or interactions users will have with your product and everything that is between those transactions and interactions - what drives us at Catchpoint Systems is the idea to measure that and to analyze it," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York Ci...
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in Embedded and IoT solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 7-9, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/Big Data, HPC and E...