Linux Containers Authors: Zakia Bouachraoui, Elizabeth White, Liz McMillan, Pat Romanski, Stefana Muller

Related Topics: Linux Containers

Linux Containers: Article

Dissolving the Limits of Linux: The Breakneck Evolution Continues

Dissolving the Limits of Linux: The Breakneck Evolution Continues

In January, as thousands of Linux OS enthusiasts streamed onto the show floor and into conference sessions at the 2004 LinuxWorld Conference & Expo, they caught up with an old friend who many barely recognized.

The breakneck evolution of Linux into a viable environment for enterprise, research and development, and high-performance computing has converted skeptics who once dismissed the open source environment as a low-cost - and not particularly scalable - alternative to more robust and proven operating environments.

To many commercial users, the earliest implementations of Linux didn't seem suited to drive anything more mission-critical than an edge server, some desktop systems, or a few specialized applications. But much has changed since those early days. Committed and enthusiastic developers have worked hard to push Linux beyond its old limitations, and the results have touched the industry. Sensing a good thing and seeing the opportunity to finally have a unified common operating system across applications and hardware, users and vendors are endorsing Linux as the "one-Unix" once sought by POSIX standards. The increased attention has resulted in added momentum behind Linux and increased expertise in the community that has allowed a multiplying effect of innovation to occur. At LinuxWorld 2003 in New York, attendees saw the unveiling of new systems designed to scale well past the operating system's storied eight-processor ceiling. Silicon Graphics, for instance, debuted a new Linux server and supercluster family - dubbed SGI Altix 3000 - that scales efficiently to 64 processors under a single Linux kernel. This product introduction has been accompanied by hardware vendors who continued to increase their support for Linux by introducing a variety of platforms (both 32- and 64-bit), in addition to growing momentum among ISVs. Support from the Linux community, hardware vendors, and developers made Linux ready for the most demanding customer environments. With Linux now poised for success at the highest levels, corporate bureaucracy and stagnant IT organizations constitute the last rapidly crumbling barrier.

In the past year, developers and users have seen that Linux, with a 2.4 kernel and enhanced software and utilities, has what it takes to drive massively scalable systems to solve HPC-class problems in commercial settings as well as in campus labs. Today, in fact, Linux habitually resets industry expectations of its scalability and applicability. As Linux on HPC platforms like the SGI Altix continues to shatter records on the highest end of computing, a rapidly growing number of end users are embracing Linux as a production environment for running everything from databases to computational fluid dynamics analysis.

Converging Trends

The rise of Linux has resulted from a confluence of crucial industry trends and the dedicated efforts of the open source community, with help from hardware and software solution providers.

64-bit Processor Technology
Cost-efficient processor technology helped prompt the Linux community's initial success in HPC by enabling hobbyists to join the global collaborative movement, and by adding to and improving the operating system's functionality. The relatively recent introduction of 64-bit technologies capable of unseating RISC processors has been vital to the development of Linux as a mature, robust, and performance-oriented operating system as an alternative to proprietary Unix implementations. The wide availability of 64-bit processors has spawned a new generation of powerful servers and supercomputers based on the Intel Itanium 2 family, AMD Opteron, and IBM POWER 4. These systems comprise a growing landscape of options for users, establishing a competitive marketplace for Linux servers that simply did not exist before. This competitive Linux server market is forcing innovation...and that's good.

HPC System Architectures
While standard "white-box" solutions inevitably find their way into any server market segment, they do little to push the boundaries of the Linux operating environment. Because price/performance is always a consideration for Linux users, the challenge for hardware vendors is to build sufficiently differentiated solutions from industry standard components.

In the HPC space, hardware vendors must meet the dramatically increasing expectation for systems to accommodate ever-growing data sets and computational needs of technical and enterprise users - needs not adequately addressed by standard solutions for those who solve complex problems in oil and gas exploration, manufacturing, aerospace design, earth and planetary sciences research, and life sciences. These applications tend to generate terabyte data sets, which in turn create I/O bottlenecks and demand enormous memory availability. As a result, they require system architectures designed to leverage each component to its fullest. SGI's built-in memory channel interconnect fabric with a 3.2GB/sec aggregate bandwidth, for instance, allows the system to deliver data across nodes five to six times faster than conventional clustering interconnects. Beyond this, Altix class systems uniquely provide up to 8 terabytes of total addressable memory. With system architectures affording so much throughput and memory availability, more complex data sets and complete workflows can be driven entirely out of memory, enabling productivity breakthroughs that traditional Linux can't tackle.

It was clear early on that the ability to scale Linux is imperative to meeting the needs of cutting-edge users. System architectures that support reliable scalability - whether scaling up from 4 to 16 processors, or from 256 to 512 - are also vital to Linux evolving as a serious multidiscipline enterprise computing environment. NASA Ames Research Center recently installed an SGI Altix 3000 system that demonstrated 512 processors in a single system image - the largest Linux supercomputer of its kind ever created. By scaling in near-linear fashion to more processors than even some Linux enthusiasts thought possible, the NASA system was the first computer in history to break the 1 terabyte/second barrier on the STREAM Triad benchmark, an industry-standard measurement of memory bandwidth. By the standards of most users, NASA's needs are extreme. But its efforts to throttle Linux into performance overdrive offer a telling example of what Linux can do on the right platform. What's more, it credibly illustrates the Linux community's ability to outpace proprietary solutions, and provides a proof-point of how far Linux can penetrate industry and science in particular.

64-bit Applications
With the initial success of these systems has come growing support from commercial software suppliers, whose customers hungrily demand ever-heightening levels of system scalability. Customer adoption has fueled aggressive Linux porting and development among ISVs - some of the world's most powerful and sophisticated technical applications now run on Linux: BLAST, GAMESS, NX.Nastran, STAR-CD, and Gaussian. Meanwhile, leading database applications from Oracle, Sybase, and MySQL provide even more options for high-performance production Linux environments.

Linux has also proven itself outside of its old perceived boundaries. By expanding its reach to end users in a vast array of markets, it's increasingly clear that Linux can serve as a unifying operating environment, rather than the affordable substitute applicable only to specific systems or application areas.

Ready for Prime Time

As the attendees of this year's LinuxWorld discovered, Linux is on an evolution curve unlike any other operating system in history. The achievements of just the past year have helped shape this OS into a production-quality environment. Customers around the world depend on it for mission-critical applications, including Tata Motors, NASA, the University of Cambridge COSMOS Project, SARA (Dutch National HPC and Networking Center), Marathon Oil Co., and the Pacific Northwest National Laboratory.

Some are first-time Linux users; others, long-time devotees. But whether investigating the origins of the universe, predicting the effects of global warming, or hunting for oil beneath the North Sea, all of these users reap the benefits of each other's efforts. They are part of a global Linux community that, every day, astounds the world with its achievements.

By all accounts, 2004 will be another groundbreaking year for everyone developing, supporting, and deploying Linux. Just imagine what surprises are in store for LinuxWorld 2005.


More Stories By Jason Pettit

Jason Pettit has been working with Linux systems since 1998. As product manager for Linux and Altix 3000, Jason is leading SGI's charge to scale Linux to even greater heights.

Comments (10)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

IoT & Smart Cities Stories
@CloudEXPO and @ExpoDX, two of the most influential technology events in the world, have hosted hundreds of sponsors and exhibitors since our launch 10 years ago. @CloudEXPO and @ExpoDX New York and Silicon Valley provide a full year of face-to-face marketing opportunities for your company. Each sponsorship and exhibit package comes with pre and post-show marketing programs. By sponsoring and exhibiting in New York and Silicon Valley, you reach a full complement of decision makers and buyers in ...
There are many examples of disruption in consumer space – Uber disrupting the cab industry, Airbnb disrupting the hospitality industry and so on; but have you wondered who is disrupting support and operations? AISERA helps make businesses and customers successful by offering consumer-like user experience for support and operations. We have built the world’s first AI-driven IT / HR / Cloud / Customer Support and Operations solution.
LogRocket helps product teams develop better experiences for users by recording videos of user sessions with logs and network data. It identifies UX problems and reveals the root cause of every bug. LogRocket presents impactful errors on a website, and how to reproduce it. With LogRocket, users can replay problems.
Data Theorem is a leading provider of modern application security. Its core mission is to analyze and secure any modern application anytime, anywhere. The Data Theorem Analyzer Engine continuously scans APIs and mobile applications in search of security flaws and data privacy gaps. Data Theorem products help organizations build safer applications that maximize data security and brand protection. The company has detected more than 300 million application eavesdropping incidents and currently secu...
Rafay enables developers to automate the distribution, operations, cross-region scaling and lifecycle management of containerized microservices across public and private clouds, and service provider networks. Rafay's platform is built around foundational elements that together deliver an optimal abstraction layer across disparate infrastructure, making it easy for developers to scale and operate applications across any number of locations or regions. Consumed as a service, Rafay's platform elimi...
Two weeks ago (November 3-5), I attended the Cloud Expo Silicon Valley as a speaker, where I presented on the security and privacy due diligence requirements for cloud solutions. Cloud security is a topical issue for every CIO, CISO, and technology buyer. Decision-makers are always looking for insights on how to mitigate the security risks of implementing and using cloud solutions. Based on the presentation topics covered at the conference, as well as the general discussions heard between sessio...
Growth hacking is common for startups to make unheard-of progress in building their business. Career Hacks can help Geek Girls and those who support them (yes, that's you too, Dad!) to excel in this typically male-dominated world. Get ready to learn the facts: Is there a bias against women in the tech / developer communities? Why are women 50% of the workforce, but hold only 24% of the STEM or IT positions? Some beginnings of what to do about it! In her Day 2 Keynote at 17th Cloud Expo, Sandy Ca...
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists examined how DevOps helps to meet the de...
According to Forrester Research, every business will become either a digital predator or digital prey by 2020. To avoid demise, organizations must rapidly create new sources of value in their end-to-end customer experiences. True digital predators also must break down information and process silos and extend digital transformation initiatives to empower employees with the digital resources needed to win, serve, and retain customers.
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, will provide an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life ...