Linux Containers Authors: Hollis Tibbetts, Pat Romanski, Elizabeth White, Carmen Gonzalez, Yeshim Deniz

Related Topics: Linux Containers

Linux Containers: Article

Can Linux Clusters Move into Mainstream Information Technology?

Exclusive Interview with Rob Lucke, Chief Solutions Officer for Vista Solutions Corp

The emergence of commodity supercomputing has driven clusters based on the Linux operating system into engineering and scientific research organizations that couldn't afford their own supercomputing resources before. But Linux clusters have the potential to become a hot topic in traditional information technology circles as well. The coming year may well be the technology tipping point when Linux cluster technology escapes its current home in research organizations and inhabits the traditional data center.

Building Clustered Linux Systems by Robert Lucke, recently published by Prentice Hall Professional Technical Reference and HP Books, attempts to provide a starting point for organizations interested in building or evaluating their first Linux cluster.

We took this opportunity to have a chat with Robert and ask him a few questions on the subject of Linux and clustering.

What made you want to write a book about building clusters?

Before starting work on clusters, I spent a considerable amount of time tackling workgroup architecture and large-scale system administration problems for my engineering and scientific customers. When I got the opportunity to work on a prototype Itanium 2 cluster at Pacific Northwest National Laboratory, I was fascinated by the new techologies like the high-speed interconnect from Quadrics, and recognized many familiar management and architectural issues. The more I learned, the more I saw applications for clusters in other more "traditional" areas. The book itself was a learning experience for me and an attempt to collect and organize cluster-building information for organizations that are investigating clustered solutions.

Why do you think that clusters are an important architecture?

If you have the proper application software, a cluster can scale high-performance, high-availability or high-throughput resources far beyond anything that is available in a single SMP system. Being able to do this with commodity hardware brings tremendous compute resources in reach of organizations that previously couldn't afford them. I see cluster architectures and techniques as the gateway to some of the resource virtualization that seems to be the Holy Grail of traditional IT departments today. I think this is exciting! I guess it's the love of finding an elegant solution to a problem that drives my excitement. Instead of being a marketechture, clusters represent a real solution to a group of scalable problems.

When someone says cluster, what does it mean to you?

I have learned to be very, very careful with the word cluster. It's overloaded and the meaning depends on the audience. Using cluster in a scientific context evokes a different mental picture than if it were used by traditional IT folks. In general, I think of a cluster as a group of separate resources like systems, CPUs and RAM that gets poured into a mold. The shape of the mold determines the final shape and behavior of the clustered solution. Sizing the problem and determining the shape of the mold is the fun part for me.

Besides scientific and engineering environments, do you see any other applications for clusters?

I sure do! There are database clusters, web server clusters, file server clusters, visualization clusters, and on and on. Instead of building clusters that push the upper limits of RAM and CPU resources, thousands to tens of thousands of CPUs, for example, the company I work for concentrates on application-specific clusters. These are smaller, single-function clusters that are meant to run application configurations that would have required a large and expensive SMP system. The intent is to lower the complexity of building and managing the cluster, but still provide a more cost-effective solution for the application. I think this type of approach is generally applicable in any type of computing environment.

 What are some of the common mistakes you've encountered in cluster building?

The single biggest problem I run into is what I call pile o'hardware syndrome. That's the notion that you just buy a whole bunch of cheap hardware, rack it up, and a cluster will magically appear out of a pile of pieces. It's still very common to underestimate the amount of work required to make physically separate resources work together as if they were one very large, manageable SMP system. A cluster is still a systems engineering problem that can turn nasty if you aren't careful. But, with advances in pre-racked, pre-cabled hardware from some of the hardware vendors and the cluster software toolkits like OSCAR and Rocks, I see cluster building getting easier all the time.

Why do you think that Linux is the best cluster operating system?

One simple answer is choice. There are commercial distributions, free distributions, white-box distributions and so forth. If you have a commercial software package like an Oracle database that's qualified against a particular Linux commercial distribution like SuSE or RedHat, you can build a fully supported cluster configuration. If you want to do research or custom work, there are free distributions like Debian or Fedora. Because the source code is available, you can choose your starting point and degree of customization. This is the best of all possible worlds.

The Linux operating system is stable, manageable and flexible. You are free to configure Linux as you see fit instead of trying to chip away at a black-box operating system that fights you every step of the way. There's a wealth of free management and development tools available. Oh, did I mention that Linux runs on a wide range of commodity hardware, both 32- and 64-bit? What's not to like? Nothing else comes close in my estimation.

What do you see ahead for clustered architectures?

I definitely see Linux clusters moving into mainstream information technology environments. If you look back, the scientific community tends to drive computing technologies that are later adopted by the more conservative IT organizations as business solutions. One modest example I can think of might be the World Wide Web and the Mosaic browser. I firmly believe that clusters, specifically Linux clusters, are poised to repeat this type of adoption pattern. I think we are very close, if not past, the tipping point.

What would you say to someone who is thinking about building his first cluster?

Do it, but do it with your eyes open. Do your homework before starting. Give yourself time to learn. Try not to fall into the pile o'hardware trap. Start small and scale up. Investigate starting points like openMosix, Rocks and OSCAR first. If you don't have time for the learning curve, then have a replicable solution designed and implemented for you.


In addition to their usefulness in scientific and engineering computing environments, I believe that Linux clusters and clustering techniques will be an important addition to the standard information technology solutions in the corporate datacenter. The trick is going to be sharing the cluster-building knowledge that's available in universities and research institutions with the traditional information technology organization. Because of its stability, flexibility, open nature, manageability and availability on a wide range of commodity hardware, I believe that Linux is the correct choice for the creation of clustered solutions. I am really looking forward to the next few years. I believe it will be an exciting time for both Linux and clusters.

About Rob Lucke

Rob Lucke is currently chief solutions officer for Vista Solutions Corp. (http://www.VistaSolutions.Net), concentrating on technical and scientific computing. Rob's field of expertise include Linux compute clusters, technical systems architecture, large-scale system administration techniques, network file systems, heterogeneous interoperability, software development and application and system-level performance tuning. Rob has over 30 years of experience in computing and software of all types from real-time data acquisition to transaction processing. His first book, Designing and Implementing Computer Workgroups, was published in 1999. His second book, Building Clustered Linux Systems, was published in September of 2004. Rob is Red Hat Linux certified engineer #807200931604117.

More Stories By Ibrahim Haddad

Ibrahim Haddad is a member of the management team at The Linux Foundation responsible for technical, legal and compliance projects and initiatives. Prior to that, he ran the Open Source Office at Palm, the Open Source Technology Group at Motorola, and Global Telecommunications Initiatives at The Open Source Development Labs. Ibrahim started his career as a member of the research team at Ericsson Research focusing on advanced research for system architecture of 3G wireless IP networks and on the adoption of open source software in telecom. Ibrahim graduated from Concordia University (Montréal, Canada) with a Ph.D. in Computer Science. He is a Contributing Editor to the Linux Journal. Ibrahim is fluent in Arabic, English and French. He can be reached via http://www.IbrahimHaddad.com.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

@ThingsExpo Stories
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
A completely new computing platform is on the horizon. They’re called Microservers by some, ARM Servers by others, and sometimes even ARM-based Servers. No matter what you call them, Microservers will have a huge impact on the data center and on server computing in general. Although few people are familiar with Microservers today, their impact will be felt very soon. This is a new category of computing platform that is available today and is predicted to have triple-digit growth rates for some ...
In past @ThingsExpo presentations, Joseph di Paolantonio has explored how various Internet of Things (IoT) and data management and analytics (DMA) solution spaces will come together as sensor analytics ecosystems. This year, in his session at @ThingsExpo, Joseph di Paolantonio from DataArchon, will be adding the numerous Transportation areas, from autonomous vehicles to “Uber for containers.” While IoT data in any one area of Transportation will have a huge impact in that area, combining sensor...
SYS-CON Events announced today that Transparent Cloud Computing (T-Cloud) Consortium will exhibit at the 19th International Cloud Expo®, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. The Transparent Cloud Computing Consortium (T-Cloud Consortium) will conduct research activities into changes in the computing model as a result of collaboration between "device" and "cloud" and the creation of new value and markets through organic data proces...
SYS-CON Events announced today that Niagara Networks will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Niagara Networks offers the highest port-density systems, and the most complete Next-Generation Network Visibility systems including Network Packet Brokers, Bypass Switches, and Network TAPs.
Everyone knows that truly innovative companies learn as they go along, pushing boundaries in response to market changes and demands. What's more of a mystery is how to balance innovation on a fresh platform built from scratch with the legacy tech stack, product suite and customers that continue to serve as the business' foundation. In his General Session at 19th Cloud Expo, Michael Chambliss, Head of Engineering at ReadyTalk, will discuss why and how ReadyTalk diverted from healthy revenue an...
SYS-CON Media announced today that @WebRTCSummit Blog, the largest WebRTC resource in the world, has been launched. @WebRTCSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. @WebRTCSummit Blog can be bookmarked ▸ Here @WebRTCSummit conference site can be bookmarked ▸ Here
Intelligent machines are here. Robots, self-driving cars, drones, bots and many IoT devices are becoming smarter with Machine Learning. In her session at @ThingsExpo, Sudha Jamthe, CEO of IoTDisruptions.com, will discuss the next wave of business disruption at the junction of IoT and AI, impacting many industries and set to change our lives, work and world as we know it.
In today's uber-connected, consumer-centric, cloud-enabled, insights-driven, multi-device, global world, the focus of solutions has shifted from the product that is sold to the person who is buying the product or service. Enterprises have rebranded their business around the consumers of their products. The buyer is the person and the focus is not on the offering. The person is connected through multiple devices, wearables, at home, on the road, and in multiple locations, sometimes simultaneously...
Established in 1998, Calsoft is a leading software product engineering Services Company specializing in Storage, Networking, Virtualization and Cloud business verticals. Calsoft provides End-to-End Product Development, Quality Assurance Sustenance, Solution Engineering and Professional Services expertise to assist customers in achieving their product development and business goals. The company's deep domain knowledge of Storage, Virtualization, Networking and Cloud verticals helps in delivering ...
SYS-CON Events announced today that Hitrons Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Hitrons Solutions Inc. is distributor in the North American market for unique products and services of small and medium-size businesses, including cloud services and solutions, SEO marketing platforms, and mobile applications.
The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today.
OnProcess Technology has announced it will be a featured speaker at @ThingsExpo, taking place November 1 - 3, 2016, in Santa Clara, California. Dan Gettens, OnProcess’ Chief Analytics Officer, will discuss how Internet of Things (IoT) data can be leveraged to predict product failures, improve uptime and slash costly inventory stock. @ThingsExpo is an annual gathering of IoT and cloud developers, practitioners and thought-leaders who exchange ideas and insights on topics ranging from Big Data in...
SYS-CON Events announced today that Enzu will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to focus on the core of their online busine...
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
The Open Connectivity Foundation (OCF), sponsor of the IoTivity open source project, and AllSeen Alliance, which provides the AllJoyn® open source IoT framework, today announced that the two organizations’ boards have approved a merger under the OCF name and bylaws. This merger will advance interoperability between connected devices from both groups, enabling the full operating potential of IoT and representing a significant step towards a connected ecosystem.
November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Penta Security is a leading vendor for data security solutions, including its encryption solution, D’Amo. By using FPE technology, D’Amo allows for the implementation of encryption technology to sensitive data fields without modification to schema in the database environment. With businesses having their data become increasingly more complicated in their mission-critical applications (such as ERP, CRM, HRM), continued ...
SYS-CON Events announced today that Embotics, the cloud automation company, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Embotics is the cloud automation company for IT organizations and service providers that need to improve provisioning or enable self-service capabilities. With a relentless focus on delivering a premier user experience and unmatched customer support, Embotics is the fas...
SYS-CON Events announced today that Cloudbric, a leading website security provider, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Cloudbric is an elite full service website protection solution specifically designed for IT novices, entrepreneurs, and small and medium businesses. First launched in 2015, Cloudbric is based on the enterprise level Web Application Firewall by Penta Security Sys...
Smart Cities are here to stay, but for their promise to be delivered, the data they produce must not be put in new siloes. In his session at @ThingsExpo, Mathias Herberts, Co-founder and CTO of Cityzen Data, will deep dive into best practices that will ensure a successful smart city journey.