Click here to close now.

Welcome!

Linux Authors: Yeshim Deniz, Liz McMillan, Pat Romanski, Elizabeth White, Kathy Thomas

Related Topics: Linux

Linux: Article

Can Linux Clusters Move into Mainstream Information Technology?

Exclusive Interview with Rob Lucke, Chief Solutions Officer for Vista Solutions Corp

The emergence of commodity supercomputing has driven clusters based on the Linux operating system into engineering and scientific research organizations that couldn't afford their own supercomputing resources before. But Linux clusters have the potential to become a hot topic in traditional information technology circles as well. The coming year may well be the technology tipping point when Linux cluster technology escapes its current home in research organizations and inhabits the traditional data center.

Building Clustered Linux Systems by Robert Lucke, recently published by Prentice Hall Professional Technical Reference and HP Books, attempts to provide a starting point for organizations interested in building or evaluating their first Linux cluster.

We took this opportunity to have a chat with Robert and ask him a few questions on the subject of Linux and clustering.

What made you want to write a book about building clusters?

Before starting work on clusters, I spent a considerable amount of time tackling workgroup architecture and large-scale system administration problems for my engineering and scientific customers. When I got the opportunity to work on a prototype Itanium 2 cluster at Pacific Northwest National Laboratory, I was fascinated by the new techologies like the high-speed interconnect from Quadrics, and recognized many familiar management and architectural issues. The more I learned, the more I saw applications for clusters in other more "traditional" areas. The book itself was a learning experience for me and an attempt to collect and organize cluster-building information for organizations that are investigating clustered solutions.

Why do you think that clusters are an important architecture?

If you have the proper application software, a cluster can scale high-performance, high-availability or high-throughput resources far beyond anything that is available in a single SMP system. Being able to do this with commodity hardware brings tremendous compute resources in reach of organizations that previously couldn't afford them. I see cluster architectures and techniques as the gateway to some of the resource virtualization that seems to be the Holy Grail of traditional IT departments today. I think this is exciting! I guess it's the love of finding an elegant solution to a problem that drives my excitement. Instead of being a marketechture, clusters represent a real solution to a group of scalable problems.

When someone says cluster, what does it mean to you?

I have learned to be very, very careful with the word cluster. It's overloaded and the meaning depends on the audience. Using cluster in a scientific context evokes a different mental picture than if it were used by traditional IT folks. In general, I think of a cluster as a group of separate resources like systems, CPUs and RAM that gets poured into a mold. The shape of the mold determines the final shape and behavior of the clustered solution. Sizing the problem and determining the shape of the mold is the fun part for me.

Besides scientific and engineering environments, do you see any other applications for clusters?

I sure do! There are database clusters, web server clusters, file server clusters, visualization clusters, and on and on. Instead of building clusters that push the upper limits of RAM and CPU resources, thousands to tens of thousands of CPUs, for example, the company I work for concentrates on application-specific clusters. These are smaller, single-function clusters that are meant to run application configurations that would have required a large and expensive SMP system. The intent is to lower the complexity of building and managing the cluster, but still provide a more cost-effective solution for the application. I think this type of approach is generally applicable in any type of computing environment.

 What are some of the common mistakes you've encountered in cluster building?

The single biggest problem I run into is what I call pile o'hardware syndrome. That's the notion that you just buy a whole bunch of cheap hardware, rack it up, and a cluster will magically appear out of a pile of pieces. It's still very common to underestimate the amount of work required to make physically separate resources work together as if they were one very large, manageable SMP system. A cluster is still a systems engineering problem that can turn nasty if you aren't careful. But, with advances in pre-racked, pre-cabled hardware from some of the hardware vendors and the cluster software toolkits like OSCAR and Rocks, I see cluster building getting easier all the time.

Why do you think that Linux is the best cluster operating system?

One simple answer is choice. There are commercial distributions, free distributions, white-box distributions and so forth. If you have a commercial software package like an Oracle database that's qualified against a particular Linux commercial distribution like SuSE or RedHat, you can build a fully supported cluster configuration. If you want to do research or custom work, there are free distributions like Debian or Fedora. Because the source code is available, you can choose your starting point and degree of customization. This is the best of all possible worlds.

The Linux operating system is stable, manageable and flexible. You are free to configure Linux as you see fit instead of trying to chip away at a black-box operating system that fights you every step of the way. There's a wealth of free management and development tools available. Oh, did I mention that Linux runs on a wide range of commodity hardware, both 32- and 64-bit? What's not to like? Nothing else comes close in my estimation.

What do you see ahead for clustered architectures?

I definitely see Linux clusters moving into mainstream information technology environments. If you look back, the scientific community tends to drive computing technologies that are later adopted by the more conservative IT organizations as business solutions. One modest example I can think of might be the World Wide Web and the Mosaic browser. I firmly believe that clusters, specifically Linux clusters, are poised to repeat this type of adoption pattern. I think we are very close, if not past, the tipping point.

What would you say to someone who is thinking about building his first cluster?

Do it, but do it with your eyes open. Do your homework before starting. Give yourself time to learn. Try not to fall into the pile o'hardware trap. Start small and scale up. Investigate starting points like openMosix, Rocks and OSCAR first. If you don't have time for the learning curve, then have a replicable solution designed and implemented for you.

Conclusion

In addition to their usefulness in scientific and engineering computing environments, I believe that Linux clusters and clustering techniques will be an important addition to the standard information technology solutions in the corporate datacenter. The trick is going to be sharing the cluster-building knowledge that's available in universities and research institutions with the traditional information technology organization. Because of its stability, flexibility, open nature, manageability and availability on a wide range of commodity hardware, I believe that Linux is the correct choice for the creation of clustered solutions. I am really looking forward to the next few years. I believe it will be an exciting time for both Linux and clusters.

About Rob Lucke

Rob Lucke is currently chief solutions officer for Vista Solutions Corp. (http://www.VistaSolutions.Net), concentrating on technical and scientific computing. Rob's field of expertise include Linux compute clusters, technical systems architecture, large-scale system administration techniques, network file systems, heterogeneous interoperability, software development and application and system-level performance tuning. Rob has over 30 years of experience in computing and software of all types from real-time data acquisition to transaction processing. His first book, Designing and Implementing Computer Workgroups, was published in 1999. His second book, Building Clustered Linux Systems, was published in September of 2004. Rob is Red Hat Linux certified engineer #807200931604117.

More Stories By Ibrahim Haddad

Ibrahim Haddad is a member of the management team at The Linux Foundation responsible for technical, legal and compliance projects and initiatives. Prior to that, he ran the Open Source Office at Palm, the Open Source Technology Group at Motorola, and Global Telecommunications Initiatives at The Open Source Development Labs. Ibrahim started his career as a member of the research team at Ericsson Research focusing on advanced research for system architecture of 3G wireless IP networks and on the adoption of open source software in telecom. Ibrahim graduated from Concordia University (Montréal, Canada) with a Ph.D. in Computer Science. He is a Contributing Editor to the Linux Journal. Ibrahim is fluent in Arabic, English and French. He can be reached via http://www.IbrahimHaddad.com.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
In the consumer IoT, everything is new, and the IT world of bits and bytes holds sway. But industrial and commercial realms encompass operational technology (OT) that has been around for 25 or 50 years. This grittier, pre-IP, more hands-on world has much to gain from Industrial IoT (IIoT) applications and principles. But adding sensors and wireless connectivity won’t work in environments that demand unwavering reliability and performance. In his session at @ThingsExpo, Ron Sege, CEO of Echelon, will discuss how as enterprise IT embraces other IoT-related technology trends, enterprises with i...
Enthusiasm for the Internet of Things has reached an all-time high. In 2013 alone, venture capitalists spent more than $1 billion dollars investing in the IoT space. With "smart" appliances and devices, IoT covers wearable smart devices, cloud services to hardware companies. Nest, a Google company, detects temperatures inside homes and automatically adjusts it by tracking its user's habit. These technologies are quickly developing and with it come challenges such as bridging infrastructure gaps, abiding by privacy concerns and making the concept a reality. These challenges can't be addressed w...
We’re no longer looking to the future for the IoT wave. It’s no longer a distant dream but a reality that has arrived. It’s now time to make sure the industry is in alignment to meet the IoT growing pains – cooperate and collaborate as well as innovate. In his session at @ThingsExpo, Jim Hunter, Chief Scientist & Technology Evangelist at Greenwave Systems, will examine the key ingredients to IoT success and identify solutions to challenges the industry is facing. The deep industry expertise behind this presentation will provide attendees with a leading edge view of rapidly emerging IoT oppor...
The industrial software market has treated data with the mentality of “collect everything now, worry about how to use it later.” We now find ourselves buried in data, with the pervasive connectivity of the (Industrial) Internet of Things only piling on more numbers. There’s too much data and not enough information. In his session at @ThingsExpo, Bob Gates, Global Marketing Director, GE’s Intelligent Platforms business, to discuss how realizing the power of IoT, software developers are now focused on understanding how industrial data can create intelligence for industrial operations. Imagine ...
SYS-CON Events announced today that Liaison Technologies, a leading provider of data management and integration cloud services and solutions, has been named "Silver Sponsor" of SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York, NY. Liaison Technologies is a recognized market leader in providing cloud-enabled data integration and data management solutions to break down complex information barriers, enabling enterprises to make smarter decisions, faster.
The 17th International Cloud Expo has announced that its Call for Papers is open. 17th International Cloud Expo, to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, APM, APIs, Microservices, Security, Big Data, Internet of Things, DevOps and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
Collecting data in the field and configuring multitudes of unique devices is a time-consuming, labor-intensive process that can stretch IT resources. Horan & Bird [H&B], Australia’s fifth-largest Solar Panel Installer, wanted to automate sensor data collection and monitoring from its solar panels and integrate the data with its business and marketing systems. After data was collected and structured, two major areas needed to be addressed: improving developer workflows and extending access to a business application to multiple users (multi-tenancy). Docker, a container technology, was used to ...
Hadoop as a Service (as offered by handful of niche vendors now) is a cloud computing solution that makes medium and large-scale data processing accessible, easy, fast and inexpensive. In his session at Big Data Expo, Kumar Ramamurthy, Vice President and Chief Technologist, EIM & Big Data, at Virtusa, will discuss how this is achieved by eliminating the operational challenges of running Hadoop, so one can focus on business growth. The fragmented Hadoop distribution world and various PaaS solutions that provide a Hadoop flavor either make choices for customers very flexible in the name of opti...
For years, we’ve relied too heavily on individual network functions or simplistic cloud controllers. However, they are no longer enough for today’s modern cloud data center. Businesses need a comprehensive platform architecture in order to deliver a complete networking suite for IoT environment based on OpenStack. In his session at @ThingsExpo, Dhiraj Sehgal from PLUMgrid will discuss what a holistic networking solution should really entail, and how to build a complete platform that is scalable, secure, agile and automated.
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo, June 9-11, 2015, at the Javits Center in New York City. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal an...
Wearable devices have come of age. The primary applications of wearables so far have been "the Quantified Self" or the tracking of one's fitness and health status. We propose the evolution of wearables into social and emotional communication devices. Our BE(tm) sensor uses light to visualize the skin conductance response. Our sensors are very inexpensive and can be massively distributed to audiences or groups of any size, in order to gauge reactions to performances, video, or any kind of presentation. In her session at @ThingsExpo, Jocelyn Scheirer, CEO & Founder of Bionolux, will discuss ho...
The true value of the Internet of Things (IoT) lies not just in the data, but through the services that protect the data, perform the analysis and present findings in a usable way. With many IoT elements rooted in traditional IT components, Big Data and IoT isn’t just a play for enterprise. In fact, the IoT presents SMBs with the prospect of launching entirely new activities and exploring innovative areas. CompTIA research identifies several areas where IoT is expected to have the greatest impact.
The 4th International Internet of @ThingsExpo, co-located with the 17th International Cloud Expo - to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA - announces that its Call for Papers is open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
Every day we read jaw-dropping stats on the explosion of data. We allocate significant resources to harness and better understand it. We build businesses around it. But we’ve only just begun. For big payoffs in Big Data, CIOs are turning to cognitive computing. Cognitive computing’s ability to securely extract insights, understand natural language, and get smarter each time it’s used is the next, logical step for Big Data.
There's no doubt that the Internet of Things is driving the next wave of innovation. Google has spent billions over the past few months vacuuming up companies that specialize in smart appliances and machine learning. Already, Philips light bulbs, Audi automobiles, and Samsung washers and dryers can communicate with and be controlled from mobile devices. To take advantage of the opportunities the Internet of Things brings to your business, you'll want to start preparing now.
17th Cloud Expo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS – software, platform, and infrastructure as a service.
P2P RTC will impact the landscape of communications, shifting from traditional telephony style communications models to OTT (Over-The-Top) cloud assisted & PaaS (Platform as a Service) communication services. The P2P shift will impact many areas of our lives, from mobile communication, human interactive web services, RTC and telephony infrastructure, user federation, security and privacy implications, business costs, and scalability. In his session at @ThingsExpo, Robin Raymond, Chief Architect at Hookflash, will walk through the shifting landscape of traditional telephone and voice services ...
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Architect for the Internet of Things and Intelligent Systems at Red Hat, described how to revolutioniz...
For IoT to grow as quickly as analyst firms’ project, a lot is going to fall on developers to quickly bring applications to market. But the lack of a standard development platform threatens to slow growth and make application development more time consuming and costly, much like we’ve seen in the mobile space. In his session at @ThingsExpo, Mike Weiner is Product Manager of the Omega DevCloud with KORE Telematics Inc., will discuss the evolving requirements for developers as IoT matures and conduct a live demonstration of how quickly application development can happen when the need to comply...