Click here to close now.




















Welcome!

Linux Containers Authors: Elizabeth White, Liz McMillan, Pat Romanski, Esmeralda Swartz, Ruxit Blog

Related Topics: Linux Containers

Linux Containers: Article

Easing Data Migration

The trick is don't move the data

Linux is emerging as the platform of choice for a growing number of enterprises across the globe. The cost, choice, and control advantages of using Open Source software for mission-critical applications have already enabled hundreds of organizations to control IT costs while expanding IT capabilities and productivity. Customers in telecommunications, financial services, and government have aggressively already deployed Linux in production workloads like databases, SAP, messaging services, and custom applications.

While moving to a new operating system is not trivial, its complexity pales in comparison to the struggles of migrating actual data from one platform to another in production environments. The ability to migrate data between different operating systems can reduce IT costs, either as part of platform migrations or multi-platform workflows.

Many companies undertake elaborate migration projects that require the manual migration of data. However, manually migrating data between typically disparate and incompatible systems requires a substantial investment of time and labor. In fact, this complexity often overwhelms the benefits it promises. Moreover, such migrations can trigger a range of risks in data loss, data corruption, policy compliance, and -worst of all - production downtime.

As a result, a growing number of organizations are turning to automated data migration tools to minimize such costs and risks in migrating production data workloads.

The Need to Migrate Data
Server and storage equipment replacements, relocation, consolidation, lease renewals, and balancing workloads all drive the need to migrate data on a regular basis. With larger disk sizes readily available, many organizations are looking to control costs by replacing a number of smaller drives with fewer but larger drives. Of course, fewer drives also means fewer spindles, which can negatively impact overall system performance.

Others simply have so much storage spread out among their worldwide data centers that storage migrations become a frequent process of removing old storage and adding new storage devices.

Other organizations discover they've outgrown their storage capabilities faster than anticipated and planned for, making their existing infrastructure unable to accommodate current and future data storage needs.

The Challenges
Migrating data to a Linux platform is easier said than done. According to a recent survey by Symantec over 72% of respondents take more than two weeks to plan an implementation and over 40% of the migrations involve more than five people to complete them. What's more, 61% exceed their planned downtime, 54% exceed their budget, and 83% exceed their staffing plan.

First, there are the operational issues to consider. Downtime must be scheduled, particularly in cases where the organization is making an application's data set accessible from another access point in the data center. And, with today's virtual environments, organizations have to be able to migrate from a physical to a virtual environment, and vice versa. Having inadequate manual and semi-automated approaches makes this even more difficult.

In all cases, coordination is key to a successful data migration. All administration groups involved in the process must be aware of the organization's data migration schedule, and process, and their role in it. And re-establishing access to storage must be done with minimal disruptions - which is very difficult when upgrading or adding another switch to a storage area network (SAN).

Beyond the operational challenges, organizations have to contend with storage-centric issues, the most daunting being file system issues. When moving data from a Unix to a Linux environment, for example, or simply adding new storage to a server and moving off an old storage device, it's necessary to resize the file system to use the new storage. A number of technologies facilitate this, enabling the virtualization of storage in such a way that the file system can interoperate better with the storage infrastructure.

Organizations must also deal with storage volumes that have incompatible formats, the challenge of preserving LUN and disk mappings across the migration, reclamation, and ensuring capacity at the destination. And as with any conversion and migration, the integrity of the data is at risk.

Application-level issues have to be considered when migrating data from one platform, such as Unix to Linux. Application data formats may not be cross-platform portable, some sort of conversion process on the data file format has to occur to be able to reach the same data on a Linux box.

Finally, organizations must contend with TCP/IP network-centric issues such as ensuring sufficient bandwidth and addressing interoperability concerns. Physical connectivity issues such as re-cabling and the implications on performance made by topological changes must also be addressed.

Easing Cross-Platform Data Migration
With half or more of enterprises' structured data stored in databases by some analyst estimates, this data is very likely to be migrated between unlike platforms at some point in its lifetime. But manual methods make the process unwieldy, time-consuming, and resource-intensive.

For example, moving a database from an Oracle instance running on a Sun Solaris server to another Oracle instance on a Linux server introduces a number of challenges. The storage volumes mounted on the existing system can't simply be unplugged and attached to the new server because the new Linux-based server can't interpret the information being sent.

There are a number of platform-specific factors that limit the ability to share volumes across servers. Among these are disk drive sector size and block size. As a result, new volumes have to be created on the Linux system, and these volumes have to be configured to get data from the existing Solaris server. All processing of applications has to be halted as the data moves from one platform to the next, and the data on the volumes has to be moved physically to the new Linux server. This can be done across the network or manually using tape backup and restore procedures. And the volumes will probably have to be converted before they are mounted or restored on the server. This typically happens when data is moved between platforms with dissimilar endians.

To overcome these challenges, a growing number of organizations are turning to new technologies that don't move the data but simply let it be accessed from another operating system host. The key to this technology is a new default disk format, the basis of platform-independent virtual volume building blocks, often called portable data containers. Volumes formatted with the new parameters of this disk format can be used with volume manager solutions regardless of the operating environment that initialized the disk (including issues like endianess). The resulting volume format enables platform-specific dependencies to be removed from the data movement equation, including sector and block size. In short, why convert and migrate the data when you can just convert the metadata and remount the storage device?

With this new technology, migrating data from Unix to Linux is a simple process, taking minutes, not days. Administrators unmount the file system on Unix, run a conversion utility, deport disks on Unix and import disks on Linux, start volumes, and mount the file system. According to laboratory tests this process can be done in less than a few minutes for a 500GB tablespace - whereas data conversion from tape backup would take five hours and the same process from NFS would need four hours. Actually the time it takes for such migrations isn't dependent on the total size (or capacity) of the data, but on the number of files in the file system.

The portable data-container building blocks simplify data migrations between heterogeneous server platforms. Application data storage can be used by any processing platform, which offers IT organizations greater leverage over existing heterogeneous computing resources in their environment.

Enhancing Business Performance
Moving data from one platform will never be trivial. In fact, it has historically been so hard that many organizations run their applications on sub-optimal and expensive legacy platforms just to avoid the complexities and downtime associated with data migration.

However, by leveraging new technologies that reduce the time and resources required to move data between unlike platforms - obviating the need and risk of traditional data migrations - volumes can easily be transported between unlike platforms. Physical disks can be grouped into logical volumes to improve disk utilization and eliminate storage-related downtime. Moreover, administrators have the flexibility to move data between storage arrays as needed, migrate data to new operating systems, and move files to the most appropriate storage device based on importance.

With these tools, organizations can reduce cost, risk, and downtime, while enhancing performance and maximizing the productivity of their heterogeneous IT environments.

Reference

More Stories By Andy Fenselau

Andy Fenselau has led product management across various parts of the Linux technology stack since 1998. He is currently the Linux Product Line Manager for Symantec's enterprise storage and server management solutions, spending most of his time with customers and partners to ensure Symantec's Linux solutions are meeting their needs. As a Linux evangelist, Andy has authored many articles and spoken at many events about the technical and business advantages of the evolving Linux solutions. He holds a BA from Harvard University and an MBA from Stanford University.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that Pythian, a global IT services company specializing in helping companies leverage disruptive technologies to optimize revenue-generating systems, has been named “Bronze Sponsor” of SYS-CON's 17th Cloud Expo, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Founded in 1997, Pythian is a global IT services company that helps companies compete by adopting disruptive technologies such as cloud, Big Data, advanced analytics, and DevOps to advance innovation and increase agility. Specializing in designing, imple...
Through WebRTC, audio and video communications are being embedded more easily than ever into applications, helping carriers, enterprises and independent software vendors deliver greater functionality to their end users. With today’s business world increasingly focused on outcomes, users’ growing calls for ease of use, and businesses craving smarter, tighter integration, what’s the next step in delivering a richer, more immersive experience? That richer, more fully integrated experience comes about through a Communications Platform as a Service which allows for messaging, screen sharing, video...
As more and more data is generated from a variety of connected devices, the need to get insights from this data and predict future behavior and trends is increasingly essential for businesses. Real-time stream processing is needed in a variety of different industries such as Manufacturing, Oil and Gas, Automobile, Finance, Online Retail, Smart Grids, and Healthcare. Azure Stream Analytics is a fully managed distributed stream computation service that provides low latency, scalable processing of streaming data in the cloud with an enterprise grade SLA. It features built-in integration with Azur...
SYS-CON Events announced today that Micron Technology, Inc., a global leader in advanced semiconductor systems, will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Micron’s broad portfolio of high-performance memory technologies – including DRAM, NAND and NOR Flash – is the basis for solid state drives, modules, multichip packages and other system solutions. Backed by more than 35 years of technology leadership, Micron's memory solutions enable the world's most innovative computing, consumer,...
Contrary to mainstream media attention, the multiple possibilities of how consumer IoT will transform our everyday lives aren’t the only angle of this headline-gaining trend. There’s a huge opportunity for “industrial IoT” and “Smart Cities” to impact the world in the same capacity – especially during critical situations. For example, a community water dam that needs to release water can leverage embedded critical communications logic to alert the appropriate individuals, on the right device, as soon as they are needed to take action.
As more intelligent IoT applications shift into gear, they’re merging into the ever-increasing traffic flow of the Internet. It won’t be long before we experience bottlenecks, as IoT traffic peaks during rush hours. Organizations that are unprepared will find themselves by the side of the road unable to cross back into the fast lane. As billions of new devices begin to communicate and exchange data – will your infrastructure be scalable enough to handle this new interconnected world?
While many app developers are comfortable building apps for the smartphone, there is a whole new world out there. In his session at @ThingsExpo, Narayan Sainaney, Co-founder and CTO of Mojio, will discuss how the business case for connected car apps is growing and, with open platform companies having already done the heavy lifting, there really is no barrier to entry.
With the Apple Watch making its way onto wrists all over the world, it’s only a matter of time before it becomes a staple in the workplace. In fact, Forrester reported that 68 percent of technology and business decision-makers characterize wearables as a top priority for 2015. Recognizing their business value early on, FinancialForce.com was the first to bring ERP to wearables, helping streamline communication across front and back office functions. In his session at @ThingsExpo, Kevin Roberts, GM of Platform at FinancialForce.com, will discuss the value of business applications on wearable ...
WebRTC has had a real tough three or four years, and so have those working with it. Only a few short years ago, the development world were excited about WebRTC and proclaiming how awesome it was. You might have played with the technology a couple of years ago, only to find the extra infrastructure requirements were painful to implement and poorly documented. This probably left a bitter taste in your mouth, especially when things went wrong.
Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome,” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.
The Internet of Things is in the early stages of mainstream deployment but it promises to unlock value and rapidly transform how organizations manage, operationalize, and monetize their assets. IoT is a complex structure of hardware, sensors, applications, analytics and devices that need to be able to communicate geographically and across all functions. Once the data is collected from numerous endpoints, the challenge then becomes converting it into actionable insight.
Consumer IoT applications provide data about the user that just doesn’t exist in traditional PC or mobile web applications. This rich data, or “context,” enables the highly personalized consumer experiences that characterize many consumer IoT apps. This same data is also providing brands with unprecedented insight into how their connected products are being used, while, at the same time, powering highly targeted engagement and marketing opportunities. In his session at @ThingsExpo, Nathan Treloar, President and COO of Bebaio, will explore examples of brands transforming their businesses by t...
With the proliferation of connected devices underpinning new Internet of Things systems, Brandon Schulz, Director of Luxoft IoT – Retail, will be looking at the transformation of the retail customer experience in brick and mortar stores in his session at @ThingsExpo. Questions he will address include: Will beacons drop to the wayside like QR codes, or be a proximity-based profit driver? How will the customer experience change in stores of all types when everything can be instrumented and analyzed? As an area of investment, how might a retail company move towards an innovation methodolo...
The Internet of Things (IoT) is about the digitization of physical assets including sensors, devices, machines, gateways, and the network. It creates possibilities for significant value creation and new revenue generating business models via data democratization and ubiquitous analytics across IoT networks. The explosion of data in all forms in IoT requires a more robust and broader lens in order to enable smarter timely actions and better outcomes. Business operations become the key driver of IoT applications and projects. Business operations, IT, and data scientists need advanced analytics t...
SYS-CON Events announced today that HPM Networks will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. For 20 years, HPM Networks has been integrating technology solutions that solve complex business challenges. HPM Networks has designed solutions for both SMB and enterprise customers throughout the San Francisco Bay Area.
In his session at @ThingsExpo, Lee Williams, a producer of the first smartphones and tablets, will talk about how he is now applying his experience in mobile technology to the design and development of the next generation of Environmental and Sustainability Services at ETwater. He will explain how M2M controllers work through wirelessly connected remote controls; and specifically delve into a retrofit option that reverse-engineers control codes of existing conventional controller systems so they don't have to be replaced and are instantly converted to become smart, connected devices.
SYS-CON Events announced today that IceWarp will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. IceWarp, the leader of cloud and on-premise messaging, delivers secured email, chat, documents, conferencing and collaboration to today's mobile workforce, all in one unified interface
Akana has announced the availability of the new Akana Healthcare Solution. The API-driven solution helps healthcare organizations accelerate their transition to being secure, digitally interoperable businesses. It leverages the Health Level Seven International Fast Healthcare Interoperability Resources (HL7 FHIR) standard to enable broader business use of medical data. Akana developed the Healthcare Solution in response to healthcare businesses that want to increase electronic, multi-device access to health records while reducing operating costs and complying with government regulations.
For IoT to grow as quickly as analyst firms’ project, a lot is going to fall on developers to quickly bring applications to market. But the lack of a standard development platform threatens to slow growth and make application development more time consuming and costly, much like we’ve seen in the mobile space. In his session at @ThingsExpo, Mike Weiner, Product Manager of the Omega DevCloud with KORE Telematics Inc., discussed the evolving requirements for developers as IoT matures and conducted a live demonstration of how quickly application development can happen when the need to comply wit...
The Internet of Everything (IoE) brings together people, process, data and things to make networked connections more relevant and valuable than ever before – transforming information into knowledge and knowledge into wisdom. IoE creates new capabilities, richer experiences, and unprecedented opportunities to improve business and government operations, decision making and mission support capabilities.