Linux Containers Authors: Liz McMillan, Elizabeth White, Pat Romanski, Yeshim Deniz, Stefana Muller

Related Topics: Linux Containers, Mobile IoT, Release Management , @CloudExpo

Linux Containers: Blog Feed Post

Amazon S3 vs Amazon EBS on the Elastic Cloud

EBS accounts can have a maximum of 20 volumes unless a higher limit is requested from Amazon

Amazon Elastic Block Storage (Amazon EBS) is a new type of storage designed specifically for Amazon EC2 instances. Amazon EBS allows you to create volumes that can be mounted as devices by EC2 instances. Amazon EBS volumes behave as if they were raw unformatted external hard drives and can be formatted using a file system such as ext3 (Linux) or NTFS (Windows) and mounted on an EC2 instance; files are accessed through the file system . They have user supplied device names and provide a block device interface.

For a 20 GB volume, Amazon estimates an annual failure rate for EBS volumes from 1-in-200 to 1-in-1000.  The failure rate increases as the size of the volume increases.  Therefore you either need to keep an up-to-date snapshot on S3, or have a backup of the contents somewhere else that you can restore quickly enough to meet your needs in the event of a failure.  

EBS accounts can have a  maximum of 20 volumes unless a higher limit is requested from Amazon. The maximum size of a volume is 1 TB and the storage on a volume is limited to the provisioned size and cannot be changed. EBS volumes can only be accessed from an EC2 instance in the same availability zone whereas snapshots on S3 can be accessed from any availability zone. 

Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers. S3 needs software to be able to read and write files but is hugely scalable, stores 6 copies of data for HA and redundancy, and is rumoured to be written in Erlang and is hugely scalable.

S3 accounts can have a maximum of 100 buckets, each with unlimited storage and an unlimited number of files. The maximum size of a single file is 5 GB.

S3 is subject to “eventual consistency”  which means that there may be a delay in writes appearing in the system whereas EBS has no consistency delays. Also EBS can only be accessed by one machine at a time whereas snapshots on S3 can be shared,  

In terms of performance S3 has the higher latency and also has higher variation in latency. S3 write latency can also be higher than read latency . EBS on the other hand has lower latency with less variation. It also has writeback caching for very low write latency. However be aware that writeback caching and out-of-order flushing could result in either an unpredictable file system or a database corruption

In terms of throughput S3 has  maximum  throughput (single threaded ) of approximately 20 MB/s or 25 MB/s  for multithreaded. This is on a small instance. This rises to 50 MB/s on the large and  extra large instances. EBS has a maximum  throughput limited by the network, This isapproximately 25 MB/s on a small instance and 50 MB/s on large instances,  and 100 MB/s on  extra large instances. As both S3 and EBS are shared resources they are subject to slowdown under heavy load.

For file listing S3 is slow and search is by prefix only  whereas EBS has fast directory listing and searching. S3 is performance optimized by using multiple buckets. The write performance is  optimized by writing keys in sorted order . EBS single volume performance is similar to a disk drive with writeback caching.

There is an alternative to EBS for EC2 and that is PersistentFS. With PersistentFS you mount a drive and use it like any other, but, and here is the crunch, the storage for the device is actually realized in many little chunks in an S3 storage bucket. PersistentFS is a closed-source product based on the FUSEapproach.

S3 costs 15 cents per GB for storage actually used and 1 cent per 10,000 GETs, and 1 cent per 1,000 PUTs. EBS costs 10 cents per GB provisioned and 1 cent per 100,000 I/O’s. For a pricing of PersistentFS and how this compares to both S3 and EBS I suggest you read this post on the Amazon forums which as posted by the PersistentFS team.

Read the original blog entry...

More Stories By Jim Liddle

Jim is CEO of Storage Made Easy. Jim is a regular blogger at SYS-CON.com since 2004, covering mobile, Grid, and Cloud Computing Topics.

IoT & Smart Cities Stories
Cloud computing delivers on-demand resources that provide businesses with flexibility and cost-savings. The challenge in moving workloads to the cloud has been the cost and complexity of ensuring the initial and ongoing security and regulatory (PCI, HIPAA, FFIEC) compliance across private and public clouds. Manual security compliance is slow, prone to human error, and represents over 50% of the cost of managing cloud applications. Determining how to automate cloud security compliance is critical...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
"MobiDev is a Ukraine-based software development company. We do mobile development, and we're specialists in that. But we do full stack software development for entrepreneurs, for emerging companies, and for enterprise ventures," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, discussed how they built...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Machine learning has taken residence at our cities' cores and now we can finally have "smart cities." Cities are a collection of buildings made to provide the structure and safety necessary for people to function, create and survive. Buildings are a pool of ever-changing performance data from large automated systems such as heating and cooling to the people that live and work within them. Through machine learning, buildings can optimize performance, reduce costs, and improve occupant comfort by ...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, we provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading...
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...