Click here to close now.




















Welcome!

Linux Containers Authors: Yeshim Deniz, Liz McMillan, Carmen Gonzalez, Samuel Scott, Tim Hinds

Related Topics: Linux Containers, Microservices Expo

Linux Containers: Blog Feed Post

How to Improve Server Performance by IO Tuning – Part 2

How is your IO characterized?

Before reading this article, I highly suggest to get familiar with all the concepts of the previous article regarding IO tuning.

Your IO please, sir
How is your IO characterized? Yes, this question has to be asked yet again. It’s a big difference when tuning for random access reads vs. sequential reads.

How is your application doing in that matter? – You should know better than me.
My main approach when optimizing for read IO is to access the disks as little as possible. Disks are slow, really slow, comparing to CPU and RAM – if we can – we avoid them.

Write some proper code
Have a look at a use case of reading large chunks from your HDs and serving them via the network for various users.

Obviously reading the file on demand will save memory, however, it will increase your latency and cause a bad user experience. The secret is here is to:

  1. Prefetch
  2. Prefetch some more
  3. Try to expect what should be prefetched

Prefetching (or buffering) the actual data you are going to serve can offload your HDs by reading the data just once and serving it multiple times.

However, when your IO is random and you can’t really expect what’s going to be fetched – you can still statistically optimize your cache hits by buffering as much as data as you can in memory.

Unfortunately for random IO, in the end of the day, expect your HDs to work and also expect the extra delay.

Ah and of course – if the “disk grinding” application is actually a database, then obviously avoiding full table scans and adding proper indexes where needed are the key for better performance in that matter.

Linux to the rescue?
I talked about IO schedulers in the previous article, together with filesystems, they affect your performance not only when writing, but also while reading.

What I did want to mention here, is that Linux does a pretty good job with its buffer cache – ever wondered how come you never have any free memory under Linux? – That is because Linux is doing an awesome job for you – caching disk IO for you without even asking you. So if you didn’t implement proper caches in your application, then Linux might take care of it for you.

Underdogs
TMPFS and RAMFS
are, for some reason, only rarely used. These are in-memory filesystems. They can be great for caching files and making sure they stay in your RAM for ultra-fast access.

This is yet another countermeasure we can undertake in order to avoid the slowness of HDs.

Yet another tuneable
For some awkward reason, the ‘noatime’ and ‘nodiratime’ attributes are not on by default on Linux filesystems.

You should switch them on, as it’ll trigger a filesystem metadata write every time you access a file or directory – yes, also for reading. If I disabled it on my humble netbook (actually for battery saving concerns – have the HD spin less) – you must disable it also on your server.

I encourage you to do it now – also on your desktop:

 # vim /etc/fstab 

Just add ‘noatime’ and ‘nodiratime’ for any non-swap filesystem.

Remember!
If you have a lot of memory in your system – then use it.

Many are the times I’ve seen people struggle with a database in the size of 10GB which has either multiple reads or writes or both of them together.

The simple solution is just to have the whole DB in-memory on a proper machine with more than 10GB or memory. Trust me – no matter what HD setup you have below – if you have 100% of your databases in-memory – HDs will never slow you down.

Consider it also for databases larger than 10GB. Unfortunately during my work experience I’ve seen what happened with a fairly optimized DB in the size of 100GB. It had gazillion (OK, maybe I’m exaggerating here) of updates and queries per minute. Jumping from 8GB of RAM to 64GB made the huge difference between a system that didn’t work to a system that actually copes with the load.

In most cases, if your system is that much loaded and you justify these huge amounts of memory – you are also probably making enough money to actually afford it.

RAID controllers
Choosing a proper RAID controller can yield a very nice IO boost in terms of read performance. And I’m going to delve a bit into RAID1 here, in order to explain this concept.

RAID1 simply mirrors your data over 2 (or more) HDs, usually to provide redundancy. Say one of your HDs is broken – your system can continue to operate from the other.

RAID1 is usually considered “wasteful” because you don’t get extra storage for every HD you add. However, with RAID1 and a proper RAID controller, you can get tremendous read performance boosts.

How come? Imagine you have 10 read IO requests and one HD to serve them – they might get ordered sequentially and served optimally, however, only one HD will serve them.

Lets imagine you now have 10 HDs in a RAID1 and the same 10 read IO requests – you now have 10 HDs to serve them!! Theoretically speaking it’ll be 10 times faster than one HD.

That is with an exception of having a proper RAID controller – some cheaper controllers wouldn’t provide you the desired behavior of reading from multiple disks at the same time.

Be cautious when building a RAID1 configuration – the write speed of the array will be the write speed of the slowest HD in the array – it will decrease write performance noticeably if you have slower disks in the array.

Another word about RAID1 vs. RAID10 – RAID1 can be easily expanded – whenever you notice a performance problem – just chuck another HD in the RAID array and you’re good to go with some more performance!

Hands up, don’t move!
SSDs
can yield a big performance boost if you care mainly about reading.
They are, however, 10 times more expensive – but also 10 times or more faster when talking about random access of data.

How come? – there’s nothing turning over there, nothing is moving – all of your stored data is “in the same distance” from you. While on traditional mechanical magnetic HDs you have to actually wait for a head to move to the correct location in order to retrieve any data.

It is a sort of a last resort optimization in my opinion, as it can easily double or triple your expenses.

One last dirty trick
Mechanical disks turn, some at 7.2K RPM, some at 10K and some even at 15K.

In addition to turning, they have a head, pointing to the location with the desired data.

For a given HD, the angular velocity of the head on the HD plate is constant, be it 7.2K, 10K or 15K RPM.

However, going further out on the HD plate, the absolute velocity of the head increases – as the plate radius is bigger and the angular velocity stays the same. This is simple physics.
For a more thorough explanation I suggest reading this brilliant article.

What does it mean for us? Well, in order to squeeze the best out of our HDs, we can use just the outer parts of its plate. Usually the outermost tracks of a HD are in the beginning of it, regarding partition creation.

Your toolbox
The bag of tools I published in the previous article are also just as good over here.

Among them are obviously Monitis and iostat.

Examining your application in the applicative level is highly recommended as well. Provide yourself verbose logs and see where you hit the caches and where you miss them. Your task, obviously, is to minimize the cache misses to a bare minimum.

That’s all folks
Truly these two IO tuning articles are not a mere grocery-store tick list you can just perform and gain an extra boost – these articles were meant to give you tools to think, plan, design and eventually carry out successful architectures. Good luck!

Read the original blog entry...

More Stories By Hovhannes Avoyan

Hovhannes Avoyan is the CEO of Monitis, Inc., a provider of on-demand systems management and monitoring software to 50,000 users spanning small businesses and Fortune 500 companies.

Prior to Monitis, he served as General Manager and Director of Development at prominent web portal Lycos Europe, where he grew the Lycos Armenia group from 30 people to over 200, making it the company's largest development center. Prior to Lycos, Avoyan was VP of Technology at Brience, Inc. (based in San Francisco and acquired by Syniverse), which delivered mobile internet content solutions to companies like Cisco, Ingram Micro, Washington Mutual, Wyndham Hotels , T-Mobile , and CNN. Prior to that, he served as the founder and CEO of CEDIT ltd., which was acquired by Brience. A 24 year veteran of the software industry, he also runs Sourcio cjsc, an IT consulting company and startup incubator specializing in web 2.0 products and open-source technologies.

Hovhannes is a senior lecturer at the American Univeristy of Armenia and has been a visiting lecturer at San Francisco State University. He is a graduate of Bertelsmann University.

@ThingsExpo Stories
Containers are not new, but renewed commitments to performance, flexibility, and agility have propelled them to the top of the agenda today. By working without the need for virtualization and its overhead, containers are seen as the perfect way to deploy apps and services across multiple clouds. Containers can handle anything from file types to operating systems and services, including microservices. What are microservices? Unlike what the name implies, microservices are not necessarily small, but are focused on specific tasks. The ability for developers to deploy multiple containers – thous...
SYS-CON Events announced today that IceWarp will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. IceWarp, the leader of cloud and on-premise messaging, delivers secured email, chat, documents, conferencing and collaboration to today's mobile workforce, all in one unified interface
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo, November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be.
Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome,” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.
The 3rd International WebRTC Summit, to be held Nov. 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA, announces that its Call for Papers is now open. Topics include all aspects of improving IT delivery by eliminating waste through automated business models leveraging cloud technologies. WebRTC Summit is co-located with 15th International Cloud Expo, 6th International Big Data Expo, 3rd International DevOps Summit and 2nd Internet of @ThingsExpo. WebRTC (Web-based Real-Time Communication) is an open source project supported by Google, Mozilla and Opera that aims to enable bro...
As more and more data is generated from a variety of connected devices, the need to get insights from this data and predict future behavior and trends is increasingly essential for businesses. Real-time stream processing is needed in a variety of different industries such as Manufacturing, Oil and Gas, Automobile, Finance, Online Retail, Smart Grids, and Healthcare. Azure Stream Analytics is a fully managed distributed stream computation service that provides low latency, scalable processing of streaming data in the cloud with an enterprise grade SLA. It features built-in integration with Azur...
With the proliferation of connected devices underpinning new Internet of Things systems, Brandon Schulz, Director of Luxoft IoT – Retail, will be looking at the transformation of the retail customer experience in brick and mortar stores in his session at @ThingsExpo. Questions he will address include: Will beacons drop to the wayside like QR codes, or be a proximity-based profit driver? How will the customer experience change in stores of all types when everything can be instrumented and analyzed? As an area of investment, how might a retail company move towards an innovation methodolo...
SYS-CON Events announced today that HPM Networks will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. For 20 years, HPM Networks has been integrating technology solutions that solve complex business challenges. HPM Networks has designed solutions for both SMB and enterprise customers throughout the San Francisco Bay Area.
SYS-CON Events announced today the Containers & Microservices Bootcamp, being held November 3-4, 2015, in conjunction with 17th Cloud Expo, @ThingsExpo, and @DevOpsSummit at the Santa Clara Convention Center in Santa Clara, CA. This is your chance to get started with the latest technology in the industry. Combined with real-world scenarios and use cases, the Containers and Microservices Bootcamp, led by Janakiram MSV, a Microsoft Regional Director, will include presentations as well as hands-on demos and comprehensive walkthroughs.
Contrary to mainstream media attention, the multiple possibilities of how consumer IoT will transform our everyday lives aren’t the only angle of this headline-gaining trend. There’s a huge opportunity for “industrial IoT” and “Smart Cities” to impact the world in the same capacity – especially during critical situations. For example, a community water dam that needs to release water can leverage embedded critical communications logic to alert the appropriate individuals, on the right device, as soon as they are needed to take action.
WebRTC services have already permeated corporate communications in the form of videoconferencing solutions. However, WebRTC has the potential of going beyond and catalyzing a new class of services providing more than calls with capabilities such as mass-scale real-time media broadcasting, enriched and augmented video, person-to-machine and machine-to-machine communications. In his session at @ThingsExpo, Luis Lopez, CEO of Kurento, will introduce the technologies required for implementing these ideas and some early experiments performed in the Kurento open source software community in areas ...
SYS-CON Events announced today that Pythian, a global IT services company specializing in helping companies leverage disruptive technologies to optimize revenue-generating systems, has been named “Bronze Sponsor” of SYS-CON's 17th Cloud Expo, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Founded in 1997, Pythian is a global IT services company that helps companies compete by adopting disruptive technologies such as cloud, Big Data, advanced analytics, and DevOps to advance innovation and increase agility. Specializing in designing, imple...
Consumer IoT applications provide data about the user that just doesn’t exist in traditional PC or mobile web applications. This rich data, or “context,” enables the highly personalized consumer experiences that characterize many consumer IoT apps. This same data is also providing brands with unprecedented insight into how their connected products are being used, while, at the same time, powering highly targeted engagement and marketing opportunities. In his session at @ThingsExpo, Nathan Treloar, President and COO of Bebaio, will explore examples of brands transforming their businesses by t...
In his session at @ThingsExpo, Lee Williams, a producer of the first smartphones and tablets, will talk about how he is now applying his experience in mobile technology to the design and development of the next generation of Environmental and Sustainability Services at ETwater. He will explain how M2M controllers work through wirelessly connected remote controls; and specifically delve into a retrofit option that reverse-engineers control codes of existing conventional controller systems so they don't have to be replaced and are instantly converted to become smart, connected devices.
With the Apple Watch making its way onto wrists all over the world, it’s only a matter of time before it becomes a staple in the workplace. In fact, Forrester reported that 68 percent of technology and business decision-makers characterize wearables as a top priority for 2015. Recognizing their business value early on, FinancialForce.com was the first to bring ERP to wearables, helping streamline communication across front and back office functions. In his session at @ThingsExpo, Kevin Roberts, GM of Platform at FinancialForce.com, will discuss the value of business applications on wearable ...
SYS-CON Events announced today that Micron Technology, Inc., a global leader in advanced semiconductor systems, will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Micron’s broad portfolio of high-performance memory technologies – including DRAM, NAND and NOR Flash – is the basis for solid state drives, modules, multichip packages and other system solutions. Backed by more than 35 years of technology leadership, Micron's memory solutions enable the world's most innovative computing, consumer,...
17th Cloud Expo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS – software, platform, and infrastructure as a service.
As more intelligent IoT applications shift into gear, they’re merging into the ever-increasing traffic flow of the Internet. It won’t be long before we experience bottlenecks, as IoT traffic peaks during rush hours. Organizations that are unprepared will find themselves by the side of the road unable to cross back into the fast lane. As billions of new devices begin to communicate and exchange data – will your infrastructure be scalable enough to handle this new interconnected world?
While many app developers are comfortable building apps for the smartphone, there is a whole new world out there. In his session at @ThingsExpo, Narayan Sainaney, Co-founder and CTO of Mojio, will discuss how the business case for connected car apps is growing and, with open platform companies having already done the heavy lifting, there really is no barrier to entry.
SYS-CON Events announced today that the "Second Containers & Microservices Expo" will take place November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Containers and microservices have become topics of intense interest throughout the cloud developer and enterprise IT communities.