Welcome!

Linux Containers Authors: Tim Hinds, AppDynamics Blog, Robert Reeves, Automic Blog, Liz McMillan

Related Topics: Linux Containers, Microservices Expo

Linux Containers: Blog Feed Post

How to Improve Server Performance by IO Tuning – Part 2

How is your IO characterized?

Before reading this article, I highly suggest to get familiar with all the concepts of the previous article regarding IO tuning.

Your IO please, sir
How is your IO characterized? Yes, this question has to be asked yet again. It’s a big difference when tuning for random access reads vs. sequential reads.

How is your application doing in that matter? – You should know better than me.
My main approach when optimizing for read IO is to access the disks as little as possible. Disks are slow, really slow, comparing to CPU and RAM – if we can – we avoid them.

Write some proper code
Have a look at a use case of reading large chunks from your HDs and serving them via the network for various users.

Obviously reading the file on demand will save memory, however, it will increase your latency and cause a bad user experience. The secret is here is to:

  1. Prefetch
  2. Prefetch some more
  3. Try to expect what should be prefetched

Prefetching (or buffering) the actual data you are going to serve can offload your HDs by reading the data just once and serving it multiple times.

However, when your IO is random and you can’t really expect what’s going to be fetched – you can still statistically optimize your cache hits by buffering as much as data as you can in memory.

Unfortunately for random IO, in the end of the day, expect your HDs to work and also expect the extra delay.

Ah and of course – if the “disk grinding” application is actually a database, then obviously avoiding full table scans and adding proper indexes where needed are the key for better performance in that matter.

Linux to the rescue?
I talked about IO schedulers in the previous article, together with filesystems, they affect your performance not only when writing, but also while reading.

What I did want to mention here, is that Linux does a pretty good job with its buffer cache – ever wondered how come you never have any free memory under Linux? – That is because Linux is doing an awesome job for you – caching disk IO for you without even asking you. So if you didn’t implement proper caches in your application, then Linux might take care of it for you.

Underdogs
TMPFS and RAMFS
are, for some reason, only rarely used. These are in-memory filesystems. They can be great for caching files and making sure they stay in your RAM for ultra-fast access.

This is yet another countermeasure we can undertake in order to avoid the slowness of HDs.

Yet another tuneable
For some awkward reason, the ‘noatime’ and ‘nodiratime’ attributes are not on by default on Linux filesystems.

You should switch them on, as it’ll trigger a filesystem metadata write every time you access a file or directory – yes, also for reading. If I disabled it on my humble netbook (actually for battery saving concerns – have the HD spin less) – you must disable it also on your server.

I encourage you to do it now – also on your desktop:

 # vim /etc/fstab 

Just add ‘noatime’ and ‘nodiratime’ for any non-swap filesystem.

Remember!
If you have a lot of memory in your system – then use it.

Many are the times I’ve seen people struggle with a database in the size of 10GB which has either multiple reads or writes or both of them together.

The simple solution is just to have the whole DB in-memory on a proper machine with more than 10GB or memory. Trust me – no matter what HD setup you have below – if you have 100% of your databases in-memory – HDs will never slow you down.

Consider it also for databases larger than 10GB. Unfortunately during my work experience I’ve seen what happened with a fairly optimized DB in the size of 100GB. It had gazillion (OK, maybe I’m exaggerating here) of updates and queries per minute. Jumping from 8GB of RAM to 64GB made the huge difference between a system that didn’t work to a system that actually copes with the load.

In most cases, if your system is that much loaded and you justify these huge amounts of memory – you are also probably making enough money to actually afford it.

RAID controllers
Choosing a proper RAID controller can yield a very nice IO boost in terms of read performance. And I’m going to delve a bit into RAID1 here, in order to explain this concept.

RAID1 simply mirrors your data over 2 (or more) HDs, usually to provide redundancy. Say one of your HDs is broken – your system can continue to operate from the other.

RAID1 is usually considered “wasteful” because you don’t get extra storage for every HD you add. However, with RAID1 and a proper RAID controller, you can get tremendous read performance boosts.

How come? Imagine you have 10 read IO requests and one HD to serve them – they might get ordered sequentially and served optimally, however, only one HD will serve them.

Lets imagine you now have 10 HDs in a RAID1 and the same 10 read IO requests – you now have 10 HDs to serve them!! Theoretically speaking it’ll be 10 times faster than one HD.

That is with an exception of having a proper RAID controller – some cheaper controllers wouldn’t provide you the desired behavior of reading from multiple disks at the same time.

Be cautious when building a RAID1 configuration – the write speed of the array will be the write speed of the slowest HD in the array – it will decrease write performance noticeably if you have slower disks in the array.

Another word about RAID1 vs. RAID10 – RAID1 can be easily expanded – whenever you notice a performance problem – just chuck another HD in the RAID array and you’re good to go with some more performance!

Hands up, don’t move!
SSDs
can yield a big performance boost if you care mainly about reading.
They are, however, 10 times more expensive – but also 10 times or more faster when talking about random access of data.

How come? – there’s nothing turning over there, nothing is moving – all of your stored data is “in the same distance” from you. While on traditional mechanical magnetic HDs you have to actually wait for a head to move to the correct location in order to retrieve any data.

It is a sort of a last resort optimization in my opinion, as it can easily double or triple your expenses.

One last dirty trick
Mechanical disks turn, some at 7.2K RPM, some at 10K and some even at 15K.

In addition to turning, they have a head, pointing to the location with the desired data.

For a given HD, the angular velocity of the head on the HD plate is constant, be it 7.2K, 10K or 15K RPM.

However, going further out on the HD plate, the absolute velocity of the head increases – as the plate radius is bigger and the angular velocity stays the same. This is simple physics.
For a more thorough explanation I suggest reading this brilliant article.

What does it mean for us? Well, in order to squeeze the best out of our HDs, we can use just the outer parts of its plate. Usually the outermost tracks of a HD are in the beginning of it, regarding partition creation.

Your toolbox
The bag of tools I published in the previous article are also just as good over here.

Among them are obviously Monitis and iostat.

Examining your application in the applicative level is highly recommended as well. Provide yourself verbose logs and see where you hit the caches and where you miss them. Your task, obviously, is to minimize the cache misses to a bare minimum.

That’s all folks
Truly these two IO tuning articles are not a mere grocery-store tick list you can just perform and gain an extra boost – these articles were meant to give you tools to think, plan, design and eventually carry out successful architectures. Good luck!

Read the original blog entry...

More Stories By Hovhannes Avoyan

Hovhannes Avoyan is the CEO of PicsArt, Inc.,

@ThingsExpo Stories
There is an ever-growing explosion of new devices that are connected to the Internet using “cloud” solutions. This rapid growth is creating a massive new demand for efficient access to data. And it’s not just about connecting to that data anymore. This new demand is bringing new issues and challenges and it is important for companies to scale for the coming growth. And with that scaling comes the need for greater security, gathering and data analysis, storage, connectivity and, of course, the...
Artificial Intelligence has the potential to massively disrupt IoT. In his session at 18th Cloud Expo, AJ Abdallat, CEO of Beyond AI, will discuss what the five main drivers are in Artificial Intelligence that could shape the future of the Internet of Things. AJ Abdallat is CEO of Beyond AI. He has over 20 years of management experience in the fields of artificial intelligence, sensors, instruments, devices and software for telecommunications, life sciences, environmental monitoring, process...
Manufacturers are embracing the Industrial Internet the same way consumers are leveraging Fitbits – to improve overall health and wellness. Both can provide consistent measurement, visibility, and suggest performance improvements customized to help reach goals. Fitbit users can view real-time data and make adjustments to increase their activity. In his session at @ThingsExpo, Mark Bernardo Professional Services Leader, Americas, at GE Digital, will discuss how leveraging the Industrial Interne...
The IoTs will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm and share the must-have mindsets for removing complexity from the development proc...
The IETF draft standard for M2M certificates is a security solution specifically designed for the demanding needs of IoT/M2M applications. In his session at @ThingsExpo, Brian Romansky, VP of Strategic Technology at TrustPoint Innovation, will explain how M2M certificates can efficiently enable confidentiality, integrity, and authenticity on highly constrained devices.
Increasing IoT connectivity is forcing enterprises to find elegant solutions to organize and visualize all incoming data from these connected devices with re-configurable dashboard widgets to effectively allow rapid decision-making for everything from immediate actions in tactical situations to strategic analysis and reporting. In his session at 18th Cloud Expo, Shikhir Singh, Senior Developer Relations Manager at Sencha, will discuss how to create HTML5 dashboards that interact with IoT devic...
We're entering the post-smartphone era, where wearable gadgets from watches and fitness bands to glasses and health aids will power the next technological revolution. With mass adoption of wearable devices comes a new data ecosystem that must be protected. Wearables open new pathways that facilitate the tracking, sharing and storing of consumers’ personal health, location and daily activity data. Consumers have some idea of the data these devices capture, but most don’t realize how revealing and...
SYS-CON Events announced today that Ericsson has been named “Gold Sponsor” of SYS-CON's @ThingsExpo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. Ericsson is a world leader in the rapidly changing environment of communications technology – providing equipment, software and services to enable transformation through mobility. Some 40 percent of global mobile traffic runs through networks we have supplied. More than 1 billion subscribers around the world re...
We’ve worked with dozens of early adopters across numerous industries and will debunk common misperceptions, which starts with understanding that many of the connected products we’ll use over the next 5 years are already products, they’re just not yet connected. With an IoT product, time-in-market provides much more essential feedback than ever before. Innovation comes from what you do with the data that the connected product provides in order to enhance the customer experience and optimize busi...
The increasing popularity of the Internet of Things necessitates that our physical and cognitive relationship with wearable technology will change rapidly in the near future. This advent means logging has become a thing of the past. Before, it was on us to track our own data, but now that data is automatically available. What does this mean for mHealth and the "connected" body? In her session at @ThingsExpo, Lisa Calkins, CEO and co-founder of Amadeus Consulting, will discuss the impact of wea...
trust and privacy in their ecosystem. Assurance and protection of device identity, secure data encryption and authentication are the key security challenges organizations are trying to address when integrating IoT devices. This holds true for IoT applications in a wide range of industries, for example, healthcare, consumer devices, and manufacturing. In his session at @ThingsExpo, Lancen LaChance, vice president of product management, IoT solutions at GlobalSign, will teach IoT developers how t...
A critical component of any IoT project is the back-end systems that capture data from remote IoT devices and structure it in a way to answer useful questions. Traditional data warehouse and analytical systems are mature technologies that can be used to handle large data sets, but they are not well suited to many IoT-scale products and the need for real-time insights. At Fuze, we have developed a backend platform as part of our mobility-oriented cloud service that uses Big Data-based approache...
In his session at @ThingsExpo, Chris Klein, CEO and Co-founder of Rachio, will discuss next generation communities that are using IoT to create more sustainable, intelligent communities. One example is Sterling Ranch, a 10,000 home development that – with the help of Siemens – will integrate IoT technology into the community to provide residents with energy and water savings as well as intelligent security. Everything from stop lights to sprinkler systems to building infrastructures will run ef...
Digital payments using wearable devices such as smart watches, fitness trackers, and payment wristbands are an increasing area of focus for industry participants, and consumer acceptance from early trials and deployments has encouraged some of the biggest names in technology and banking to continue their push to drive growth in this nascent market. Wearable payment systems may utilize near field communication (NFC), radio frequency identification (RFID), or quick response (QR) codes and barcodes...
Whether your IoT service is connecting cars, homes, appliances, wearable, cameras or other devices, one question hangs in the balance – how do you actually make money from this service? The ability to turn your IoT service into profit requires the ability to create a monetization strategy that is flexible, scalable and working for you in real-time. It must be a transparent, smoothly implemented strategy that all stakeholders – from customers to the board – will be able to understand and comprehe...
You deployed your app with the Bluemix PaaS and it's gaining some serious traction, so it's time to make some tweaks. Did you design your application in a way that it can scale in the cloud? Were you even thinking about the cloud when you built the app? If not, chances are your app is going to break. Check out this webcast to learn various techniques for designing applications that will scale successfully in Bluemix, for the confidence you need to take your apps to the next level and beyond.
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus inter...
So, you bought into the current machine learning craze and went on to collect millions/billions of records from this promising new data source. Now, what do you do with them? Too often, the abundance of data quickly turns into an abundance of problems. How do you extract that "magic essence" from your data without falling into the common pitfalls? In her session at @ThingsExpo, Natalia Ponomareva, Software Engineer at Google, will provide tips on how to be successful in large scale machine lear...
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry's single source for the cloud. Fusion's advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including cloud...