Welcome!

Linux Containers Authors: Elizabeth White, Liz McMillan, Yeshim Deniz, Pat Romanski, Zakia Bouachraoui

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Blog Feed Post

Three Levels of Network Monitoring for DevOps By @Wall_Dirk | @DevOpsSummit #DevOps

Back in the days when individual services ran on standalone machines, measuring performance was easy

Three Levels of Network Monitoring for DevOps
By Dirk Wallerstorfer

Network communications are a top priority for DevOps teams working in support of modern globally-distributed systems and microservices. But basic network interface statistics like received and sent traffic aren’t as useful as they once were because multiple microservices may share the same network interface. For meaningful analysis, you need to dig deeper and correlate network-traffic metrics with individual processes. This is however just the beginning … In this article, I’ll show you how deep the network monitoring rabbit hole goes. The types of meaningful analysis that DevOps engineers have been waiting for are finally available.

Back in the days when individual services ran on standalone machines, measuring performance was easy. Most of a machine’s CPU, memory, and network load were dedicated to support the one service. If the response time of the service increased, it was easy to determine if the problem was processing, memory, or network related. But times have changed. Resource shortages are now largely a thing of the past. The way that applications are built, deployed, and operated is also different. And, once again, error-free network performance is a top priority.

Level 1: Host-based monitoring

Modern performance-monitoring tools provide network-related metrics by default. In addition to throughput data though, you need to know the quality of your network connections. Knowing that your host transfers a certain amount of kilobytes per second is interesting, but it’s only the beginning. For example, knowing that half of your traffic is comprised of TCP retransmissions is extremely valuable information. The amount of incoming and outgoing traffic, connectivity, and information about connection quality (i.e., number of dropped packets and retransmissions) are the metrics that serious performance monitoring tools must provide.

When compared with overall traffic patterns passing through the host NIC, such metrics can provide important insights into network quality. If there is only one service process running on a host, all the host metrics are representative of the one process. If there are several processes running, these metrics provide information about the overall availability and connection quality of all the processes. But host-based monitoring can’t show you if a process has a network problem or the amount of resources that are consumed by each process (e.g., network bandwidth). Host-based network metrics can however be good indicators that something has gone wrong in your network. The question is, who you gonna call to tell you exactly what’s gone wrong?

Standard network metrics on host level

Level 2: Process-based monitoring

Monitoring resource consumption at the process level is a more sophisticated approach. Analyzing the throughput, connectivity, and connection quality of each process is a good starting point for productive analysis.

When monitoring at the process level you might expect to see network-volume metrics like incoming and outgoing network traffic for each process (i.e., the average rate at which data is transmitted to and from the process during a given time interval). But such volume-based metrics alone aren’t sufficient for meaningful analysis because they don’t tell you anything about the communication behavior of the process. If you take the number of TCP requests into account you have a three-dimensional model of process characteristics. High network traffic and few TCP requests can indicate, for example, an FTP server providing large files. Low traffic and many requests can indicate a service that has a small data footprint (e.g., an authentication service). If you only monitor network traffic volume, you won’t be able to tell the difference between an occasionally used, throttled FTP server and a frequently used web service. Clearly, the number of processed TCP requests is essential. You can use the combined network volume information to check your architectural design and expectations against empirical data and identify issues if something hasn’t worked as planned, or is getting out of hand.

The rate of properly established TCP connections, both inbound and outbound, is representative of the connection availability of a process. The number of refused and timed-out TCP connections per second need to be included in an integrated view that’s focused on process connectivity. With this information you can easily identify connectivity problems. Closed ports or full queues of pending connections can be the cause of connection refusals. Firewalls that don’t send a TCP reject or ICMP errors and hosts that die during transmissions can be reasons for timeouts.

In addition to quantitative data, a qualitative analysis of network connections is necessary for providing a holistic view of the network properties of a process. Assessing TCP retransmissions, round-trip times, and the effective use of network bandwidth provide additional insights. Opposing host and process retransmission rates can help in identifying the source of network connection problems.

Round-trip times are an important measure, especially when clients from remote locations or hosts in different availability zones play a role. The most precise measurement is handshake round-trip time measured during TCP session establishment. With persistent connections, for example in the backend of an application infrastructure, these handshakes occur rarely. Round-trip time during data transfer isn’t as accurate but it reveals fluctuations in network latency. Typically these values don’t exceed a few milliseconds for hosts on the same LAN and 50-100ms for geographically close nodes from different networks.

Apart from nominal network interface speed, the actual throughput that a process can realize is interesting data. Regardless of how fast a process responds, when large quantities of data need to be transferred, the bandwidth that is available to the process is the limiting factor. Keeping in mind that the network interface of the host running the process, the local network, and the Internet are shared resources, there are dozens of things that can affect data transfer and cause fluctuations over time. Average transfer speed per client session under current network conditions is vital information.

Obviously, having all this information about the quality of your network connections is useful and can provide exceptionally deep insights. Ultimately, this information enables you to pinpoint the exact processes that are having network problems. However, one piece of the puzzle is still missing: It takes two communicating parties to produce any sort of networking problem. Wouldn’t it be good to know what’s going on on the remote side of the network as well?

Network metrics on the process level

Level 3: Communications-based monitoring

Although network monitoring on the process level is innovative, you need more to properly diagnose and troubleshoot problems that can occur between the components of your application infrastructure. To get the best out of network monitoring you have to monitor the volume and quality of communication between processes. Only then can you unambiguously identify process pairs that have, for example, high traffic or connectivity problems.

With this approach you can check the bandwidth usage on both ends of a communication and identify which end might be the bottleneck. You can also single out process pairs that have connectivity problems or numerous TCP retransmissions. This obviously is way faster and less error-prone than manual checks on both sides. Aside from network overlays and SDN, you can pinpoint erroneous connections down to a level where you can start doing health checks on cables and switch ports because you know exactly which components participate in the conversation.

Monitoring volume and quality of network connections on the process/communications level makes detecting and resolving issues easier, more efficient, and more comfortable. It’s a bit like geotagging for network problems in your application environment.

The Ruxit way of network monitoring
Ruxit provides host-based network metrics and detailed metrics about each monitored process. Ruxit even goes the extra mile, providing all necessary information related to process-to-process communication.

Process connections overview

It shows all processes running on a host in the middle of the connection graph. On the left side you see all incoming connections from other processes. On the far left you see the hosts where these processes are running. To the right, you see all outgoing connections to other processes. On the far right, you see the hosts where the processes are running. You can select any process within that view to view traffic, connectivity, and retransmission rates for each conversation.

You can even use this feature to verify your software architecture and infrastructure. Are all processes talking to the processes they are supposed to talk to? Are all necessary services connected to load balancers? Is network bandwidth causing a performance bottleneck between my web and database servers?

This is the sort of information that DevOps engineers need. This feature eliminates the need to comb through log files and use tcpdump to locate networking problems and identify affected hosts and processes. It helps saving time and sparing nerves when analyzing networking problems. For insights into how networking errors can impact the performance of services, have a look at my recent article: Detecting network errors and their impact on services.

The post 3 levels of network monitoring for DevOps appeared first on #monitoringlife.

Read the original blog entry...

More Stories By Dynatrace Blog

Building a revolutionary approach to software performance monitoring takes an extraordinary team. With decades of combined experience and an impressive history of disruptive innovation, that’s exactly what we ruxit has.

Get to know ruxit, and get to know the future of data analytics.

IoT & Smart Cities Stories
In this Women in Technology Power Panel at 15th Cloud Expo, moderated by Anne Plese, Senior Consultant, Cloud Product Marketing at Verizon Enterprise, Esmeralda Swartz, CMO at MetraTech; Evelyn de Souza, Data Privacy and Compliance Strategy Leader at Cisco Systems; Seema Jethani, Director of Product Management at Basho Technologies; Victoria Livschitz, CEO of Qubell Inc.; Anne Hungate, Senior Director of Software Quality at DIRECTV, discussed what path they took to find their spot within the tec...
To Really Work for Enterprises, MultiCloud Adoption Requires Far Better and Inclusive Cloud Monitoring and Cost Management … But How? Overwhelmingly, even as enterprises have adopted cloud computing and are expanding to multi-cloud computing, IT leaders remain concerned about how to monitor, manage and control costs across hybrid and multi-cloud deployments. It’s clear that traditional IT monitoring and management approaches, designed after all for on-premises data centers, are falling short in ...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true ...
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
We are seeing a major migration of enterprises applications to the cloud. As cloud and business use of real time applications accelerate, legacy networks are no longer able to architecturally support cloud adoption and deliver the performance and security required by highly distributed enterprises. These outdated solutions have become more costly and complicated to implement, install, manage, and maintain.SD-WAN offers unlimited capabilities for accessing the benefits of the cloud and Internet. ...
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and G...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
DXWorldEXPO LLC announced today that "IoT Now" was named media sponsor of CloudEXPO | DXWorldEXPO 2018 New York, which will take place on November 11-13, 2018 in New York City, NY. IoT Now explores the evolving opportunities and challenges facing CSPs, and it passes on some lessons learned from those who have taken the first steps in next-gen IoT services.
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...