Welcome!

Linux Containers Authors: Liz McMillan, Yeshim Deniz, Elizabeth White, Pat Romanski, Stackify Blog

Related Topics: @DevOpsSummit, Linux Containers, Containers Expo Blog

@DevOpsSummit: Article

The Rule of Log for DevOps | @DevOpsSummit #DevOps #Microservices

Logs have made many heroes. I know, because I’ve been one. But let’s understand why

"This is interesting-didn't we see this same thing in the logs just before we crashed last time?"
-
an anonymous and somewhat embarrassed DevOps team member

Unfortunately the "same thing" is an obscure, otherwise uninteresting log entry, in a directory structure that's rarely checked. More unfortunate is that the "last time" was two months ago, when their eCommerce site went down on Black Friday costing hundreds of thousands of dollars, and they had noticed this same entry but just now made the correlation. Companies have made great strides in attempting to speed up correlations like this and improve MTTR. The question that always arises (usually not long after the CFO sees the price tag) goes something like, "Is it really worth deploying an agent to all of our key app servers with the sole purpose of gathering and merging disconnected artifacts (logs) with the hope of finding an interesting correlation when it counts?"

Early in my career I remember finding a "too many open files" error in MS-DOS. I knew how to fix that. As the designer and programmer, I would either be more judicious with my file handles, or do the simple thing and increase the allotment directly in the OS instance. The reality is that today, with agile dev cycles and outsourced staff, the immediate fix is not always known, and in production it gets tricky to get buy-off for a change request that you are not 100 percent confident in. Further, microservices teams are taking over the role of ops, and supporting the DevOps culture as small teams write and manage their own code well into production. Great, now we have someone who knows how to fix it right? Kind of. What about two years from now when they are redeployed to different work streams, or what if their webservice was a really small piece of a larger system that is completely down, due only in part to their code?

Logs have made many heroes. I know, because I've been one. But let's understand why. It's typically because some programmer 10 years ago put some small artifact into their crappy code that provided an inkling that the app was in a bad state. Perhaps they even told you what part of the application was in trouble or the service it was attempting to connect to. The reality is that never, not ever, does a programmer write out to a log something that's immediately actionable. If they did, they would have written the code to remedy it. That said, I would love someday to see the log entry that states,

10 SEP 2015, 05:03:23 hey guys, sorry about that poorly formed parameterization for that web service, and if you're seeing this, it's likely because I forgot to account for special characters in the password. So, please refer to line number 407 and tell dev to fix it. Oh, and you'll need to do the same thing on the other four parallel modules and restart all services.

Companies who have opted to tackle the very difficult task of understanding app server logic take much of the burden and correlation guessing games away from DevOps. Some APM vendors are now realizing the magic in automatic correlation, anomaly detection, and smart analytics based on data within the running code. Because of native code profiling techniques and process inspection, code written in Java, .NET, PHP, node.js, Python, and C/C++ can be monitored in prod without dev intervention - no code change required - and with very little overhead.

Screenshot: AppDynamics resides within appserver code to create dynamic visualizations of related systems

Given new solutions like this, do logs still matter? Yes, but much less so. It is more valuable for an ops team to know where to look, being taken directly to the exact SQL query, webservice call, method, or line number where the problem manifested itself, while the supporting system logs are still there for audit and validation. The key is context - context that ties interesting log information with what actually happened for that single, key transaction.

Screenshot: AppDynamics keeps log information associated with key business transactions in context

Looking at the question (and cost) of deploying an agent: Does it make sense to deploy a single-purpose log aggregator to all related systems? Complicate that question by containerization within VMs within clusters, and more evolved DevOps groups are finding that getting into the code is key. With logs at each container layer, OS, and app level, it's difficult to keep the context and know what's relevant, especially when scaled to any significant deployment.

So stop looking at artifacts, stop examining remnants, stop the manual correlation. The technology now exists where you can truly understand your modern apps in ways never before possible. If you're not in the code, you do not have true context, no matter how detailed the log.

More Stories By Jason Trunk

Jason Trunk is Field Chief Technology Officer at AppDynamics.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
We are seeing a major migration of enterprises applications to the cloud. As cloud and business use of real time applications accelerate, legacy networks are no longer able to architecturally support cloud adoption and deliver the performance and security required by highly distributed enterprises. These outdated solutions have become more costly and complicated to implement, install, manage, and maintain.SD-WAN offers unlimited capabilities for accessing the benefits of the cloud and Internet. ...
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and ...
Founded in 2000, Chetu Inc. is a global provider of customized software development solutions and IT staff augmentation services for software technology providers. By providing clients with unparalleled niche technology expertise and industry experience, Chetu has become the premiere long-term, back-end software development partner for start-ups, SMBs, and Fortune 500 companies. Chetu is headquartered in Plantation, Florida, with thirteen offices throughout the U.S. and abroad.
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smart...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
Michael Maximilien, better known as max or Dr. Max, is a computer scientist with IBM. At IBM Research Triangle Park, he was a principal engineer for the worldwide industry point-of-sale standard: JavaPOS. At IBM Research, some highlights include pioneering research on semantic Web services, mashups, and cloud computing, and platform-as-a-service. He joined the IBM Cloud Labs in 2014 and works closely with Pivotal Inc., to help make the Cloud Found the best PaaS.
As data explodes in quantity, importance and from new sources, the need for managing and protecting data residing across physical, virtual, and cloud environments grow with it. Managing data includes protecting it, indexing and classifying it for true, long-term management, compliance and E-Discovery. Commvault can ensure this with a single pane of glass solution – whether in a private cloud, a Service Provider delivered public cloud or a hybrid cloud environment – across the heterogeneous enter...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
"We view the cloud not as a specific technology but as a way of doing business and that way of doing business is transforming the way software, infrastructure and services are being delivered to business," explained Matthew Rosen, CEO and Director at Fusion, in this SYS-CON.tv interview at 18th Cloud Expo (http://www.CloudComputingExpo.com), held June 7-9 at the Javits Center in New York City, NY.
The Founder of NostaLab and a member of the Google Health Advisory Board, John is a unique combination of strategic thinker, marketer and entrepreneur. His career was built on the "science of advertising" combining strategy, creativity and marketing for industry-leading results. Combined with his ability to communicate complicated scientific concepts in a way that consumers and scientists alike can appreciate, John is a sought-after speaker for conferences on the forefront of healthcare science,...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
In his session at Cloud Expo, Alan Winters, U.S. Head of Business Development at MobiDev, presented a success story of an entrepreneur who has both suffered through and benefited from offshore development across multiple businesses: The smart choice, or how to select the right offshore development partner Warning signs, or how to minimize chances of making the wrong choice Collaboration, or how to establish the most effective work processes Budget control, or how to maximize project result...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...