Linux Containers Authors: Yeshim Deniz, Pat Romanski, Elizabeth White, Liz McMillan, Vaibhaw Pandey

Related Topics: Java IoT, Microservices Expo

Java IoT: Article

How to Make Developers Write Performance Tests

Week 6 of our 2010 Application Performance Almanac

I had an interesting conversation with our Test Automation team lead Stefan – who Andi interviewed for our “Eating our own Dog Food ” article – on his experiences with the willingness of developers to write performance tests.

I asked a provocative question: do developers really want to write them in the first place? First he smiled but then he said that they do. I honestly was a bit surprised because in my own experience as a developer was that I rather wanted to write code instead of tests.

He agreed that it was not that easy in the beginning; however, after the first developers started it became a more and more common practice. I then asked him what he thinks are the main reasons why developers want to write performance tests. The following sums up our experience at dynaTrace into what causes developers themselves write performance tests.

Reason 1: Everyone wants to be a hero
Developers like to write well-performing code. Even more, they like to write faster code. Just like Captain Kirk saying “I need the ship ready in 2 hours” – and Scottie replying “I will do it in 20 minutes”.

Every developer wants to show how well his code performs. The performance testing hype at dynaTrace really started when a developer sent out an email about how his performance tests showed him a regression immediately after checking in. The figure below shows exactly the image the developer sent out.

CI  Performance Graph Showing Several Performance Regressions

CI Performance Graph Showing Several Performance Regressions

The developer wanted to share his experience and also show others the benefits of the tests he wrote. A couple of weeks after this email another developer sent out an email about the improvements he achieved for a component between two releases.

Another team of developers managed to optimize the throughput of our server by up to four times. This was a really great achievement and the result of hard work – earning them the dynaTrace culture award. So their contribution was acknowledged as one of the most significant in the last quarter.

So if you want people to deliver certain results then award them for those results and give them the fame they deserve.

Reason 2: The “It will come back anyway” Experience
This one is nice as nobody believes it first. You can always tell people that if they do not get things right during development, a potential problem will come back to them later. So while performance testing might mean some additional work right now it is better than having even more work later on when you have to fix performance bugs.

As I said, nobody believes this until they experience it for the first time themselves. Then people start to write their tests to avoid running into this situation again. There is nothing more frustrating than fixing bugs – possibly in older versions – when all your colleagues are working on the cool new features.

Reason 3: He is working on my code too
At dynaTrace we have always taken performance seriously. Developers have always been profiling their code to optimize it or add additional output for performance measurement. This works fine if you are the only one working on your code and do not have any dependencies on other components. If you not earning money by writing “Hello World” applications this is not the case ; -). Other developers are working on the code base as well.

So beside the fact that your profiling results are only available to you locally, they might already be outdated after the next check-in. As you have no automated means to find those changes, you will not be aware of it. Only a standardized test suite that is automatically executed as part of your Continuous Integration environment can help here

Some people also use specific logging code which they think will help them to track down problems. While this works fine sometimes, our experience has shown that others might change the code causing part of the logging statement not to be executed anymore. The reason might be that your code just got deprecated or somebody added an additional conditional statement, so logging code now only gets executed in certain cases.

Reason 4: Only let developers write test code
This is a key lesson we learned in making performance testing successful. The more complex it is to write test code, the fewer tests are written. Especially performance tests can easily become quite complex – sometimes even more complex than the actual code you want to test. In our case, we want to start our own software, start a couple of application server instances and then trigger a load test. All this has to be managed across multiple nodes.

If we forced developers to write that code themselves they would not want to write any tests at all. So we took a massive investment in writing our own testing framework that takes away that burden from the developer. The example below shows how to start a remote WebSphere server; all done by just adding some simple annotations.

host = "lab2",
name = "WAS7.0",
startupPriority = 1,
postStartClosure = WaitForWebSphereSudIsUp.class
private SudInterface webSphereSud;

Additionally, dynaTrace itself provides extensions which developers can use to control our own software components. Developers now simply focus on writing their test cases. All this together help us to create a good test suite quickly and make it attractive to developers to write tests.

Reason 5: Provide Feedback
If you invest your time, you want to get something back. So the key to make developers write more tests is to make the test results easily accessible to them. The immediate value for the developer is to get feedback about the performance impact of his code changes very quickly. We run our performance test suite two times a day. So ideally you can fix a performance bug the same day you created it :-

Developers get this feedback via email just as build or functional test results. So they invest time once and get feedback anytime they – or somebody else – changes the code.

Reason 6: The old code threat
Not only we but also our code gets older. The older code is, the less you want to touch it. However very often you have no choice. Martin Fowler already wrote in his Refactoring book that you also start refactoring by having the proper tests in place. While Martin was talking about functional tests here, the same is true for performance tests.

The really evil thing is that if you have any modifying components which heavily interact with each other you can very easily introduce performance problems. So instead of just ruining your own code, you will also ruin the code of other developers.

We have seen especially bug fixes introducing such problems. Just as you can see in the picture above, a functional bug fix introduced a serious performance problem. Additionally you want to be fast in fixing the old code and have more time to develop the really cool new features – don’t you?

Reason 7: I might be the newb but not the noob
Stefan realized that the time when the most test cases are written is when a new developer takes over a feature. This is for two main reasons: First, you do not want to break features of other developers; Second, you are new and want to learn how the code works. The best way to do this is – except reading source code and asking former developers – by testing. So developers who take over a feature have the highest acceptance of writing tests.

Related reading:

  1. Performance Management in Continuous Integration I recently gave  presentations on Performance Management as part of...
  2. dynaTrace Application Performance Almanac 2010 Inspired by the work of Stoyan on his performance advent...
  3. Do more with Functional Testing – Take the Next Evolutionary Step Functional Testing has always been an activity done by Test...
  4. Randomizing Input Data for Visual Studio Load Tests While preparing for my presentation Load and Performance Testing: How...
  5. Eating Our Own Dog Food: How dynaTrace does Continuous APM Internally in Development with dynaTrace TA Lead Stefan Frandl I sat together with Stefan Frandl, Test Automation Lead in...

More Stories By Alois Reitbauer

Alois Reitbauer is Chief Technical Strategist at Dynatrace. He has spent most of his career building monitoring tools and fine-tuning application performance. A regular conference speaker, blogger, author, and sushi maniac, Alois currently shares his professional time between Linz, Boston, and San Francisco.

@ThingsExpo Stories
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things’). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing? IoT is not about the devices, it’s about the data consumed and generated. The devices are tools, mechanisms, conduits. In his session at Internet of Things at Cloud Expo | DXWor...