Linux Containers Authors: Zakia Bouachraoui, Yeshim Deniz, Elizabeth White, Liz McMillan, Pat Romanski

Related Topics: Java IoT, Microservices Expo

Java IoT: Article

How to Make Developers Write Performance Tests

Week 6 of our 2010 Application Performance Almanac

I had an interesting conversation with our Test Automation team lead Stefan – who Andi interviewed for our “Eating our own Dog Food ” article – on his experiences with the willingness of developers to write performance tests.

I asked a provocative question: do developers really want to write them in the first place? First he smiled but then he said that they do. I honestly was a bit surprised because in my own experience as a developer was that I rather wanted to write code instead of tests.

He agreed that it was not that easy in the beginning; however, after the first developers started it became a more and more common practice. I then asked him what he thinks are the main reasons why developers want to write performance tests. The following sums up our experience at dynaTrace into what causes developers themselves write performance tests.

Reason 1: Everyone wants to be a hero
Developers like to write well-performing code. Even more, they like to write faster code. Just like Captain Kirk saying “I need the ship ready in 2 hours” – and Scottie replying “I will do it in 20 minutes”.

Every developer wants to show how well his code performs. The performance testing hype at dynaTrace really started when a developer sent out an email about how his performance tests showed him a regression immediately after checking in. The figure below shows exactly the image the developer sent out.

CI  Performance Graph Showing Several Performance Regressions

CI Performance Graph Showing Several Performance Regressions

The developer wanted to share his experience and also show others the benefits of the tests he wrote. A couple of weeks after this email another developer sent out an email about the improvements he achieved for a component between two releases.

Another team of developers managed to optimize the throughput of our server by up to four times. This was a really great achievement and the result of hard work – earning them the dynaTrace culture award. So their contribution was acknowledged as one of the most significant in the last quarter.

So if you want people to deliver certain results then award them for those results and give them the fame they deserve.

Reason 2: The “It will come back anyway” Experience
This one is nice as nobody believes it first. You can always tell people that if they do not get things right during development, a potential problem will come back to them later. So while performance testing might mean some additional work right now it is better than having even more work later on when you have to fix performance bugs.

As I said, nobody believes this until they experience it for the first time themselves. Then people start to write their tests to avoid running into this situation again. There is nothing more frustrating than fixing bugs – possibly in older versions – when all your colleagues are working on the cool new features.

Reason 3: He is working on my code too
At dynaTrace we have always taken performance seriously. Developers have always been profiling their code to optimize it or add additional output for performance measurement. This works fine if you are the only one working on your code and do not have any dependencies on other components. If you not earning money by writing “Hello World” applications this is not the case ; -). Other developers are working on the code base as well.

So beside the fact that your profiling results are only available to you locally, they might already be outdated after the next check-in. As you have no automated means to find those changes, you will not be aware of it. Only a standardized test suite that is automatically executed as part of your Continuous Integration environment can help here

Some people also use specific logging code which they think will help them to track down problems. While this works fine sometimes, our experience has shown that others might change the code causing part of the logging statement not to be executed anymore. The reason might be that your code just got deprecated or somebody added an additional conditional statement, so logging code now only gets executed in certain cases.

Reason 4: Only let developers write test code
This is a key lesson we learned in making performance testing successful. The more complex it is to write test code, the fewer tests are written. Especially performance tests can easily become quite complex – sometimes even more complex than the actual code you want to test. In our case, we want to start our own software, start a couple of application server instances and then trigger a load test. All this has to be managed across multiple nodes.

If we forced developers to write that code themselves they would not want to write any tests at all. So we took a massive investment in writing our own testing framework that takes away that burden from the developer. The example below shows how to start a remote WebSphere server; all done by just adding some simple annotations.

host = "lab2",
name = "WAS7.0",
startupPriority = 1,
postStartClosure = WaitForWebSphereSudIsUp.class
private SudInterface webSphereSud;

Additionally, dynaTrace itself provides extensions which developers can use to control our own software components. Developers now simply focus on writing their test cases. All this together help us to create a good test suite quickly and make it attractive to developers to write tests.

Reason 5: Provide Feedback
If you invest your time, you want to get something back. So the key to make developers write more tests is to make the test results easily accessible to them. The immediate value for the developer is to get feedback about the performance impact of his code changes very quickly. We run our performance test suite two times a day. So ideally you can fix a performance bug the same day you created it :-

Developers get this feedback via email just as build or functional test results. So they invest time once and get feedback anytime they – or somebody else – changes the code.

Reason 6: The old code threat
Not only we but also our code gets older. The older code is, the less you want to touch it. However very often you have no choice. Martin Fowler already wrote in his Refactoring book that you also start refactoring by having the proper tests in place. While Martin was talking about functional tests here, the same is true for performance tests.

The really evil thing is that if you have any modifying components which heavily interact with each other you can very easily introduce performance problems. So instead of just ruining your own code, you will also ruin the code of other developers.

We have seen especially bug fixes introducing such problems. Just as you can see in the picture above, a functional bug fix introduced a serious performance problem. Additionally you want to be fast in fixing the old code and have more time to develop the really cool new features – don’t you?

Reason 7: I might be the newb but not the noob
Stefan realized that the time when the most test cases are written is when a new developer takes over a feature. This is for two main reasons: First, you do not want to break features of other developers; Second, you are new and want to learn how the code works. The best way to do this is – except reading source code and asking former developers – by testing. So developers who take over a feature have the highest acceptance of writing tests.

Related reading:

  1. Performance Management in Continuous Integration I recently gave  presentations on Performance Management as part of...
  2. dynaTrace Application Performance Almanac 2010 Inspired by the work of Stoyan on his performance advent...
  3. Do more with Functional Testing – Take the Next Evolutionary Step Functional Testing has always been an activity done by Test...
  4. Randomizing Input Data for Visual Studio Load Tests While preparing for my presentation Load and Performance Testing: How...
  5. Eating Our Own Dog Food: How dynaTrace does Continuous APM Internally in Development with dynaTrace TA Lead Stefan Frandl I sat together with Stefan Frandl, Test Automation Lead in...

More Stories By Alois Reitbauer

Alois Reitbauer is Chief Technical Strategist at Dynatrace. He has spent most of his career building monitoring tools and fine-tuning application performance. A regular conference speaker, blogger, author, and sushi maniac, Alois currently shares his professional time between Linz, Boston, and San Francisco.

IoT & Smart Cities Stories
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Cell networks have the advantage of long-range communications, reaching an estimated 90% of the world. But cell networks such as 2G, 3G and LTE consume lots of power and were designed for connecting people. They are not optimized for low- or battery-powered devices or for IoT applications with infrequently transmitted data. Cell IoT modules that support narrow-band IoT and 4G cell networks will enable cell connectivity, device management, and app enablement for low-power wide-area network IoT. B...
The hierarchical architecture that distributes "compute" within the network specially at the edge can enable new services by harnessing emerging technologies. But Edge-Compute comes at increased cost that needs to be managed and potentially augmented by creative architecture solutions as there will always a catching-up with the capacity demands. Processing power in smartphones has enhanced YoY and there is increasingly spare compute capacity that can be potentially pooled. Uber has successfully ...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...