Linux Containers Authors: Zakia Bouachraoui, Elizabeth White, Liz McMillan, Yeshim Deniz, Pat Romanski

Related Topics: Java IoT, Microservices Expo

Java IoT: Article

How to Make Developers Write Performance Tests

Week 6 of our 2010 Application Performance Almanac

I had an interesting conversation with our Test Automation team lead Stefan – who Andi interviewed for our “Eating our own Dog Food ” article – on his experiences with the willingness of developers to write performance tests.

I asked a provocative question: do developers really want to write them in the first place? First he smiled but then he said that they do. I honestly was a bit surprised because in my own experience as a developer was that I rather wanted to write code instead of tests.

He agreed that it was not that easy in the beginning; however, after the first developers started it became a more and more common practice. I then asked him what he thinks are the main reasons why developers want to write performance tests. The following sums up our experience at dynaTrace into what causes developers themselves write performance tests.

Reason 1: Everyone wants to be a hero
Developers like to write well-performing code. Even more, they like to write faster code. Just like Captain Kirk saying “I need the ship ready in 2 hours” – and Scottie replying “I will do it in 20 minutes”.

Every developer wants to show how well his code performs. The performance testing hype at dynaTrace really started when a developer sent out an email about how his performance tests showed him a regression immediately after checking in. The figure below shows exactly the image the developer sent out.

CI  Performance Graph Showing Several Performance Regressions

CI Performance Graph Showing Several Performance Regressions

The developer wanted to share his experience and also show others the benefits of the tests he wrote. A couple of weeks after this email another developer sent out an email about the improvements he achieved for a component between two releases.

Another team of developers managed to optimize the throughput of our server by up to four times. This was a really great achievement and the result of hard work – earning them the dynaTrace culture award. So their contribution was acknowledged as one of the most significant in the last quarter.

So if you want people to deliver certain results then award them for those results and give them the fame they deserve.

Reason 2: The “It will come back anyway” Experience
This one is nice as nobody believes it first. You can always tell people that if they do not get things right during development, a potential problem will come back to them later. So while performance testing might mean some additional work right now it is better than having even more work later on when you have to fix performance bugs.

As I said, nobody believes this until they experience it for the first time themselves. Then people start to write their tests to avoid running into this situation again. There is nothing more frustrating than fixing bugs – possibly in older versions – when all your colleagues are working on the cool new features.

Reason 3: He is working on my code too
At dynaTrace we have always taken performance seriously. Developers have always been profiling their code to optimize it or add additional output for performance measurement. This works fine if you are the only one working on your code and do not have any dependencies on other components. If you not earning money by writing “Hello World” applications this is not the case ; -). Other developers are working on the code base as well.

So beside the fact that your profiling results are only available to you locally, they might already be outdated after the next check-in. As you have no automated means to find those changes, you will not be aware of it. Only a standardized test suite that is automatically executed as part of your Continuous Integration environment can help here

Some people also use specific logging code which they think will help them to track down problems. While this works fine sometimes, our experience has shown that others might change the code causing part of the logging statement not to be executed anymore. The reason might be that your code just got deprecated or somebody added an additional conditional statement, so logging code now only gets executed in certain cases.

Reason 4: Only let developers write test code
This is a key lesson we learned in making performance testing successful. The more complex it is to write test code, the fewer tests are written. Especially performance tests can easily become quite complex – sometimes even more complex than the actual code you want to test. In our case, we want to start our own software, start a couple of application server instances and then trigger a load test. All this has to be managed across multiple nodes.

If we forced developers to write that code themselves they would not want to write any tests at all. So we took a massive investment in writing our own testing framework that takes away that burden from the developer. The example below shows how to start a remote WebSphere server; all done by just adding some simple annotations.

host = "lab2",
name = "WAS7.0",
startupPriority = 1,
postStartClosure = WaitForWebSphereSudIsUp.class
private SudInterface webSphereSud;

Additionally, dynaTrace itself provides extensions which developers can use to control our own software components. Developers now simply focus on writing their test cases. All this together help us to create a good test suite quickly and make it attractive to developers to write tests.

Reason 5: Provide Feedback
If you invest your time, you want to get something back. So the key to make developers write more tests is to make the test results easily accessible to them. The immediate value for the developer is to get feedback about the performance impact of his code changes very quickly. We run our performance test suite two times a day. So ideally you can fix a performance bug the same day you created it :-

Developers get this feedback via email just as build or functional test results. So they invest time once and get feedback anytime they – or somebody else – changes the code.

Reason 6: The old code threat
Not only we but also our code gets older. The older code is, the less you want to touch it. However very often you have no choice. Martin Fowler already wrote in his Refactoring book that you also start refactoring by having the proper tests in place. While Martin was talking about functional tests here, the same is true for performance tests.

The really evil thing is that if you have any modifying components which heavily interact with each other you can very easily introduce performance problems. So instead of just ruining your own code, you will also ruin the code of other developers.

We have seen especially bug fixes introducing such problems. Just as you can see in the picture above, a functional bug fix introduced a serious performance problem. Additionally you want to be fast in fixing the old code and have more time to develop the really cool new features – don’t you?

Reason 7: I might be the newb but not the noob
Stefan realized that the time when the most test cases are written is when a new developer takes over a feature. This is for two main reasons: First, you do not want to break features of other developers; Second, you are new and want to learn how the code works. The best way to do this is – except reading source code and asking former developers – by testing. So developers who take over a feature have the highest acceptance of writing tests.

Related reading:

  1. Performance Management in Continuous Integration I recently gave  presentations on Performance Management as part of...
  2. dynaTrace Application Performance Almanac 2010 Inspired by the work of Stoyan on his performance advent...
  3. Do more with Functional Testing – Take the Next Evolutionary Step Functional Testing has always been an activity done by Test...
  4. Randomizing Input Data for Visual Studio Load Tests While preparing for my presentation Load and Performance Testing: How...
  5. Eating Our Own Dog Food: How dynaTrace does Continuous APM Internally in Development with dynaTrace TA Lead Stefan Frandl I sat together with Stefan Frandl, Test Automation Lead in...

More Stories By Alois Reitbauer

Alois Reitbauer is Chief Technical Strategist at Dynatrace. He has spent most of his career building monitoring tools and fine-tuning application performance. A regular conference speaker, blogger, author, and sushi maniac, Alois currently shares his professional time between Linz, Boston, and San Francisco.

IoT & Smart Cities Stories
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments t...
If a machine can invent, does this mean the end of the patent system as we know it? The patent system, both in the US and Europe, allows companies to protect their inventions and helps foster innovation. However, Artificial Intelligence (AI) could be set to disrupt the patent system as we know it. This talk will examine how AI may change the patent landscape in the years to come. Furthermore, ways in which companies can best protect their AI related inventions will be examined from both a US and...
Bill Schmarzo, Tech Chair of "Big Data | Analytics" of upcoming CloudEXPO | DXWorldEXPO New York (November 12-13, 2018, New York City) today announced the outline and schedule of the track. "The track has been designed in experience/degree order," said Schmarzo. "So, that folks who attend the entire track can leave the conference with some of the skills necessary to get their work done when they get back to their offices. It actually ties back to some work that I'm doing at the University of San...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...