Linux Containers Authors: Liz McMillan, Yeshim Deniz, Elizabeth White, Pat Romanski, Stackify Blog

Related Topics: Java IoT, Microservices Expo

Java IoT: Article

How to Make Developers Write Performance Tests

Week 6 of our 2010 Application Performance Almanac

I had an interesting conversation with our Test Automation team lead Stefan – who Andi interviewed for our “Eating our own Dog Food ” article – on his experiences with the willingness of developers to write performance tests.

I asked a provocative question: do developers really want to write them in the first place? First he smiled but then he said that they do. I honestly was a bit surprised because in my own experience as a developer was that I rather wanted to write code instead of tests.

He agreed that it was not that easy in the beginning; however, after the first developers started it became a more and more common practice. I then asked him what he thinks are the main reasons why developers want to write performance tests. The following sums up our experience at dynaTrace into what causes developers themselves write performance tests.

Reason 1: Everyone wants to be a hero
Developers like to write well-performing code. Even more, they like to write faster code. Just like Captain Kirk saying “I need the ship ready in 2 hours” – and Scottie replying “I will do it in 20 minutes”.

Every developer wants to show how well his code performs. The performance testing hype at dynaTrace really started when a developer sent out an email about how his performance tests showed him a regression immediately after checking in. The figure below shows exactly the image the developer sent out.

CI  Performance Graph Showing Several Performance Regressions

CI Performance Graph Showing Several Performance Regressions

The developer wanted to share his experience and also show others the benefits of the tests he wrote. A couple of weeks after this email another developer sent out an email about the improvements he achieved for a component between two releases.

Another team of developers managed to optimize the throughput of our server by up to four times. This was a really great achievement and the result of hard work – earning them the dynaTrace culture award. So their contribution was acknowledged as one of the most significant in the last quarter.

So if you want people to deliver certain results then award them for those results and give them the fame they deserve.

Reason 2: The “It will come back anyway” Experience
This one is nice as nobody believes it first. You can always tell people that if they do not get things right during development, a potential problem will come back to them later. So while performance testing might mean some additional work right now it is better than having even more work later on when you have to fix performance bugs.

As I said, nobody believes this until they experience it for the first time themselves. Then people start to write their tests to avoid running into this situation again. There is nothing more frustrating than fixing bugs – possibly in older versions – when all your colleagues are working on the cool new features.

Reason 3: He is working on my code too
At dynaTrace we have always taken performance seriously. Developers have always been profiling their code to optimize it or add additional output for performance measurement. This works fine if you are the only one working on your code and do not have any dependencies on other components. If you not earning money by writing “Hello World” applications this is not the case ; -). Other developers are working on the code base as well.

So beside the fact that your profiling results are only available to you locally, they might already be outdated after the next check-in. As you have no automated means to find those changes, you will not be aware of it. Only a standardized test suite that is automatically executed as part of your Continuous Integration environment can help here

Some people also use specific logging code which they think will help them to track down problems. While this works fine sometimes, our experience has shown that others might change the code causing part of the logging statement not to be executed anymore. The reason might be that your code just got deprecated or somebody added an additional conditional statement, so logging code now only gets executed in certain cases.

Reason 4: Only let developers write test code
This is a key lesson we learned in making performance testing successful. The more complex it is to write test code, the fewer tests are written. Especially performance tests can easily become quite complex – sometimes even more complex than the actual code you want to test. In our case, we want to start our own software, start a couple of application server instances and then trigger a load test. All this has to be managed across multiple nodes.

If we forced developers to write that code themselves they would not want to write any tests at all. So we took a massive investment in writing our own testing framework that takes away that burden from the developer. The example below shows how to start a remote WebSphere server; all done by just adding some simple annotations.

host = "lab2",
name = "WAS7.0",
startupPriority = 1,
postStartClosure = WaitForWebSphereSudIsUp.class
private SudInterface webSphereSud;

Additionally, dynaTrace itself provides extensions which developers can use to control our own software components. Developers now simply focus on writing their test cases. All this together help us to create a good test suite quickly and make it attractive to developers to write tests.

Reason 5: Provide Feedback
If you invest your time, you want to get something back. So the key to make developers write more tests is to make the test results easily accessible to them. The immediate value for the developer is to get feedback about the performance impact of his code changes very quickly. We run our performance test suite two times a day. So ideally you can fix a performance bug the same day you created it :-

Developers get this feedback via email just as build or functional test results. So they invest time once and get feedback anytime they – or somebody else – changes the code.

Reason 6: The old code threat
Not only we but also our code gets older. The older code is, the less you want to touch it. However very often you have no choice. Martin Fowler already wrote in his Refactoring book that you also start refactoring by having the proper tests in place. While Martin was talking about functional tests here, the same is true for performance tests.

The really evil thing is that if you have any modifying components which heavily interact with each other you can very easily introduce performance problems. So instead of just ruining your own code, you will also ruin the code of other developers.

We have seen especially bug fixes introducing such problems. Just as you can see in the picture above, a functional bug fix introduced a serious performance problem. Additionally you want to be fast in fixing the old code and have more time to develop the really cool new features – don’t you?

Reason 7: I might be the newb but not the noob
Stefan realized that the time when the most test cases are written is when a new developer takes over a feature. This is for two main reasons: First, you do not want to break features of other developers; Second, you are new and want to learn how the code works. The best way to do this is – except reading source code and asking former developers – by testing. So developers who take over a feature have the highest acceptance of writing tests.

Related reading:

  1. Performance Management in Continuous Integration I recently gave  presentations on Performance Management as part of...
  2. dynaTrace Application Performance Almanac 2010 Inspired by the work of Stoyan on his performance advent...
  3. Do more with Functional Testing – Take the Next Evolutionary Step Functional Testing has always been an activity done by Test...
  4. Randomizing Input Data for Visual Studio Load Tests While preparing for my presentation Load and Performance Testing: How...
  5. Eating Our Own Dog Food: How dynaTrace does Continuous APM Internally in Development with dynaTrace TA Lead Stefan Frandl I sat together with Stefan Frandl, Test Automation Lead in...

More Stories By Alois Reitbauer

Alois Reitbauer is Chief Technical Strategist at Dynatrace. He has spent most of his career building monitoring tools and fine-tuning application performance. A regular conference speaker, blogger, author, and sushi maniac, Alois currently shares his professional time between Linz, Boston, and San Francisco.

@ThingsExpo Stories
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
The Founder of NostaLab and a member of the Google Health Advisory Board, John is a unique combination of strategic thinker, marketer and entrepreneur. His career was built on the "science of advertising" combining strategy, creativity and marketing for industry-leading results. Combined with his ability to communicate complicated scientific concepts in a way that consumers and scientists alike can appreciate, John is a sought-after speaker for conferences on the forefront of healthcare science,...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at @ThingsExpo, James Kirkland, Red Hat's Chief Archi...
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settl...
Organizations planning enterprise data center consolidation and modernization projects are faced with a challenging, costly reality. Requirements to deploy modern, cloud-native applications simultaneously with traditional client/server applications are almost impossible to achieve with hardware-centric enterprise infrastructure. Compute and network infrastructure are fast moving down a software-defined path, but storage has been a laggard. Until now.
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
DXWorldEXPO LLC announced today that the upcoming DXWorldEXPO | CloudEXPO New York event will feature 10 companies from Poland to participate at the "Poland Digital Transformation Pavilion" on November 12-13, 2018.
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
The best way to leverage your CloudEXPO | DXWorldEXPO presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering CloudEXPO | DXWorldEXPO will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at CloudEXPO. Product announcements during our show provide your company with the most reach through our targeted audienc...
JETRO showcased Japan Digital Transformation Pavilion at SYS-CON's 21st International Cloud Expo® at the Santa Clara Convention Center in Santa Clara, CA. The Japan External Trade Organization (JETRO) is a non-profit organization that provides business support services to companies expanding to Japan. With the support of JETRO's dedicated staff, clients can incorporate their business; receive visa, immigration, and HR support; find dedicated office space; identify local government subsidies; get...
DXWorldEXPO LLC announced today that All in Mobile, a mobile app development company from Poland, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. All In Mobile is a mobile app development company from Poland. Since 2014, they maintain passion for developing mobile applications for enterprises and startups worldwide.
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.
As data explodes in quantity, importance and from new sources, the need for managing and protecting data residing across physical, virtual, and cloud environments grow with it. Managing data includes protecting it, indexing and classifying it for true, long-term management, compliance and E-Discovery. Commvault can ensure this with a single pane of glass solution – whether in a private cloud, a Service Provider delivered public cloud or a hybrid cloud environment – across the heterogeneous enter...
DXWorldEXPO LLC announced today that ICC-USA, a computer systems integrator and server manufacturing company focused on developing products and product appliances, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of ...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smart...
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
Founded in 2000, Chetu Inc. is a global provider of customized software development solutions and IT staff augmentation services for software technology providers. By providing clients with unparalleled niche technology expertise and industry experience, Chetu has become the premiere long-term, back-end software development partner for start-ups, SMBs, and Fortune 500 companies. Chetu is headquartered in Plantation, Florida, with thirteen offices throughout the U.S. and abroad.