March 2011 Monthly Meeting Summary
Topic:
5 Ways to Do More Performance Testing in Less Time - Presentation by James Pulley of Newcoe Performance Engineering and Steve Sturtevant of OC Systems
Performance testing often bears the brunt of budget and schedule pressures. It can be particularly challenging when managers who do not understand the
discipline dismiss performance as a “non-functional” requirement. This session provided strategies for addressing the challenges test teams face –
leveraging work developers and functional testers are already doing to help you get performance testing started earlier,
do more testing in less time, and increase software performance quality under tight budget and schedule constraints.
Discussion included the following strategies, and then how they were successfully implemented in a multi-billion dollar enterprise application.
- Measure twice, test once - leveraging development & integration test cases to find performance defects.
- Accountability is king - getting developers to buy into performance.
- Functional testing is another opportunity for performance smoke testing.
- Don't just test, diagnose - in today’s world testers must go beyond pass/fail to give developers the context they need to solve problems in a shorter time period.
- Don't forget about your data - why test data integrity may be the single most important aspect of performance testing.
Bios:
James Pulley is the CTO for Newcoe Performance Engineering, as well as Newcoe’s sister organizations, The ScriptFarm and LoadRunner By The Hour.
Previously, James managed the professional services delivery for PowerTest, founded iTest Solutions and worked for Mercury Interactive’s PSO &
Sales organizations, Gigalabs, NetIQ, Banyan Systems and Microsoft. His background includes roles in both sales and professional services
delivery, with technical concentrations on OS and application architectures, network and application infrastructure performance. He is a
moderator for several professional forums related to performance testing and tuning (with respect to Mercury/HP tools) and the
cloud-computing forum on SQAForums.
Steve Sturtevant is a consultant working on the Automated Commercial Environment (ACE) program at U.S. Customs & Border Protection
where his focus has been providing performance engineering and testing services across the software lifecycle. He is employed as a
Senior Software & Performance Engineer by OC Systems, as well as Product Manager for the OC Systems RTI product -
www.rtiperformance.com - which delivers software performance diagnostics for
J2EE systems. Prior to ACE, Steve worked as a performance engineer for several large DoD programs as well as a developer
working on real-time systems.
Took place on: Wed. March 9 2011 6:30 PM
Attendance: 28
Meeting Notes:
- There was some discussion re how to persuade others on a project team to pay attention to performance issues; this included a discussion
of using public (within the project) accountability types of approaches - for example if periodic
testing/diagnostics indicated a certain part of code had a large impact on slowing performace then such information/results would be
posted along with the persons/group responsible for that part of the code.
- It was mentioned that performance testing results can often best be used in the context of trends or relative performance, as
opposed to stand-alone information
- There was discussion that some load testing tools are more intrusive than others and some can add significantly to processing overhead thus skewing
performance results; also some tools help provide more diagnostic information than others
- There was some discussion regarding determining how realistic a test environment might be for use in performance testing; sometimes
there is an issue of testing/engineering 'ghost' issues that can be created during testing due to the forced simulations sometimes needed for
performance testing; an example might be when there are only 25 test user ID's available but you need to simulate 25,000 users, thus the
performance testing may not produce truly realistic results, miss problems, or find problems that would not occur in the production system.
- Automated functional testing can often provide some useful performance 'smoke testing' information.
- The powerpoint file from the presentation is available (1.4 MB file size).
NoVaTAIG Home Page
