Topic: Automation Estimation - Roundtable Discussion
Initial proposed subtopics were:
* Issues in estimation of test automation projects
* Automation estimation experiences
* Automation requirements planning
* Agile approaches to test automation
* Test automation metrics
* Defining and verifying test automation progress/success
Took place on: Wed. March 4 2009 6:30 PM
Attendance: 15
Meeting Notes from roundtable discussion:
Considerations in test automation project estimation -
Maturity and stability of AUT
Tool chosen
Experience and knowledge of automation engineer(s)
Maintenance of automation suite over time; eg with each AUT release or change
Automation stakeholder expectations
Needed refactoring of automation code over time
Use of common modules/modular codebase
To baseline an estimate, get some experience with a tool, doing some initial tests first; subsequent test dev typically goes much faster
Some QA engineers more interested in or better at test automation than others; automation test engineers need to be able to take a developer perspective
Automation test environment consideration can sometimes be critical
Consider automation result reporting requirements
Start with understanding what can be automated and what can't.
Expect to have missteps/failures in initial automation efforts
SW estimation is hard, so automation estimation is hard too especially if not as experienced at estimates as the dev teams
Estimates based on historical data better than guesstimating
Useful to have some automation requirements or criteria before doing estimates, and to prioritize them
Useful to consider automation 'effectiveness', what that means, and how it can be measured or monitored
As with most software project estimates, an accurate full automation estimate may result in decision not to pursue the automation project. Sometimes an initial pilot automation project that does not include all the long-term considerations may make initial automation spending easier to accept and allow for time to build support and understanding of the benefits of automation and support of a larger budget.
Consider types of automated tests needed - just functional tests? Performance tests? Security tests?
Be aware of possibility of overemphasizing value or importance of a particular tool; ignoring its limitations
Sometimes a more effective automation approach is to start out small and grow
Usually manual testing is needed first as a precursor to planning and developing automation
Consider requirements for information reported by the automation tool for its usefulness in analyzing problems found.
Consider automation of real user scenarios and real paths
Consider automation of the more tedious manual test cases as higher priority automation candidates
Targeting a 'coverage' level can be problematical given the wide possible varieties of the meaning of 'coverage'
In reporting automation results, ROI, etc it may be useful to consider the variations in the meaning of the term 'test case'
ROI determinations may give an incomplete picture of pros/cons of automation - calculation based solely on cost of manual vs automated may or may not be appropriate
'Speed' of testing may be improved by automation once tests are developed, but the time needed to develop automated tests can slow things down; however once developed they could make it easier to meet schedules.
May need to manage expectations of automation stakeholders.
Automation does not eliminate need for manual testing but may be able to replace some
Good automation may require close interaction/cooperation between dev and QA
Can find many bugs during automation development and also learn a lot about AUT
Pitching the 'speed' of testing via automation may confuse management about appropriate goals of automation
A disadvantage of automation is that while people running manual tests can be tolerant/adaptable to variations, automated tests usually are not. Also people can notice anomalies during manual testing that an automated test might not detect
There may be significant productivity value from automation due to the more immediate feedback to developers
Consider how much the automated tests will be reused, and how much a common codebase will be reused.
In agile projects, may not see automation value until 2 iteration cycles past the dev cycle - need to do manual testing initially
