August 2010 Monthly Meeting Summary
Topic:
Automation of Test Environment Management - roundtable discussion
Setting up, configuring, tracking, and managing test environments can be a challenge, especially when large numbers
of differing test environments are needed and quick setup and teardown is needed for automated test runs. We will
discuss these issues and tools and techniques for dealing with them. Potential subtopics were:
* Virtual machines and managing them
* Cloud-based approaches
* Tools
* Remote/automated installation and configuration
* Test data management
* System monitoring and reporting
Took place on: Wed. August 4 2010 6:30 PM
Attendance: 15
Meeting Notes:
- There was a question - to the attendees using virtual machines for test environments - re the number of virtual machines being managed:
Ranged as high as 1600; many in the 10-50 range
- To set up virtual machines and automate environment management, typically first have to set things up manually to learn setup/install/config issues and what to automate
- For managing MS software on multiple machines - MS Lab Manager app
- Load testing with virtual machines can be complex, in dealing with networking and other issues
- VM apps used: VMWare, Virtual Box - open source, Xen - open source
- It was suggested that it may be best to have server apps on VMs but not some of the middleware such as DB
- Cloud-based environments - was mentioned that some organizations have achieved large savings with cloud-based environments
- Some tools mentioned for stack management and configuration - MS Powershell, shell scripting, Perl, VBScript
- Discussion re management and tracking of what is installed/versions/patches etc on VM or real machines - many said things like 'tribal knowledge' and there was much discussion indicating its an issue
- It was mentioned that standard network/asset management tools typically used by system administrators are of course useful
- A common issue in dealing with test environments - it's common to assume that if SA has an environment set up and is maintaining it, then its OK and as expected; may be a good idea to have some kind of monitoring/verification to check it against the production environment or whatever is the expected environment.
- There was a suggestion to refresh environments daily since it is just not possible in reality to prevent unknown/unexpected changes to environments.
- There was a suggestion re: Always turn off automatic software updates for software in test environments, so as to keep it controlled/tracked
- Always set things up to synchronize server clocks among servers so that log messages with timestamps can be synchronized across the servers, to make it easier to trace problems when analyzing logs from multiple servers.
- There was a discussion that began with: 'If I could change anything about my VM setup it would be....'; various comments in the ensuing discussion were:
* sharing it with other groups
* more open, not controlled by one person. It can help to have a person on team focused on test environment management
rather than taking a dispersed/spare time approach, but then the one person may be a bottleneck or single point of failure, so maybe need at least 2 people.
* a more structured approach to configuration management
* 'that it didn't crash on me'
* crashes related to bad management, or host OS updates which sometimes crashes the VM host.
* smarter updates of VM images from build repository.
* tests run faster
NoVaTAIG Home Page
