The Beta Testing Plan:
- Ensures the project parameters, goals, milestones, activities, and participant demographics are well defined
- Outlines all internal resources (project members and teams) involved, as well as their responsibilities
- Is the basis of the report which will summarize the results and effectiveness of the beta project
Plan Definition:
- Construct an outline of the product itself, including information regarding it’s current state (alpha, beta, beta2, etc), high-level modules, etc.
- Beta Target Market Definition –Detail the types of individuals this product will be used by.
- PEcAn
- CZO communities
- DataONE
- value for the ecological community - specifically allowing for data to be downloaded in a variety of different forms directly from DataONE
- Research Data Services,
- beginning with our own University of Illinois RDS
- Test Parameters –Outline the basic parameters unique to the project including number of beta testers and projected timeline.
- Include the tester requirements (time, hardware, etc) in this definition
- Windows or Mac what about Linux?
- looking at a ... 2 hour window?
- What is space requirement
- What is RAM/Processor requirement
- Test Goals (SUPER KEY)–List the unique goals of the project.
- Items such as general quality improvement - testing for user experience with initial set of tools
- Interface acceptance - API works in desired browser
- Product functionality in real-world environments
- Test support infrastructure - this will be after release
- Collect customer suggestions and testimonials - testimonials is essential for expanding user acceptance and for use in newsletters
- Include the areas (modules) of the product that will be tested
- Incentives–Plan out the incentives that will be rewarded for participation, as well as the participation levels which they will be based on.
- Will we offer incentives?
- Project Team Responsibilities
- Define the responsibilities of everyone involved in the project
- Define how feedback will be managed as it changes hands and progresses
Initial Activities –
Plan out the initial activities which will be performed throughout this beta (activities will likely change in response to the project progression)
Recruiting Testers
- BD Team
- ISDA Team
- Any SSA Team Members?
- Students
- Current Collaborator / Students
Tester Management
Clearly Express Expectations Early on and Throughout
- Do we have anything that will need Confidentiality (NDA, etc)?
- Balance Gratitude with Expectations - Thank you for testing but make sure to meet deadlines for reporting
- Keep Exec Committee Involved with On-going Activities
- Keep Exec Committee Current Regarding Project Progress
- Offer a Simple Consistent Method for Testers to Provide Feedback in the Forms we want
- Allow Testers to Communicate Among Themselves (increases participation, secure outlet for excitement)
- Respond Quickly to all Issues and Requests
- Contact Inactive Testers Directly (by phone if possible)
Beta Activities
Common Beta Activities
- Bug Reports - Google Spreadsheet for bug tracking (above)
- Specific Testing Tasks - create a set of testing matrixes for usability and break testing
- Forum Conversations (Open and Hosted) - Hip Chat room specifically for testing
- Surveys and Polls (Technical and Subjective) - Set up Google Survey for post testing
- Do we have a way to run regression testing between builds? - we seem to keep making changes and then breaking things that worked previously
- Collect Feature Suggestions and Testimonials - this is important for newsletters going forward
Managing Feedback
Effective Data Management
- Use Your Tools Effectively -
- have spreadsheet tracker and matrixes reviewed
- ask for other ideas
- Drive For Details
- are we capturing enough detail?
- Develop a Strategy to Handle Duplicates
- do we need more details in the spreadsheet so we can sort for duplicates?
- Quickly Respond to the Needs of the Test
- Ensure the Right People Get the Right Data
- make sure workflow gets issues from tester, to PM, to JIRA, to resolution without getting dropped in a crack
Closing a Beta Test
Keys to Closing a Beta Test
- Give Your Testers Time to Submit Final Issues
- Cut-off Tester Access to Submit Issues
- can the link to a Google Spreadsheet be changed? If I change the name will it change the shared link?
- Close all Open Issues - don't leave a bug behind
- Offer Testers a Simple Means to Return Product
- This is an interesting thought! - do we have a way once people are finished testing Brown Dog that it can be cleaned from their machine? an "Uninstall Script"
Incentives! - should we do this?
- Ensure Incentives Match Participation Requirements
- Include Multiple Levels of Incentives
- Award All Who Meet Those Levels
- Distribute Incentives Quickly
- If Possible, Thank Testers Individually
Beta Closure Report
Developing a Closure Report
- Develop an Executive Summary of the Project - done ... would it need changed for just the focus of the beta?
- Document Issues Found (by Severity and Repetition) - do we want a column in the bug sheet to track # times bug found/hit?
- Document Requested Features - oh yes - these would be documented in JIRA as features not bugs ... do we need to make a column for that in the tracking sheet?
- Document Survey Results - Google Survey should do this ...
- Document Top Testers & Incentives Rewarded
- Ensure Entire Team Gets a copy of the Report
Common Beta Testing Mistakes
- Lack of a Serious Program or Program Support
- Using the Beta Program For Sales
- Beta Test Period is Too Short or Too Long
- Release of Unviable Product to Beta Test
- Too Few or Too Many Beta Testers
- Poorly Motivated Testers
- Ineffective Communication or Bad Beta Testing Tools
- Poorly Managed Beta Testers and Test Data
- Badly Managed Incentives
Beta Testing Success
- Good Communication –Bidirectional communication with testers regarding timelines, requirements, and progress
- Responsiveness–Make testers feel involved on a constant basis. Treat them as an integral part of team
- Effective Tools –Use the right tools to increase the efficiency of everyone involved (team and participants)
- Organization–Beta tests produce an enormous amount of varied data
- Effective Site Selection –The wrong testers can produce useless results
- Proper Incentives?
Adapted from www.centercode.com - https://www.centercode.com/docs/Centercode%20-%20Sucessful%20Beta%20Testing.pdf