The Beta Testing Plan:
- Ensures the project parameters, goals, milestones, activities, and participant demographics are well defined
- Outlines all internal resources (project members and teams) involved, as well as their responsibilities
- Is the basis of the report which will summarize the results and effectiveness of the beta project (more later)
Plan Definition:
- Construct an outline of the product itself, including information regarding it’s current state (alpha, beta, beta2, etc), high-level modules, etc.
- Beta Target Market Definition –Detail the types of individuals this product will be sold to. This could include consumers, business IT departments, soccer moms, previous customers, and everything in between. Most products span multiple markets. Break down the types of users you plan to recruit into pools. Include the tester requirements (time, hardware, etc) in this definitionused by.
- PEcAn
- CZO communities
- DataONE
- value for the ecological community - specifically allowing for data to be downloaded in a variety of different forms directly from DataONE
- Research Data Services,
- beginning with our own University of Illinois RDS
- Test Parameters –Outline the basic parameters unique to the project including number of beta testers and projected timeline.
- Include the tester requirements (time, hardware, etc) in this definition
- Windows or Mac what about Linux?
- looking at a ... 2 hour window?
- What is space requirement
- What is RAM/Processor requirement
- Include the tester requirements (time, hardware, etc) in this definition
- Test Goals (SUPER KEY)–List the unique goals of the project.
- Items such as general quality improvement
- - testing for user experience with initial set of tools
- Interface acceptance - API works in desired browser
- Product functionality in real-world environments
- Test support infrastructure
- - this will be after release
- Collect customer suggestions and testimonials
- - testimonials is essential for expanding user acceptance and for use in newsletters
- Include the areas (modules) of the product that will be tested
- Incentives–Plan out the incentives that’ll that will be rewarded for participation, as well as the participation levels which they will be based on. (more later)
- Will we offer incentives?
- Project Team Responsibilities –Define
- Define the responsibilities of everyone involved in the project
- Define how feedback will be managed as it changes hands and progresses
Initial Activities –
Plan out the initial activities which will be performed throughout this beta (activities will likely change in response to the project progression)
Recruiting Testers
- BD Team
- ISDA Team
- Any SSA Team Members?
- Students
- Current Collaborator / Students
Tester Management
Clearly Express Expectations Early on and Throughout
- Consistently Encourage Do we have anything that will need Confidentiality (NDA, etc)?
- Balance Gratitude with Expectations - Thank you for testing but make sure to meet deadlines for reporting
- Keep Sites Exec Committee Involved with On-going Activities
- Keep Sites Exec Committee Current Regarding Project Progress
- Offer a Simple Consistent Method for Testers to Provide Feedback in the Forms you Wantwe want
- Testing Matrix - need to offer paper copy as well as online - how can online version be copied multiple times
- Shared Google Doc for bugs?
- One person enter bugs in JIRA
- We could use Google Spreadsheet https://docs.google.com/spreadsheets/d/1ofG21FyCoHQ381iPvd59-2w0pljDxFIIkWdg85TUjZM/edit?usp=sharing
- What about Google Survey for post testing impressions? https://support.google.com/docs/answer/87809?hl=en
- Allow Testers to Communicate Amongst Among Themselves (increases participation, secure outlet for excitement)
- Respond Quickly to all Issues and Requests
- Contact Inactive Testers Directly (by phone if possible)
Beta Activities
Common Beta Activities
•Bug Reports
•Specific Testing Tasks
- Bug Reports - Google Spreadsheet for bug tracking (above)
- Specific Testing Tasks - create a set of testing matrixes for usability and break testing
- Forum
...
- Conversations (Open and Hosted)
...
- - Hip Chat room specifically for testing
- Surveys and Polls (Technical and Subjective)
...
•If Possible, Regression Between Builds
- - Set up Google Survey for post testing
- Do we have a way to run regression testing between builds? - we seem to keep making changes and then breaking things that worked previously
- Collect
...
- Feature Suggestions and Testimonials - this is important for newsletters going forward
Managing Feedback
Effective Data Management•Use
- Use Your Tools Effectively
...
- -
- have spreadsheet tracker and matrixes reviewed
- ask for other ideas
- Drive For Details
...
- are we capturing enough detail?
- Develop a Strategy to Handle Duplicates
...
- do we need more details in the spreadsheet so we can sort for duplicates?
- Quickly Respond to the Needs of the Test
...
- Ensure the Right People Get the Right Data
- make sure workflow gets issues from tester, to PM, to JIRA, to resolution without getting dropped in a crack
Closing a Beta Test
Keys to Closing a Beta Test•Give
- Give Your Testers Time to Submit Final Issues
...
- Cut-off Tester Access to Submit Issues
...
- can the link to a Google Spreadsheet be changed? If I change the name will it change the shared link?
- Close all Open Issues
...
- - don't leave a bug behind
- Offer Testers a Simple Means to Return Product
- This is an interesting thought! - do we have a way once people are finished testing Brown Dog that it can be cleaned from their machine? an "Uninstall Script"
Incentives!
...
- should we do this?
- Ensure
...
- Incentives Match Participation Requirements
...
- Include Multiple Levels of Incentives
...
- Award All Who Meet Those Levels
...
- Distribute Incentives Quickly
...
- If Possible, Thank Testers Individually
...
Beta Closure Report
Developing a Closure Report•Develop
- Develop an Executive Summary of the Project
...
- - done ... would it need changed for just the focus of the beta?
- Document Issues Found (by Severity and Repetition)
...
•Document Requested Features
•Document Survey Results
- - do we want a column in the bug sheet to track # times bug found/hit?
- Document Requested Features - oh yes - these would be documented in JIRA as features not bugs ... do we need to make a column for that in the tracking sheet?
- Document Survey Results - Google Survey should do this ...
- Document
...
- Top Testers & Incentives Rewarded
...
- Ensure Entire Team Gets a copy of the Report
Common Beta Testing Mistakes
Developing a Closure Report
•Develop an Executive Summary of the Project
•Document Issues Found (by Severity and Repetition)
•Document Requested Features
•Document Survey Results
•Document Top Testers & Incentives Rewarded
•Ensure All Departments Obtain Report
- Lack of a Serious Program or Program Support
- Using the Beta Program For Sales
- Beta Test Period is Too Short or Too Long
- Release of Unviable Product to Beta Test
- Too Few or Too Many Beta Testers
...
- Poorly Motivated Testers
...
- Ineffective Communication or Bad Beta Testing Tools
...
- Poorly Managed Beta Testers and Test Data
...
- Badly Managed Incentives
...
Beta Testing Success
...
- Good Communication –Bidirectional communication with testers regarding timelines, requirements, and progress
...
...
- Responsiveness–Make testers feel involved on a constant basis. Treat them as an integral part of
...
- team
...
...
- Effective Tools –Use the right tools to increase the efficiency of everyone involved (
...
- team
...
- and participants)
...
...
- Organization–Beta tests produce an enormous amount of varied data
...
...
- Effective Site Selection –The wrong testers can produce useless results
...
...
- Proper Incentives?
Adapted from www.centercode.com - https://www.centercode.com/docs/Centercode%20-%20Sucessful%20Beta%20Testing.pdf
...