The Beta Testing Plan:

  • Ensures the project parameters, goals, milestones, activities, and participant demographics are well defined
  • Outlines all internal resources (project members and teams) involved, as well as their responsibilities
  • Is the basis of the report which will summarize the results and effectiveness of the beta project

Plan Definition:

  • Construct an outline of the product itself, including information regarding it’s current state (alpha, beta, beta2, etc), high-level modules, etc.
  • Beta Target Market Definition –Detail the types of individuals this product will be used by.
    • PEcAn
    • CZO communities
    • DataONE
      • value for the ecological community - specifically allowing for data to be downloaded in a variety of different forms directly from DataONE
    • Research Data Services,
      • beginning with our own University of Illinois RDS
  • Test Parameters –Outline the basic parameters unique to the project including number of beta testers and projected timeline.
    • Include the tester requirements (time, hardware, etc) in this definition
      • Windows or Mac what about Linux?
      • looking at a ... 2 hour window?
      • What is space requirement
      • What is RAM/Processor requirement
  • Test Goals (SUPER KEY)–List the unique goals of the project.
    • Items such as general quality improvement - testing for user experience with initial set of tools
    • Interface acceptance - API works in desired browser
    • Product functionality in real-world environments
    • Test support infrastructure - this will be after release
    • Collect customer suggestions and testimonials - testimonials is essential for expanding user acceptance and for use in newsletters 
    • Include the areas (modules) of the product that will be tested
  • Incentives–Plan out the incentives that will be rewarded for participation, as well as the participation levels which they will be based on.
    • Will we offer incentives?
  • Project Team Responsibilities
    • Define the responsibilities of everyone involved in the project
    • Define how feedback will be managed as it changes hands and progresses

Initial Activities –

Plan out the initial activities which will be performed throughout this beta (activities will likely change in response to the project progression)

Recruiting Testers

  • BD Team
  • ISDA Team
  • Any SSA Team Members?
  • Students
  • Current Collaborator / Students

Tester Management

Clearly Express Expectations Early on and Throughout

  • Do we have anything that will need Confidentiality (NDA, etc)?
  • Balance Gratitude with Expectations - Thank you for testing but make sure to meet deadlines for reporting
  • Keep Exec Committee Involved with On-going Activities
  • Keep Exec Committee Current Regarding Project Progress
  • Offer a Simple Consistent Method for Testers to Provide Feedback in the Forms we want
  • Allow Testers to Communicate Among Themselves (increases participation, secure outlet for excitement)
  • Respond Quickly to all Issues and Requests
  • Contact Inactive Testers Directly (by phone if possible)

Beta Activities

Common Beta Activities

  • Bug Reports - Google Spreadsheet for bug tracking (above)
  • Specific Testing Tasks - create a set of testing matrixes for usability and break testing
  • Forum Conversations (Open and Hosted) - Hip Chat room specifically for testing 
  • Surveys and Polls (Technical and Subjective) - Set up Google Survey for post testing
  • Do we have a way to run regression testing between builds? -  we seem to keep making changes and then breaking things that worked previously
  • Collect Feature Suggestions and Testimonials - this is important for newsletters going forward

Managing Feedback

Effective Data Management

  • Use Your Tools Effectively -
    • have spreadsheet tracker and matrixes reviewed
    • ask for other ideas
  • Drive For Details
    • are we capturing enough detail?
  • Develop a Strategy to Handle Duplicates
    • do we need more details in the spreadsheet so we can sort for duplicates?
  • Quickly Respond to the Needs of the Test
  • Ensure the Right People Get the Right Data
    • make sure workflow gets issues from tester, to PM, to JIRA, to resolution without getting dropped in a crack

Closing a Beta Test

Keys to Closing a Beta Test

  • Give Your Testers Time to Submit Final Issues
  • Cut-off Tester Access to Submit Issues
    • can the link to a Google Spreadsheet be changed?  If I change the name will it change the shared link?
  • Close all Open Issues - don't leave a bug behind
  • Offer Testers a Simple Means to Return Product
    • This is an interesting thought! - do we have a way once people are finished testing Brown Dog that it can be cleaned from their machine?  an "Uninstall Script"

Incentives! - should we do this?

  • Ensure Incentives Match Participation Requirements
  • Include Multiple Levels of Incentives
  • Award All Who Meet Those Levels
  • Distribute Incentives Quickly
  • If Possible, Thank Testers Individually

Beta Closure Report

Developing a Closure Report

  • Develop an Executive Summary of the Project - done ... would it need changed for just the focus of the beta?
  • Document Issues Found (by Severity and Repetition) - do we want a column in the bug sheet to track # times bug found/hit?
  • Document Requested Features - oh yes - these would be documented in JIRA as features not bugs ... do we need to make a column for that in the tracking sheet?
  • Document Survey Results - Google Survey should do this ...
  • Document Top Testers & Incentives Rewarded
  • Ensure Entire Team Gets a copy of the Report

Common Beta Testing Mistakes

  • Lack of a Serious Program or Program Support
  • Using the Beta Program For Sales
  • Beta Test Period is Too Short or Too Long
  • Release of Unviable Product to Beta Test
  • Too Few or Too Many Beta Testers
  • Poorly Motivated Testers
  • Ineffective Communication or Bad Beta Testing Tools
  • Poorly Managed Beta Testers and Test Data
  • Badly Managed Incentives

Beta Testing Success

  • Good Communication –Bidirectional communication with testers regarding timelines, requirements, and progress
  • Responsiveness–Make testers feel involved on a constant basis. Treat them as an integral part of team
  • Effective Tools –Use the right tools to increase the efficiency of everyone involved (team and participants)
  • Organization–Beta tests produce an enormous amount of varied data
  • Effective Site Selection –The wrong testers can produce useless results
  • Proper Incentives?

 

Adapted from www.centercode.com - https://www.centercode.com/docs/Centercode%20-%20Sucessful%20Beta%20Testing.pdf

 

  • No labels