The BRD should define the ideal profile of the participants you want to have in the beta. How many total beta participants are required? What mix of new and experienced users do you want? What mix of direct customers and partners are needed, etc.?
But getting the right number and mix of participants to commit to beta is another story. Remember, participating in a beta program means that people are going to spend their time on your unreleased software.
Look at prospecting in the same way you look at sales: build a funnel of qualified prospects that is some multiple e. And of course, to build this funnel, you will need to contact an even larger multiple of people. As an example, in my recent beta program, we identified six major functional areas where we wanted beta feedback. We also decided that our goal was to have between three to five active beta participants per functional area.
So, three to five active participants per area translated to 18 to 30 active beta participants overall. But ensuring that we met this number meant recruiting at least twice as many into the beta program. There are always sites that initially commit to beta but then drop out or do not participate as actively as planned, so ensure that you have significantly more than the target number committed to beta.
We did a lot of prospecting to acquire these qualified sites. Over customers and partners were contacted. We worked closely with our sales and marketing teams to identify prospects. We also canvassed customers we had visited in early requirements gathering trips for functionality in this release.
Part of our qualification process included understanding why the person or people were interested in participating in the beta. Most had a clear business reason for wanting access to the beta and had staff that could be dedicated to the beta and work with us to provide very detailed product feedback. With direct customers, the story was a little different. While we identified many well-intentioned direct customers, spending time on beta meant an additional task that had to be done on top of their full-time jobs.
Thus, it was something that many of our beta contacts were not keen to tackle. In those cases, we worked with our account manager to escalate the benefits of the beta program higher up at the customer site and tried to get a management sponsor for participating in the beta. Overall, this was a very laborious process, and took about two months to complete. Keep in mind that recruiting is the most critical aspect of the beta program. And while getting it right does not guarantee success, it does significantly increase the probability in reaching those goals.
Not all beta customers are created equal. During the prospecting process, it will become evident that some customers are more likely to provide good feedback than others. As mentioned earlier, those with clear business needs addressed by the beta software, as well as partner companies, typically have the incentive to invest time and resources into your beta program. In addition to these companies, there may be some beta participants who are considered marquee customers and are potential beta reference sites.
In short, a subset of the beta participants will warrant extra attention and support during the beta program as they are more likely than others to help you reach your targets. Identify these premier companies early. Make all internal parties in your company who are involved in the beta program aware that these special firms will be monitored closely, and if needed, given additional help and support to ensure they are successful in the beta. Assign named technical resources to these sites to better ensure their beta program success.
The product management team should work especially close with these customers, visiting them or engaging in additional contact to help resolve issues that arise. You may not be able to accurately identify all premier sites until the beta is actually in progress. There are many things that can happen between the time a customer commits to participating in the beta and when the beta actually begins.
In some cases, what looked like a promising site during registration turns out to be a dud once testing starts. And the opposite is also true. What sounds like a sleepy beta prospect turns into a very active and engaging beta site that far exceeds the original objectives.
There is no harm in updating the premier list once beta starts, and in fact, it may become a necessity as the beta progresses. As beta recruiting is taking place, you need to ensure internal resources are identified and will be prepared for beta.
Who will provide technical support to the beta sites? Is it individuals from the technical support organization, from professional services, sales consulting, or could it even be from your development and quality assurance teams?
It varies across companies, typically depending on the nature of the software and the size of the company. For smaller companies, beta support may be delivered by Engineering, whereas in larger companies, the customer support team may handle it.
Once you have defined the internal team members, ensure appropriate individuals—particularly those who will provide the technical support for the beta—are properly educated on the beta software. Depending on the complexity of the product, this could be as simple as giving the internal groups access to the software and having them educate themselves via the documentation, or it could range all the way up to delivering formal training classes ahead of the release of beta software to customers.
The objective ensures that your internal teams are at least one step ahead of customers, so that as customers start installing and using the software, the internal teams are ready to respond to any problems.
Beta customers typically need some form of education or training before using the beta software. If the software is a consumer-oriented product, then ideally, little or no training should be required. Worst case, the customer should be able to read or view a short online tutorial to get them going. Conversely, complex business or enterprise software requires extensive training.
As mentioned earlier, we held three minute Webinars for our beta customers. Virtually all beta customers attended all three Webinars and we received positive feedback from the attendees. While that made us feel good, we also knew that Webinars would only partially prepare customers to use the product and we would have to help them by explaining functionality during the beta itself. Without some guidance, a lot of beta feedback can be vague and generic.
To help guide the testing done at each site, provide the beta customer with some structure. For example, define beta test scenarios for each area of functionality that needs to be tested. The purpose of these scenarios is to provide a clear framework to the customer in which to test the software. It is not to provide a step-by-step set of actions that they should perform.
The scenarios should focus on typical use cases you expect customers to perform once the software is released. The scenarios can also focus on specific aspects of functionality that need testing in customer environments.
Do not expect beta customers to be able to perform in-depth stress testing of your software. This can be rather time consuming to prepare for and execute, and most beta customers will not have the time and resources to do this level of testing. The scenarios should be made available at the beginning of Phase II and can be introduced during the beta Webinars or in the first customer calls once beta has begun.
Another important task in preparing for beta is to track the readiness of the beta software. There are two major questions to answer. How can you measure if the software is ready to ship when the beta begins, and what do you do if you determine that it is not ready?
It comes down to whether you want to be date-driven, quality-driven, or in most cases, an acceptable mix of the two. Determining shipping readiness means measuring the quality of the software and deciding if it meets preset, agreed-upon criteria. If it meets the criteria, it is ready to ship. If not, more work needs to be done on it. Software quality can be an elusive concept, so how does one measure it? The first thing that comes to mind is the number of open bugs and their severity.
A lot of discussion will take place in Development or QA meetings about the number of critical or showstopper bugs that are in the current release. The problem with a statement like this is that, in reality, it is very difficult to enforce. First, as the target beta date approaches, and if the number of critical and severe bugs remains high, the definitions of both critical and severe will magically loosen.
What was once a critical bug two months prior to beta becomes a severe bug two weeks before beta. And what was once a severe bug, may become a moderate bug. Third, the statements about number of bugs are one-dimensional. The statements treat all critical bugs the same and all severe bugs the same. In reality, different functional areas need to be tracked separately with metrics for each area. In some cases, there may be lower tolerance levels acceptable for bugs, in other cases, it may be higher.
Overall, things like usability, performance, installation, and upgrade path need to be measured and tracked separately with their own criteria and requirements. For example, if your software has an installer, the installer must be bug free. If the installer is buggy, not only will the initial user experience be negative, but in cases where the user encounters a bug, the user will either have to call Support for help, or perhaps worse, abandon the install altogether.
Usability issues are another facet of applications that must be tracked separately. They may be implemented by engineers, who may not be GUI experts, and then modified later as issues are raised. All of the readiness criteria should be converted to a number of metrics that are tracked and reviewed via a management dashboard.
This dashboard shows high-level status for each category e. If available, you can use a dedicated reporting or dashboarding tool.
But if not, then virtually any application, such as a spreadsheet, presentation tool or word processor will do. Regardless of how it is implemented, there are a number of benefits to this approach. A management dashboard provides a top-down view of the state of product quality. A lot of red on the dashboard a few weeks before beta is set to launch may give you early warning that you will need to slip the release date or that additional focus or resources are needed to meet your existing dates.
If most areas show green, but a few are persistently red or yellow, it may indicate issues with the teams working on those areas of functionality, and specific action can be taken to help those team members.
Another obvious benefit of such a dashboard is that it gives upper management clear visibility into the status of the software and the ability to make better and quicker decisions to reallocate resources to where they may be needed.
After all of this preparation, you are now ready to deliver the software to customers and begin the actual beta testing. In most cases, customers should be able to install the software on their own. If the installation is complex, you can either send someone onsite to help install the software, or provide preinstalled software on a server or desktop machine. Keep in mind that while it may be possible to provide extra installation assistance during beta, this is most likely not going to be the case when the software is generally available.
So, if the beta requires extra effort to install and configure, make sure you know why that is the case, and have a very clear and firm plan to address it before final software is released. The major aim of the beta is to collect feedback from the beta sites.
Despite any assurances to the contrary from beta sites, most sites will need a lot of prodding and assistance during the beta program if you want their substantive feedback.
Thus, if you start with 50 registered beta sites, approximately 20 to 30 will actually install the software and provide some level of useful feedback, with about 10 to 15 giving detailed and very meaningful feedback. Of course every situation is different, but these ratios have held true for me across several beta programs. Regardless of the number of active beta sites, there are many ways to collect feedback.
Depending on your goals, you can use the following techniques:. Weekly calls with individual participants work very well. Although it can be time consuming to hold weekly meetings with each beta site, customers will commit to a weekly minute call, and in some cases a minute call, to discuss their issues and feedback.
The benefit of the weekly call is that the beta customer can block off a fixed amount of time each week and knows they will get your undivided attention for that period of time. If they have problems with the beta software, the weekly call creates an opportunity to discuss the issues and get the customer back on track. If they are making progress with the beta, this weekly call gives you the opportunity to discuss their feedback in great detail and guide them further along in the directions you want them to go.
In addition to weekly calls with individual beta customers, you can set up regular, perhaps biweekly, conference calls with multiple customers. There is a risk of these calls devolving into a gripe session—it only takes one unhappy beta customer to get a group started— but if managed well, listening to customers discuss their issues with one another can lead to some valuable insights. Another mechanism for communicating with beta customers is to provide a blog or web-based discussion forum to enable them to ask and answer questions, post findings or simply discuss relevant issues with one another.
The benefit of providing an electronic forum is that the information is automatically persisted for you, and if supported by the software, customers. These web-based forums can also be used as customer self-help tools, enabling beta customers to ask and answer questions amongst themselves.
Last and—in my opinion—least, let the customer contact you via email or phone on their own schedule and not set up formal weekly or biweekly calls.
While a dedicated beta customer may be quite proactive and eloquent if left on their own, in my experience, leaving it up. Collecting customer feedback is one thing, but recording and disseminating it within your organization is something completely different. Remember that different internal groups require various levels of detail of the beta feedback.
Obviously the most detailed information is in the form of the raw notes and findings taken from weekly customer calls. This raw feedback should be posted in an internal forum, such as on a wiki or portal, so that all parties can access it easily. The main consumers of this level of information will most likely be engineers and product managers. Raw feedback contains various types of information about customer issues, positive and negative comments on various parts of the software, access to technical support, plans, expectations, etc.
The information collected from these calls should be sliced and diced, and presented in multiple ways so that it can be efficiently utilized. Centralized Log Management on 19 January Beta Systems in figures 1.
X This website uses cookies We use cookies on our website. Some of them are essential, while others help us improve this website and your experience.
You decide which cookie categories you want to agree to. Please note that depending on your selection, you may no longer be able to use all the features of our website. More information Save Settings Accept all. Close Data protection overview This website uses cookies to improve your experience while you navigate through the website.
Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website.
These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
BetaBooks simplifies the entire process and saves me an enormous amount of time so I can actually focus on my writing. I was able to get feedback from three different people on three different stories last night. I could never have that kind of turnaround with email or even Google Docs. Each reader being able to login and see things queued up for them with exactly what I need done makes it a better flow for them and for me. BetaBooks has made the beta reading process so much easier on me and my readers.
Your web app is one of those solutions which get me excited: clean, concise, and meant for the user! Thank you for your vision. I can't tell you guys how much BetaBooks has slipped into my daily life.
I've never felt more "on top of" sorting through all this feedback, able to see the patterns I sensed present in feedback. I think y'all have changed my writing life for good! BetaBooks has been splendid so far, I've already gotten way more feedback than I expected and I love how it is all being tracked separately per reader, rather than in one enormous Google Doc or something.
0コメント