Answers From Experiments
You choose technology, architecture, and design alternatives by using experiments. You adopt new ways of working together by using Adoption Experiments. With an experiment, you can get some degree of objective evidence that a technology will work for you, and see what needs to be tuned and tweaked as necessary.
- No one ever tries to introduce new or better tools, designs, or practices
- New approaches don’t work out; new practices don’t seem to stick
- You’re not sure what advice to take from different sources
- You’ve tried the latest and greatest technology stack, tool, or design approach and it failed to deliver results
- You’ve tried “adopting agile” or some other approach and it failed to deliver results
- Lower risk and cost: before investing in a wide-scale rollout or even a full-fledged pilot program, you’re just trying a small, low-risk, low-cost experiment first.
- Better learning: it can be hard to learn the limitations and opportunities of a technology or a technique when it’s embedded in a large project with lots of moving pieces. As an isolated experiment, you can hone in on the aspects that matter to you more quickly, without distraction.
- Easier acceptance: instead of mandating a change to something new and scary, you’re just trying it first. This helps get buy-in from the folks actually doing the work.
- Local adaptation: by explicitly using a trial with feedback, you can tweak and tune the practice to better respond to local requirements.
Rather than relying on what might have worked somewhere else with different people, you can inspect and adapt in the proper context: yours.
- Experiments are performed regularly. All experiments provide data for the next experiment (100pts)
- Experiments are occasionally performed for tech questions only (20pts)
- Experiments are occasionally performed for process adoption only (20pts)
- Experiments are not performed at all (-50pts)
- Experiment results are ignored (-100pts)
- Experiments that do not produce the expected results lead to blame, arguments, retaliation (-500pts)
- Company wide shift to new tools, methods and processes with no experiments (-1000pts)
✓ Critical ❑ Helpful ❑ Experimental
Each experiment in The GROWS Method™ lists the following sections: a Setup, Trial, and Evaluate Feedback. These sections give a simple overview of steps to first adopt this practice. But these steps are just a summary, be sure to read the What Does it Look like? section of each practice for important information.
- Ensure you have a basic understanding of the scientific method and the importance of a theory, a well run experiment and objective results
- Spend time thinking about how you could set up an experiment to provide unbiased insight into your theory
- Remember that this takes time and it’s unlikely a single experiment will result in complete clarity or insight into your theory
- Be especially aware of the dangers of confirmation bias and it’s affect on evaluating results
- Try to engage others to help run experiments and limit the impact of bias on the outcome
- Deploy confederates and double blind evaluations where possible
[Sample] Evaluate Feedback
- Be open to what the results show you and don’t be surprised by the unexpected
- If practical / possible, have someone else replicate your study to validate your findings
What Does it Look like?
When choosing a new technology or design approach, have at least three alternatives to choose from.
Keep the experiment short: no more than a week or two, or a few weeks at the very most. Bear in mind that development cost is only part of the criteria for a new tool or design. You should also consider:
- Short-term Development cost
- Long-term Maintenance costs
- Ease of replacement (if it’s not easy to rip out and replace, don’t chose it)
- Overall impact of this decision (is it reversible if you choose poorly or conditions change?)
Experiments are time-boxed, which limits commitment and risk, unlike the more amorphous “change,” which is permanent and open-ended. It’s very clear all involved that you aren’t yet adopting this practice or committing to it. You’re just going to give it a try.
Everyone participates in the experiment and in evaluating the outcome, which gives the participants a chance to “put their own egg in,” as the saying goes. (When Betty Crocker first came out with an instant cake mix, it was a failure. All you had to do was add water to to the mix. They changed the formula so you had to add an fresh egg and water, and now consumers felt like cooks again. That version was a success: level of participation makes a difference!)
In the introductory skill stages, the experiment will be defined for you in each practice. By the higher stages, we’ll show you how to construct your own.
- Experiments last six months. They should be no more than a few weeks at the very most
- Experiments requires a large investment in expensive tools.
- Experiments becomes a rubber-stamp approval
- You experiment with less than three alternatives when there’s a question
- An experiment impacts production code
- The decision at the conclusion of the experiment is irreversible.
- Feedback indicates this is working well, and you don’t adopt the practice.
- Feedback indicates this is not going to help, and you adopt it anyway.
How To Fail Spectacularly
While evaluating different approaches to technology, including the age-old tension between developing software from scratch versus adopting software that’s already built by someone else. The key question to ask is why you might be developing this piece yourself.
Deciding to re-implement your own custom version of a commonly available, off-the-shelf technology is almost always a very, very bad idea.
Unless this is your core product itself, you have no business trying to develop your own:
- Object-relational mapping layer
- Database connection pool
- Version control
- Widget set
This is all well-trodden commodity territory, and it’s unlikely you’ll produce anything of value beyond what currently exists off-the-shelf.
Save your development resources for areas that can really make a difference.
Trying everything all at once. Remember to honor all of the core concepts, especially Take Small Bites, One Thing at a Time, and Agree to Try.
Ignoring feedback if it’s not what you expect. Remember, no experiment fails; all experiments provide data. It’s very possible to get unwanted feedback, but that just means you need to go in a slightly different direction than you’d planned.
←Prev (Measure Actuals)(Seeds to Success) Next→