Individuals, teams, and executives need to make decisions about learning, the development process, and delivering user capabilities. Without data, decisions become “auto responses” based on impressions and “gut” feel, and often miss the hoped for outcome.
- Teams’ behavior remains steady regardless of results from their prior work.
- Projects get managed using the same tools, processes and thinking in spite continued poor results.
- No one seems concerned about the continued poor results and delivery of user capabilities slowing down.
When you Gather Data benefits include:
- Teams have current data on which to base their decisions.
- Projects get evaluated on actual status, not a intermediate proxy value.
- Both qualitative and quantitative data feed all decision making systems.
- A team member observes something unusual and regardless of good or bad calls a team huddle to share (100)
- Team members observe something usual and good, and bring it up at the next team meeting (50)
- Observations are not gathered or recorded (-50)
- Data gathered goes into a report that gets filed somewhere with no one reviewing the observations and results (-100)
- A corporate “Agile Tool” becomes the repository for data so everyone can see the data, and no one looks at the data (-500)
✓ Critical ❑ Helpful ❑ Experimental
Steps to first adopt this practice:
- Look around and see what already exits for observable data. Use this data whenever possible. These data may exists as
- Discrete items - such as new user capabilities being used
- Those items over time - such as user capabilities being used per release/timebox or month (if doing continuous deployment)
- Ratios - such as defects injected per user capability being used per release or new defects created when fixing other defects (aka Fault Feedback Ratio)
- We look for existing data to avoid influencing behavior. When people know they’re being observed, the Hawthorne Effect (also referred to as the Observer Effect) engages and people are likely to alter (either consciously or unconsciously) their behavior
- Focus as much as possible on actual items such as shipped user capabilities rather than proxy measurements such as “Story Points”
- Think about observable team data such as:
- The number of times people get interrupted in a meeting
- How many times people laugh in a meeting
- Team members leaving and joining
Start observing. Avoid making judgments or interpretations of the data. Be a “video camera” or “tape recorder” Leave the interpretations for later. For example
|| mad / angry
Keep a record of your observations.
As you look at your observations:
- In general what do you notice?
- What exceptions to those general items do you notice?
- Can you find contradictory data?
- What surprises you?
What Does it Look like?
While metrics programs involve gathering data, we’re NOT talking a metrics program. Tom DeMarco said, “What ever gets measured, improves”. And simultaneously the shift in activities to create that improvement removes effort from some other necessary activity which then creates dysfunctional results (Robert Austin, Measuring and Managing Performance in Organizations)
Even more so, this practice does NOT address what to do with the data. This part is particularly difficult as the human mind is a sense making machine. As Goodhart’s Law states:
“When a measure becomes a target, it ceases to be a good measure.”
Data Hunting Questions
- Do I observe what I expect to observe
- Do I not observe what I don’t expect to
While useful, these questions run into the Confirmation Bias, our tendency to observe data which confirms our beliefs. There best defense against Confirmation Bias involves looking for data which disproves our current beliefs. Other useful questions are
- Do I expect to observe something but don’t
- Do I observe something I don’t expect to
These open the path to new ideas and possibilites. They can result from novel situations or looking at common events from new viewpoints.
- Dependent on who is observing
so ask yourself:
- Do our data create as complete a picture as possible
- Has my data gathering influenced behavior
- Has the data shifted over time
- How can I improve our data gathering
- Am I gathering actual not proxy data [insert link for actuals.html here]
- You have problems gathering data
- People change their behavior when they see you coming
- You quit looking for data after the first item that confirms your thinking
- You don’t keep looking for contradicting data
- Take the time to become more observant, simply watch what is normal around you and establish a baseline of what goes on
- Determine to dial down your internal dialog and try to intently listen to others and what they are communicating
- Look for areas where proxies represent a larger or more diverse group you support or need input from within your organization. Is there a way to get even a little more input from the larger community or remove the proxy?
- Take time to look at data gathered. Does it match what you see? Do you see any potential discrepancies? Is there a chance any of this data was estimated (potentially biased) in the process of gathering the data?
How to Fail Spectacularly
Anti-patterns from the Hall of Shame
- You decide your “gut” knows everything and you don’t need any stinking data
- Teammates / reports hide when they see you coming
- People modify their behavior and change the data you’re gathering
- You never stop gathering a particular set of data, you just keep adding more items to collect.
←Prev (Checklist for the Team (Stage 1))