Ask a large software company for its daily active users and, in Dino Repo's experience, you can get the same metric five different ways from five different places. At F-Secure, with millions of daily actives, that was the problem. At ArcticRex, with barely any users, the problem looks different. His diagnosis is the same in both places.
The short answer: Untrustworthy data is an ownership problem before it is a tooling problem, and the cheapest fix is a product-data check that runs every time the build loop turns.
What Dino Repo trusts, and for what
| Evidence | Trusted for | Not trusted for |
|---|---|---|
| Product data in a large company with legacy paths | Little: in important places there were big problems | Agreement: one metric came out five ways |
| Recorded conversations, with context attached | The dataset before there are users | Anything uncurated: one misunderstanding spreads |
| A product-data table plus a check in the loop | Knowing the events exist as the product changes | Deciding what the events should be |
| A model reviewing who was interviewed | Spotting a sample that does not match the buyer | Judging without the buyer profile as context |
Ownership, not tooling
Old products with configurable paths spread data across several places, nothing gets collected in one sensible way, and nobody feels strong ownership of what the product data should be. Product owners own a bit, the software side owns a bit, and the number comes out differently depending on who pulls it. His starting point is unglamorous: name an owner and hold them to it.
It would be good to define who owns it, and then hold that person accountable for actually owning the thing.
Generated data needs curation points
ArcticRex records every conversation, internal ones included, because both founders admit they are bad at taking notes. Transcripts become data only once context is attached, starting with who was in the room. The risk with generated data is not measurement error. It is one misunderstanding being copied into everything the system writes next.
if there’s even one misunderstanding in there, you might accidentally push it into twenty places, because it starts to live on from there.
The ten-minute prompt
Asked what he expected to be asked and was not, he gave what the series editor called the most practical advice in the series so far. Put one instruction into the harness your code goes through, telling the agent to make sure the product has data on the topic, and define a skill that names the table and the events you want.
His own setup is a product-data event table and a check in the development loop that confirms, every time the loop turns, that clicking through a flow produces the expected events. He says that is all he does, and that he is not a product data expert.
It takes you ten minutes now, but if you’re doing this for a year, or half a year, you save an absolutely staggering amount of time.
What to do with this
Name one owner per metric before you buy anything. Then make the check automatic, the way Dino's loop check is, so it runs every time the product changes rather than when someone remembers.
For seven checks drawn from across the series, see the questions to ask before trusting a number.
Frameworks this interview tests
- North Star Metric: A North Star Metric is the one number that captures the value customers get. It is usually built from several events, so a break in any one of them moves every team.
- AARRR (pirate metrics): Dave McClure's AARRR pirate metrics count acquisition, activation, retention, referral and revenue. Each stage rests on one or two events; activation breaks most often.
Read next in The Question
Source and disclosure
Read Issue 7 on A Founder's Friend. Quotations on this page are copied from the published issue.
Skene sponsored this research and commissioned it on a single question. A Founder's Friend ran the interviews, wrote them up and published them, and the series states that Skene paid for the research and did not direct the questions, the analysis or the conclusions. Each of the thirteen interviews was reviewed and signed off by the person in it, named or anonymous by their choice. Appearing in the series is not an endorsement of Skene, and nobody here is presented as a Skene customer.