The first thing that broke when John Adib's agents started querying Mixpanel was not the pipeline. It was the event names. The numbers came back wrong because the events were not described clearly enough for a machine to pick the right one.
The short answer: Once an agent rather than an analyst queries the analytics, ambiguous event names stop being hygiene and start producing wrong numbers in the business review.
What John Adib trusts, and for what
| Evidence | Trusted for | Not trusted for |
|---|---|---|
| Event names as they were | Agent queries, where the title was clear | Agent queries on unclear titles, which sometimes returned another event's data |
| Guardrails in the data layer | Revalidating numbers before people see them | Fixing a misnamed event upstream |
| Numbers agents pull for the weekly business review | Discussion, once a person has signed them off | Going to the whole business unchecked |
| Three AI reviewers on every pull request | Catching different issues, with little overlap | The merge decision, which a person makes |
Everyone became a product engineer
John has spent more than seventeen years in the industry, co-founded two startups and still codes every day. He recently made every engineer on his team a product engineer, and rewrote the job descriptions and hiring to match: define the KPI before starting, measure the impact after, talk to other departments.
Agents do much of the execution. Three AI reviewers run on every pull request, and agents attach their own evidence, including a recording of the app and the root cause of a bug. The human review moved to the end of the process, deliberately.
The hallucination was a naming problem
When agents started querying the analytics directly, they returned wrong numbers. The pipeline was fine. The event names were too ambiguous for a machine to choose between them.
John's fix was to go through every event and make it descriptive enough to disambiguate. Guardrails in the data layer revalidate numbers before they reach anyone, and a person still signs off the weekly business review, because the whole company will discuss it and nobody wants to discuss a number that is not true.
Building is no longer the constraint
He says he could rebuild his analytics tool in a weekend and will not, because writing the code is the cheap part and running it is not. His team opened agent access to its analytics, attribution, incident and error tools to the whole company, not only engineering. His closing point is that when building is cheap, the scarce thing is the judgment about what to build. For large, ambiguous features, he says, that judgment still means people checking the data before deciding.
What to do with this
Read your event list as an agent would. If two events could plausibly answer the same question, rename or describe them until only one can. Treat a rename as a change that needs review, because a report or agent query that still uses the old name can stop matching.
For seven checks drawn from across the series, see the questions to ask before trusting a number.
Frameworks this interview tests
- AARRR (pirate metrics): Dave McClure's AARRR pirate metrics count acquisition, activation, retention, referral and revenue. Each stage rests on one or two events; activation breaks most often.
- North Star Metric: A North Star Metric is the one number that captures the value customers get. It is usually built from several events, so a break in any one of them moves every team.
- Cohort retention analysis: Cohort retention analysis tracks each signup group's curve, so you can see whether retention is improving. It assumes 'active' means the same thing every month.
Read next in The Question
Source and disclosure
Read Issue 12 on A Founder's Friend. The full interview is for subscribers. This page paraphrases it and quotes nothing that sits behind the paywall.
Skene sponsored this research and commissioned it on a single question. A Founder's Friend ran the interviews, wrote them up and published them, and the series states that Skene paid for the research and did not direct the questions, the analysis or the conclusions. Each of the thirteen interviews was reviewed and signed off by the person in it, named or anonymous by their choice. Appearing in the series is not an endorsement of Skene, and nobody here is presented as a Skene customer.