Skip to content
Resources / Frameworks

Churn prediction scorecard: what the model learns when an event breaks

Churn prediction scorecard. Credit: A customer success practice with no single originator. By Teemu Kinos, co-founder of Skene.

A churn prediction scorecard flags accounts likely to leave, using leading indicators such as declining usage, rising support volume and souring sentiment, early enough to act before the cancellation request arrives. It can be a set of rules or a trained model. Either way it learns from history, so a tracking break does not just produce one wrong alert. It teaches the model something false.

How it works

The simple version is rules: usage down 40% over four weeks, no admin login in thirty days, a support ticket marked urgent. The trained version learns weights from accounts that churned and accounts that stayed. Dave Boyce's Freemium describes Snyk's version, which scores product-qualified accounts for churn risk, maps each risk tier to an action, from a reminder email to a person, and feeds the results back into the model (Ch 11).

Every one of those features is an event or a count of events. A churn model cannot tell an account that stopped using a feature from a feature that stopped reporting, because in the data they look identical.

What it assumes your events already do

EventWhat the framework uses it forWhat it reports when the event breaks
Core value action
report_viewed
Usage decline, the strongest leading indicator.A dropped event looks like abandonment, and save plays go to accounts that are fine.
Return visit
session_started
Login recency.A move to single sign-on can change how logins are recorded, and every account using it looks dormant.
Activation or First Impact event
first_report_shared
Early-life features, such as whether the account finished setup.If the setup event changes, the model loses its best early signal for new accounts.

Event names are illustrative; yours will differ. The dependency map shows which other frameworks share each event.

The model that learned a release date

Illustrative example. The company and the figures are invented to show the mechanism.

A mobile release drops the sync_completed event for Android users for three weeks before anyone notices. Android-heavy accounts all show a usage collapse, and a few of them churn that quarter for unrelated reasons.

When the model is retrained, it learns that a sudden sync drop is a strong churn signal. For the next year it flags Android-heavy accounts after every app update, and the success team learns to ignore it.

Checks before you trust it

  • Exclude periods with known tracking breaks from training data, and keep a log of them.
  • Separate sudden drops from gradual ones; most real churn is gradual.
  • Hold out some flagged accounts with no action, so you can see whether the save plays work.
  • Re-check feature definitions whenever the product changes an event the model uses.

Heard in The Question

Interviews in The Question that bear on this framework:

Check the events under it

Before Churn prediction scorecard goes into a plan, find out whether the events it needs exist and still fire the way it assumes. The free audit maps what your product collects today: sign in, connect a GitHub repository or Supabase, and Skene returns one audit, with no card. On Pro, Skene Cloud reviews every pull request on a linked GitHub repository for removed or broken tracking, so a pull request that removes or renames one of these events is flagged before it merges.