Evaluator
Evaluator holds your plans. A plan makes one product change measurable before you launch it. You're about to ship a feature. The plan states what it should change for users, which questions the data has to answer, and which events and fields must exist to answer them. Then it checks those requirements against your actual code, so you find the missing event before launch, not three weeks after. After launch, it computes the success metric from your own data and tells you whether it moved.
Where it sits: the Lifecycle page describes your full product journey, and Events inventories the analytics calls and database evidence you already have. A plan scopes all of that down to a single launch.
Evaluator lives at /workspace/<slug>/evaluator; each plan opens at /workspace/<slug>/evaluator/<planId>.
Creating a plan
Click New plan and name the feature. The Product hypothesis field decides what happens next:
- Leave it empty for a blank plan you fill in yourself.
- Describe the hypothesis and Skene generates the first draft for you: the plan is created, the draft streams in, and it's saved to the plan (status moves to Planning).
Under Plan options, pick the Plan depth:
- Full plan: outcome, supporting metrics, and a complete event spec.
- Lightweight: a focused tracking change. Same machinery, smaller scope.
From the Skene Agent
You can also hand the Agent a hypothesis in plain language ("I think weekly digest emails will bring dormant users back"). Its build_feature_plan tool turns it into a full measurement plan: outcome metrics, questions, events, properties, acceptance criteria, and implementation notes. You don't need to provide event names.
The Agent always returns a draft first. Nothing is saved until you review the plan and explicitly confirm it.
What's in a plan
Every plan has two halves:
- A Markdown document with three sections: Target users, Problem, and Expected behavior. The narrative of who this is for and what should change.
- A structured spec shown on the Plan tab:
| Section | What it holds |
|---|---|
| Success metric | The product result this feature should change, its definition, the reporting window, the relevant segment, and supporting metrics |
| Success formula | How the metric is calculated from the data changes you connected |
| Database data changes | Product tables linked to the plan when the outcome already lives in rows |
| Required events | The events and fields that must exist before this feature is ready |
| Event definition and properties | Each event spelled out: properties, whether required, expected types |
| Implementation details | Acceptance criteria and implementation notes |
| Validation questions | What the team must learn from feature data to validate the plan |
| Optional plan details | Lifecycle steps, known limitations, and launch context |
Success formula
The formula is what Check evaluates. It combines data changes, and a data change is either an event your product sends or a table you linked under Database data changes. Neither kind is preferred: both feed the same formula.
| Field | Options |
|---|---|
| Calculation | Use one result (a single input) or Divide two results (a ratio, needs two inputs) |
| Result | The input to use. A ratio splits this into Numerator and Denominator |
| Success rule | Track only, Result is at least, Result is at most, Increase is at least, Change is at most |
To link a table, open Database data changes, pick a table from the suggestions or search, and confirm what is counted (the subject, the completion or value field, and the date field). Linking a table also registers it for Supabase Pipeline data on the Integrations page.
Event inputs count events received through Skene ingest. Verifying an event in code does not produce a metric value on its own.
The three tabs
A plan detail page has three tabs: Plan (the spec above), Verify (does the tracking exist), and Check (did the metric move).
Verify
When the plan is ready to check, use Continue to verify (or Approve plan when the plan defines no events), then run verification. Skene compares the plan's required events and fields against the analytics calls in your linked repository and the runtime evidence it has collected. The view is scoped to this plan: only the events and fields the launch depends on.
Each requirement comes back Ready (found in code or runtime evidence) or Missing. The plan's header rolls them up as Coverage: Ready or Missing fields, with a count of requirements confirmed. If Skene has no code or runtime evidence to check against yet, you'll see More evidence needed instead: connect a repository and run analysis first.
Database samples lists every table the formula depends on. A table is ready when the Development Database, the Production Database, or Pipeline has a sample, and each side is badged independently. Missing samples do not block verification.
When every requirement checks out, the plan becomes Launch ready. Ship it.
Check
Run check computes the success metric from live data. Skene fetches each formula input from the Development and Production Databases you linked (whichever are available), from Pipeline, and from event telemetry, then shows the calculation:
- The value of every input, and which side it came from
- Current, Previous, and Change over the plan's reporting window
- A goal badge: Goal met, Goal not met, No target set, or No data yet
When both databases hold data, the composed metric uses the Production Database and says so; when only the Development Database is available for the window, it uses that instead and notes it.
A check needs a success formula. Without one, the tab points you back to the Plan tab to define it.
After the numbers, Skene adds a short read on whether it moved. When the read suggests a different angle, it names a Next version focus and offers Create next version, which opens the next iteration already pointed at that focus. Check history keeps the earlier runs, each marked Ready, No data, Error, or Stale.
Measurement stages
The Evaluator list shows how far each plan has come toward a live measurement, next to its status:
| Stage | Meaning |
|---|---|
| Planning | No formula yet, or an early draft |
| Data selection | The formula needs database tables that aren't fully selected |
| Data ready | Tables selected and verified |
| Tracking ready | A check has computed the metric from Postgres |
| Live | Pipeline is publishing the tables and the snapshot comes from the warehouse |
Statuses
Draft -> Planning -> Ready to implement -> Verifying -> Launch ready -> Archived
| Status | When |
|---|---|
| Draft | Just created, nothing generated yet |
| Planning | A draft has been generated; you're refining the spec |
| Ready to implement | You approved the plan for verification (Continue to verify) |
| Verifying | Verification ran and requirements are still missing |
| Launch ready | Verification confirmed every required event and field |
| Archived | Kept for history, read-only |
Versions
Plans are versioned, and only the current version is editable. Create next version starts a fresh iteration while preserving the previous hypothesis and verification results; historical versions are immutable. Archived plans can't be edited either.
Next steps
- Lifecycle & Analysis: the lifecycle and tracking evidence your plans verify against
- GitHub Integration: link the repository verification reads
- Supabase Integration: link the databases Check reads
