This is the written version of episode 2 of our build series, where we take our own Supabase project from an empty BigQuery dataset to a table sitting at Live. It runs three minutes. Episode 1 covered what each layer of the stack owns; this is the first layer you build.
Supabase Pipelines is managed change data capture: Postgres logical replication, run for you, on the open-source Supabase ETL engine. It is in private alpha as of this writing, and BigQuery is the destination that exists today.
Key takeaways
- Two things have to be true before anything moves: a publication (what to replicate) and a destination (where it goes).
- The publication is SQL you write yourself, which is a feature. The list of what leaves your database is something you author on purpose.
- The dataset ID field wants only the part after the dot. The Google console shows
project-id.dataset-id; Supabase wantsdataset-id. - Supabase runs the pipeline and Google bills the warehouse. Budget for BigQuery separately from the database.
- A BigQuery dataset's location is set once and cannot be changed. It is the only decision here you cannot revise.
- The initial sync is the part to watch. A large, busy table can lose its replication slot mid-copy and start over.
Write the publication
The publication is SQL, not a dashboard toggle.
create publication analytics_pub
for table workspaces, repositories, checks;
That tracks inserts, updates, deletes and truncates on those three tables. You can also publish a whole schema with for tables in schema public, or the entire database with for all tables. Do not. A publication is not a backup, it is an interface, and every table in it is a promise you now have to keep. Episode 3 is the long version of why, including the tables we leave behind on purpose.
Once the publication exists it shows up under Database, then Publications, so you can see what you committed to.
Point it at BigQuery
On the Google side you need three things: a project, a dataset, and a service account key with the BigQuery Data Editor role.
In Supabase, go to Database, then Replication, click Add destination, choose BigQuery, and enable Pipelines. Then fill in the destination: a name, the publication you just created, the project ID, the dataset ID, and the service account JSON.
One field catches almost everyone on the first try. The Google console shows your dataset as project-id.dataset-id, and Supabase wants only the part after the dot. Paste the whole thing and the destination will not connect, and the error will not tell you which half was wrong.
Who gets the bill
Supabase runs the pipeline. Google runs BigQuery, and BigQuery is billed to your Google account for storage and for every byte a query scans. Nobody's free tier absorbs a wide select * against a replicated table at nine in the morning, and the person who writes that query is usually not the person watching the bill.
Budget for the warehouse separately from the database, and say out loud who owns that line before you invite the rest of the company to query it.
The field you cannot change
A BigQuery dataset's location is set when you create it and cannot be changed afterwards. Our Supabase project is in Frankfurt, so our dataset is in Frankfurt, europe-west3.
Same continent is not the same as same region. The difference shows up as egress on the Google bill, and if you have made a data-residency promise to a customer, this is the field that keeps it. It is one dropdown, it is permanent, and moving later means creating a new dataset and copying everything into it.
Then it runs
Click Create and start. Pipelines copies each table once, the initial sync, then switches that table to continuous streaming. Tables move through Copying and then sit at Live.
The initial sync is the part worth watching. A large, busy table can still be copying while writes pile up behind it, and if the copy falls far enough behind, its temporary replication slot is lost and the table starts over. If you have a big table, run the first copy during a quiet window. It is a one-time cost, and it is much cheaper than paying it twice.
What this does not give you
A table at Live means the rows are moving. It does not mean the rows are right.
Pipelines is faithful by design, and faithful replication carries a broken event rename to the warehouse as reliably as it carries a good one, with every hop green while it does. We wrote up four ways an ordinary pull request poisons a warehouse this way in Supabase Pipelines is faithful. Your tracking has to be too, and episode 8 of the series takes four real migrations through the same chain.
That gap is the one Skene works on. It reviews the pull request before the schema change lands, and comments on the line that breaks what your tracking records. Everything in this post is downstream of that.
Next in the series: what to publish, where publishing every table turns out to be the wrong answer, and it takes about a month to find out why.
Continue with Skene
How Skene works
Skene reads the event writes in your code and checks them against your Supabase schema on every PR.
Skene vs. analytics tools
Compare keeping your telemetry in your own Supabase to shipping events to hosted product analytics platforms.
Pricing
Skene OSS is free and the free audit costs nothing. Pro is $249 a month; Enterprise is quoted.
