Lakewright solutions / brief
Copy a Postgres table into Apache Iceberg without Spark or Kafka
One command extracts a Postgres table — or any SELECT you can write — into a typed Apache Iceberg table your warehouse reads in place. No Spark cluster, no Kafka, no connector running inside Snowflake or Databricks.
What you run
Credentials come from the environment, never a flag or a config file:
export LAKEWRIGHT_SOURCE_DSN=postgres://[email protected]/app export LAKEWRIGHT_STATE_DSN=postgres://… # audit trail + dedup journal lakewright snapshot "public.customers" --feed customers --rest $CATALOG
A bare schema.table becomes SELECT *. Anything else is your own SQL, so a filtered slice, a view, or a five-table join lands as one table — you are not restricted to mirroring what the schema happens to look like.
No schema file, no DDL
The result set is serialised with row_to_json on the server, which turns every Postgres type into one string problem we already solve. Lakewright then profiles the values and picks the Iceberg column types — integers, decimals, booleans, timestamps — the same inference the CSV and JSON paths use. You do not write a mapping, and you do not hand-maintain a CREATE TABLE that drifts from the source.
Running it again is a no-op
The identity of a snapshot is the digest of its canonical rows, journaled alongside the load. Re-run it and an unchanged result set commits nothing at all — no new Iceberg snapshot, no duplicate rows, no wasted warehouse credits. That is what makes this safe to put on a schedule for the reference and dimension tables that change rarely: currency codes, plan catalogues, provider directories, the store list.
For a table that changes continuously, this is the wrong tool and the sibling brief is the right one — lakewright listen tails Postgres logical replication into an append-only changelog instead, which is how you get every intermediate state rather than a series of stills.
FAQ
The question that matters
"Which accounts in the product database have never appeared in a payment file?"
Nobody asks this of Postgres, because Postgres cannot see the payment files — they arrive as BAI2 or as 835 remittances and live somewhere else entirely. It is a question about the join, and the join has no home until both sides are in the same lakehouse.
Once the accounts table lands in Iceberg beside the banking feeds, the answer is an anti-join anyone can write, and in the sample data it returns two distinct populations that deserve very different treatment: a long tail of genuinely dormant accounts, and a much smaller group that is active, invoiced, and simply never reconciled — which is the group worth a phone call this week.
That second group is the reason to land the operational database at all. It is invisible in Postgres, invisible in the bank files, and obvious the moment the two sit next to each other.
Want this run against your data?
We start with a read-only scan and an inventory report — no installation on your systems, nothing leaves your environment. Most engagements produce findings the team didn't know about in the first afternoon.