A quarter-million contacts, made searchable
- Organisation
- tmrwX · vector database & marketing persona system
- Role
- Product and data architecture
- Period
- 2026 – present · Malaysia
Migrating a long-accumulated contact database to Neon PostgreSQL with pgvector, turning raw records into embedding-indexed personas, and putting a governance layer around who can query and export them.
- contact records migrated
- 250K–300Kcontact records migrated
- segments, down from 26 improvised ones
- 14segments, down from 26 improvised ones
- whitelisted filter fields, parameterised only
- 35+whitelisted filter fields, parameterised only
- preview and export audit-logged
- Everypreview and export audit-logged
How the system actually worked
Scroll the figure sideways to read it, or turn your phone.
- 250–300K records migrated off RDS. Rich, and effectively unusable, because nobody could ask it a question without an engineer.
- Cleansing is where the calendar time goes and where every downstream mistake originates — phone normalisation, deduplication, three-point ownership matching.
- 26 improvised segments became 14, on a rule anyone can say out loud: six behavioural types, split by inside or outside 35km, plus two for contacts with no location signal. Location leads, because distance predicts attendance at a physical venue better than any behavioural score.
- The field registry is the only way to add a filter. Column names are trusted literals defined in one place and request input is never interpolated — which makes the feature slower to extend and stops an admin query builder quietly becoming an injection surface.
Context
A contact database in the high hundreds of thousands of records sitting in RDS, accumulated over years of insurance operations — vehicles, orders, leads, addresses, inbound WhatsApp threads. Rich, and effectively unusable for marketing, because nobody could ask it a useful question without an engineer.
What was actually needed
A way to get from "who should we talk to this month" to a campaign-ready list without a one-off query — and without anyone quietly exporting the customer database to a personal laptop.
How I used AI here
Embeddings turn contact records into personas that can be searched by similarity rather than only filtered by column. That means you can ask for contacts resembling a cohort that already converted, instead of hand-writing the rule that you hope describes them. The embedding layer proposes; the segment rules and the governance layer still decide.
What I did technically
The migration from RDS to Neon PostgreSQL with pgvector, the cleansing pipeline underneath it — phone normalisation to one canonical format, deduplication, three-point ownership matching — and a Data Explorer on top: a registry of filterable fields compiled into parameterised SQL, a preview showing the exact export rows before anything leaves, an allowlist of permitted recipients, and an audit row for every preview and every export.
Decisions that mattered
Collapsing 26 improvised segments into 14 with a rule anyone can say out loud: six behavioural types, each split by whether the contact sits inside or outside a 35km radius, plus two buckets for contacts with no location signal. Location leads, because for a business whose events happen at a physical venue, distance predicts attendance better than any behavioural score. And making the field registry the only way to add a filter — column names are trusted literals defined in one place, request input is never interpolated. It makes the feature slower to extend, and it means an admin query builder cannot quietly become an injection surface.
What changed
Marketing can build a cohort, see exactly which rows it contains, and send it to an approved recipient — with a permanent record of who looked at what. Questions that used to be a ticket to an engineer are now a page.
The insight I kept
Giving non-technical teams direct access to customer data is a governance design problem before it is a UI problem. The preview, the allowlist and the audit log are what made it safe enough to actually ship.
Stack & practices
- Neon PostgreSQL
- pgvector
- Embeddings
- Semantic search
- Express
- TypeScript
- React