Skip to content
Faidhi Fahmi.Contact
All selected works
AIDataTechnical

A quarter-million contacts, made searchable

Organisation
tmrwX · vector database & marketing persona system
Role
Product and data architecture
Period
2026 – present · Malaysia

Migrating a long-accumulated contact database to Neon PostgreSQL with pgvector, turning raw records into embedding-indexed personas, and putting a governance layer around who can query and export them.

At a glance

Situation
250K–300K contact records accumulated over years of insurance operations. The data was rich but effectively unusable because no one could ask the database a question without an engineer.
Stakes
The working alternative was ad-hoc exports of a customer database to personal laptops, with no record of who took what.
My role
Product and data architecture: the migration, the cleansing pipeline, the persona and segment model, and the governance layer on top.
Constraints
Years of inconsistent records, sensitive customer data, and non-technical operators as the end users of the query surface.
What changed
  • 26 improvised segments collapsed into 14 with rules anyone can say out loud
  • Cohort building went from a ticket to an engineer to a page marketing uses directly
  • Every preview and export is audit-logged against an allowlist of recipients
Read this if
You have data everyone wants to use and nobody can query safely.
contact records migrated
250K–300Kcontact records migrated
segments, down from 26 improvised ones
14segments, down from 26 improvised ones
whitelisted filter fields, parameterised only
35+whitelisted filter fields, parameterised only
preview and export audit-logged
Everypreview and export audit-logged

How the system actually worked

From raw records to a cohort somebody is allowed to exportA five-stage pipeline. Extract: 250,000 to 300,000 contact records migrated off RDS to Neon PostgreSQL with pgvector. Cleanse, which is where most of the calendar time goes: phone normalisation to one canonical format, deduplication, and three-point ownership matching. Embed: records become personas searchable by similarity rather than only filterable by column. Segment: 26 improvised segments collapsed into 14, being six behavioural types split by whether the contact is inside or outside a 35km radius, plus two buckets for contacts with no location signal. Then a governance gate: a registry of 35-plus whitelisted filter fields compiled to parameterised SQL, a preview of the exact export rows, and an allowlist of permitted recipients. Only then does a cohort reach an approved recipient. Every preview and every export writes an audit row.Extract250–300K recordsRDS → Neon + pgvector1CLEANSENormalise, dedupe,match ownershipone canonical phone format2Embedpersonas searchableby similaritySegment14, down from 266 types × 35km radius3Governance gateregistry · preview · allowlist4Approved recipientcampaign-ready cohortBlockednot on the allowlistnot permittedAn audit row for every preview and every exporta permanent record of who looked at what

Scroll the figure sideways to read it, or turn your phone.

  1. 250–300K records migrated off RDS. Rich, and effectively unusable, because nobody could ask it a question without an engineer.
  2. Cleansing is where the calendar time goes and where every downstream mistake originates: phone normalisation, deduplication, three-point ownership matching.
  3. 26 improvised segments became 14, on a rule anyone can say out loud: six behavioural types, split by inside or outside 35km, plus two for contacts with no location signal. Location leads, because distance predicts attendance at a physical venue better than any behavioural score.
  4. The field registry is the only way to add a filter. Column names are trusted literals defined in one place and request input is never interpolated, which makes the feature slower to extend and stops an admin query builder quietly becoming an injection surface.
Extract, cleanse, embed, segment, activate. The cleanse band is where most of the calendar time goes and where every downstream mistake originates.

Stack & practices

  • Neon PostgreSQL
  • pgvector
  • Embeddings
  • Semantic search
  • Express
  • TypeScript
  • React

Got a problem shaped like this one?

A short conversation is usually enough to see if there's a fit.

Email meRead another case study