A governed data layer for AI features — BigQuery foundation with lineage and access controls
The product team wanted AI features — summaries, recommendations, natural-language reporting — but every prototype answered from inconsistent, partly duplicated data. The fix wasn't a better model; it was a data layer the AI could trust.
The challenge
Operational data was copied into several analytics stores with different definitions of the same metrics. Access was all-or-nothing, so AI prototypes either saw too much or too little. Nobody could say where a number came from — which blocked both compliance sign-off and customer trust in any AI feature built on top.
The approach
- Modelone definition per metric
- PipelineDataflow → BigQuery
- Governlineage + row-level access
- Checkquality gates, freshness
- EnableAI features on top
- Agree the definitions. A governed semantic layer with one definition per metric, owned by named people.
- Build reliable pipelines. Dataflow ingestion into BigQuery with transformations under version control and tests.
- Make governance visible. Lineage captured automatically, row- and column-level access policies tied to the product's own permissions, and freshness and quality checks that block bad data from reaching AI features.
- Then ship AI. With a trustworthy layer in place, the first natural-language reporting feature went from prototype to production in weeks.
What we built
- BigQuery foundation with a governed semantic layer and versioned transformations
- Automated lineage and row/column-level access controls
- Data quality and freshness gates with alerting
- The first production AI feature (natural-language reporting) on top of the layer
Results
- One source of truth for metrics that product, finance and AI features all use
- Compliance sign-off for AI features, because every answer traces to governed data
- AI prototypes stopped being blocked on data quality — the first feature shipped within weeks
- Access mirrors the product's own permissions, so AI features never over-share
What made it work
Fix the data, then the model
Most 'AI accuracy' problems were data-definition problems in disguise.
Lineage sells itself
Being able to click from an AI answer back to the source table is what won compliance over.
Permissions are data too
Encoding access policies in the warehouse meant every AI feature inherited them for free.
Government and enterprise programmes are described under confidentiality agreements; client identities, figures and architectures are generalised where required. Marketing results are reported as the clients described them.