Enterprise search & secure generative AI over a petabyte-scale compliance corpus
A global UN agency needed its compliance, audit and KYC teams to find the right clause in seconds across millions of pages — without a single uncited answer, and without moving sensitive data outside its own cloud perimeter.
The challenge
Compliance and audit officers across regional offices were spending days manually searching policies, regulations, KYC files and audit evidence spread across shared drives, document systems and email. A keyword search returned thousands of hits; a wrong answer was worse than no answer. The organisation's standards were strict: answers had to be traceable to a source document, access had to respect existing permissions, and nothing could leave its own cloud tenancy or be used to train a public model.
Earlier pilots had looked good on a handful of sample documents and stalled on the real corpus — scale, document quality, permissions and auditability.
The approach
- Frameone corpus, measurable outcomes
- Ingestpetabyte-scale pipeline
- Groundpermission-aware search
- Evaluatecited answers, red-team
- OperateDevSecOps, monitoring
- Frame the first use case tightly. KYC and audit document lookup for a defined user group, with success measured in time-to-answer and citation accuracy rather than "AI adoption".
- Build the ingestion pipeline for the real corpus, not the sample. Dataflow-based ingestion with document parsing, chunking and metadata extraction, handling scanned and mixed-quality files; over 1 TB processed in the first phase.
- Ground every answer. Vertex AI Search over the indexed corpus, with access control lists mirrored from the organisation's identity provider so each user only ever sees what they are already entitled to see. Generated answers quote and link the passages they came from, and decline when nothing supports them.
- Evaluate before expanding. An evaluation set written with compliance officers, red-team prompts for leakage and hallucination, and a sign-off gate before each new team was onboarded.
- Run it like production. Infrastructure as code, CI/CD with security scanning, centralised logging and audit trails — the DevSecOps posture the agency's security team expected.
What we built
- Secure enterprise search with generative answers, citations and permission-aware retrieval
- Scalable ingestion pipeline for policies, regulations, KYC and audit records (1 TB+)
- Migration of the search workload into the agency's own Google Cloud tenancy
- DevSecOps pipelines, monitoring and an audit trail for every query and answer
- Evaluation set and runbooks handed to the agency's internal team
Results
- 200+ active users across compliance, audit and KYC functions
- Search time cut from days of manual cross-reading to seconds, with a citation on every answer
- 100+ core compliance documents indexed in the first phase, with the pipeline built to scale to the full corpus
- Passed internal security review: data stays in-tenancy, no training on agency content, full audit trail
What made it work
Permissions first
Mirroring existing entitlements into retrieval was the single biggest trust-builder — and the thing most pilots skip.
Evaluation is the product
A written evaluation set turned 'does it work?' into a number the steering group could sign off.
Ship the operations too
Logging, CI/CD and runbooks are why the platform is still running without the vendor in the room.
Government and enterprise programmes are described under confidentiality agreements; client identities, figures and architectures are generalised where required. Marketing results are reported as the clients described them.