Eval / Data Flywheel + Governed Feature Store + Recommendation Platform

Close the loop: evaluations and production signals feed a governed feature store.

This system is planned. The case study describes the intended design; metrics are targets or planned, not measured results.

Problem

Without a flywheel, evaluation is ad hoc and features drift silently. Model quality regresses and no one notices until users do.

Why this matters

Evaluation rigor and the data flywheel are the moat. Labs and applied-AI teams live or die on how well they measure quality and how quickly production signals turn into better data and models.

Constraints

Offline and online features must be consistent.
Every feature and dataset has documented lineage.
Eval regressions block promotion above a threshold.
Data quality (schema, freshness, nulls) is validated at boundaries.

Architecture

Data

Ingestion + contracts
Quality checks

Features

Feature storeoffline + online
Lineage

Serving

Ranking / recs
Eval harness

Flywheel

Feedback signals
Dataset refresh

Raw data is validated and materialized into a feature store; a ranking service serves features online; evaluation runs on golden and regression sets, and feedback signals loop back into datasets.

Data flow

Raw data is validated, materialized into offline/online features, served to ranking, evaluated on golden sets, and production feedback flows back to refresh datasets.

Control plane vs data plane

Control: Data contracts, lineage, eval thresholds, and promotion gates.

Data: Feature materialization and online serving to the ranking/recommendation path.

Core capabilities

A governed feature store with point-in-time correctness.
An eval harness with golden sets and regression tracking.
A ranking / recommendation service consuming governed features.
Production signals routed back into the training data.

Staff-level tradeoffs

Governed feature store over ad-hoc feature code.
Point-in-time correctness and lineage prevent training/serving skew and silent drift.
Eval gates as CI, not manual review.
Regression above a threshold should fail automatically, like any other test.

Tech stack

ML / Data

Python
feature store
batch + streaming

Backend

Ranking service
eval harness

Infrastructure

Kubernetes

Governance

data contracts
lineage
regression gates

Metrics

Train/serve feature skew: Point-in-time correct
Eval regression gate: Blocks promotion above threshold
Data lineage coverage: Raw to feature to model
Online feature p99: Bounded for serving

Metrics are labeled measured, target, or planned. Nothing here is an achieved result unless it is marked measured.

Failure modes

Train/serve skew detected
Serving falls back to a safe default and alerts rather than serving skewed features.
Eval regression over threshold
Promotion is blocked automatically in CI.
Data contract violation
The pipeline halts at the boundary instead of propagating bad data.

What's next

Define feature/data contracts and a small golden eval set.
Implement the offline store and one online-served feature.