Repository navigation
feat: Support Feast DQM for Chronon outputs with freshness and lineage context #6984
Description
Problem
Teams should be able to inspect Chronon feature quality in their existing Feast project. Our working demonstration currently profiles an exported checkout snapshot using a separate Dask/FileSource setup. That export's timestamp describes the snapshot, not freshness of Chronon's source data or serving pipeline.
The Chronon adapter can already read published Parquet for profiling, and Feast has a Python metric-computation fallback. The missing integration includes persistent monitoring storage: the adapter does not implement the monitoring storage hooks expected by MonitoringService. Online reads also currently return no feature timestamp, so they cannot establish source freshness.
Proposed solution
Start with persisted DQM over Chronon batch outputs in a Chronon-configured Feast repository:
- Implement the required metrics, baseline, and monitoring-job persistence, either through the adapter or an explicitly configured metrics backend.
- Reuse the existing Feast monitoring API/UI for profiles and historical date buckets.
- Associate monitored outputs with the Feast FeatureView and Chronon GroupBy/Join, plus output version or pipeline run when available.
- Keep event time, feature-computation time, export time, and metric-computation time distinct. Missing timestamp or provenance metadata should remain unknown.
Continuous serving-log ingestion and online/offline consistency checks should have separate follow-ups unless explicitly implemented here. Output profiling alone does not establish those properties.
Acceptance criteria
- One Chronon-configured repository computes and displays profiles, baselines, and multiple date buckets through the existing REST API/UI, without duplicating definitions in a separate Dask/FileSource repository.
- Metrics and job state survive restart, retries do not create duplicate metrics, and baseline replacement works.
- Integration fixtures verify null preservation, mapped feature names, entity keys, timestamp filtering, and known metric values.
- A stale-output fixture with a recent export is not reported as source-fresh. Absent timestamp/run metadata has an explicit unknown state.
- API/UI metadata lets users trace a monitored feature to its Chronon definition and available output provenance.
- Documentation and a runnable example explain supported storage, permissions, date semantics, and the distinction between output quality, pipeline freshness, and online/offline consistency.
Alternatives considered
Continue exporting snapshots into a separate monitoring repository, or use an external quality tool. A configurable metrics backend may be preferable to forcing monitoring writes into Chronon's feature storage; evaluate that choice in the design.
Context
Follow-up to #6188; related definition and execution metadata work: #6982 and #6983.
Starting points: Chronon offline adapter, monitoring storage contract, MonitoringService, and Chronon online adapter.
Activity
ArjunPakhan commentedon Oct 21, 2026
Hi team! I've implemented full Feast Data Quality Monitoring (DQM) support for Chronon outputs as requested in #6984.
What's included:
Persistent Monitoring Storage: Implemented all required monitoring hooks (ensure_monitoring_tables, save_monitoring_metrics, query_monitoring_metrics, clear_monitoring_baseline, and get_monitoring_max_timestamp) via atomic Parquet upserts under feast_monitoring/.*
Source Freshness & Provenance: Extended ChrononOnlineStore.online_read() to parse freshness timestamps and added lineage metadata (output_version, pipeline_run_id, export_timestamp_column) to ChrononSource.*
API/UI Integration: Exposed new REST endpoints (/monitoring/date_buckets, /monitoring/profiles) with matching UI hooks.*
Verification: Added a comprehensive integration test suite (test_chronon_dqm_integration.py) covering all acceptance criteria (null preservation, retry idempotency, baseline replacement, stale-output alerts, and restart survivability) — all 10 integration tests pass successfully locally.*
Whenever a maintainer has time to approve the workflow runs, I'd love a review on this!