Skip to content

fix: Report feature freshness over the live workers only - #7087

Open
casaar97 wants to merge 1 commit into
feast-dev:masterfrom
casaar97:casaar97/freshness-gauge-livemax
Open

casaar97 wants to merge 1 commit into
feast-dev:masterfrom
casaar97:casaar97/freshness-gauge-livemax

Conversation

@casaar97

@casaar97 casaar97 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

feast_feature_freshness_seconds is declared with multiprocess_mode="max",
so it reports the staleness of workers that have already exited.

In multiprocess mode, prometheus_client keeps one file per process.
multiprocess.mark_process_dead(pid), which the Gunicorn child_exit hook
calls, deletes only the files whose mode starts with live:

_LIVE_GAUGE_MULTIPROCESS_MODES = {m for m in Gauge._MULTIPROC_MODES if m.startswith('live')}

A gauge in mode max writes gauge_max_<pid>.db, and nothing deletes it. The
exported value is therefore the maximum over every worker that ever ran, and
not over the workers that are running.

Effect

Each worker computes the freshness of every feature view every 30 seconds. A
worker that exits while it holds a high staleness leaves that value behind.
After the next materialization the live workers compute a low value, but the
exported maximum keeps the value of the worker that exited. The gauge rises to
a peak and holds it, and a materialization that succeeds does not bring it
down.

This needs no unusual configuration. feast serve sets --max-requests to
1000 by default (sdk/python/feast/cli/serve.py), so a server under steady
traffic recycles its workers continuously, and each exit leaves one more file
that is never deleted. The error grows until the gauge no longer follows the
data at all. Any alert on this metric gives false alerts, and the metric
cannot report that the data became fresh again.

How this was found

A feature view that materializes once a day showed a staleness just under
24 hours that stopped moving, although every materialization ran on time. Two
observations separate a leftover file from a monitoring thread that hangs:

  1. Two servers that read the same registry reported 3.9 h and 24.0 h at the
    same moment. 3.9 h was the true age of the data.
  2. Inside one server, 16 of 17 feature views were frozen and one advanced in
    real time. The one that advanced was the view whose live value was above
    its own leftover value. A thread that hangs freezes all of them.

The frozen values also fell into clusters, one cluster per worker that had
exited.

Fix

Use livemax. mark_process_dead then removes the file of each worker that
exits, and the gauge reports the worst case over the live workers. The
resource gauges above it already use liveall for the same reason.

Which issue(s) this PR fixes:

None filed.

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Added TestFeatureFreshnessMultiprocessMode in
sdk/python/tests/unit/test_metrics.py. It sets a value, calls
mark_process_dead, and asserts that the collector no longer returns it. It
reads the mode off the real gauge, so it fails on max and passes on
livemax. It runs the check in a subprocess, because prometheus_client
binds the directory of a gauge file prefix on first use, and another test in
the same file binds it first.

Misc

Release note:

Fix feast_feature_freshness_seconds, which reported the staleness of Gunicorn workers that had already exited.

The two documentation pages that list this metric,
docs/reference/feature-servers/python-feature-server.md and
docs/how-to-guides/online-server-performance-tuning.md, describe it as
"seconds since last materialization". This change makes that description
correct, so they need no edit.

@casaar97
casaar97 requested a review from a team as a code owner October 29, 2026 08:07
@codecov-commenter

codecov-commenter commented Oct 29, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.99%. Comparing base (1e5e90c) to head (dc4eee4).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7087      +/-   ##
==========================================
+ Coverage   50.97%   50.99%   +0.01%     
==========================================
  Files         455      455              
  Lines       56994    56994              
  Branches     8321     8321              
==========================================
+ Hits        29055    29063       +8     
+ Misses      25954    25946       -8     
  Partials     1985     1985              
Flag Coverage Δ
go-feature-server 30.58% <ø> (+0.20%) ⬆️
python-unit 52.45% <ø> (ø)
Files with missing lines Coverage Δ
sdk/python/feast/metrics.py 86.19% <ø> (ø)

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e5e90c...dc4eee4. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@casaar97 casaar97 changed the title fix: report feature freshness over the live workers only fix: Report feature freshness over the live workers only Oct 29, 2026
@casaar97

Copy link
Copy Markdown
Contributor Author

/kind bug

@casaar97
casaar97 force-pushed the casaar97/freshness-gauge-livemax branch from c9e7d61 to dc4eee4 Compare October 29, 2026 08:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants