Repository navigation
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7087 +/- ##
==========================================
+ Coverage 50.97% 50.99% +0.01%
==========================================
Files 455 455
Lines 56994 56994
Branches 8321 8321
==========================================
+ Hits 29055 29063 +8
+ Misses 25954 25946 -8
Partials 1985 1985
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Contributor
Author
|
/kind bug |
Signed-off-by: Carlos Sánchez <[email protected]>
casaar97
force-pushed
the
casaar97/freshness-gauge-livemax
branch
from
October 29, 2026 08:17
c9e7d61 to
dc4eee4
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
feast_feature_freshness_secondsis declared withmultiprocess_mode="max",so it reports the staleness of workers that have already exited.
In multiprocess mode,
prometheus_clientkeeps one file per process.multiprocess.mark_process_dead(pid), which the Gunicornchild_exithookcalls, deletes only the files whose mode starts with
live:A gauge in mode
maxwritesgauge_max_<pid>.db, and nothing deletes it. Theexported value is therefore the maximum over every worker that ever ran, and
not over the workers that are running.
Effect
Each worker computes the freshness of every feature view every 30 seconds. A
worker that exits while it holds a high staleness leaves that value behind.
After the next materialization the live workers compute a low value, but the
exported maximum keeps the value of the worker that exited. The gauge rises to
a peak and holds it, and a materialization that succeeds does not bring it
down.
This needs no unusual configuration.
feast servesets--max-requeststo1000 by default (
sdk/python/feast/cli/serve.py), so a server under steadytraffic recycles its workers continuously, and each exit leaves one more file
that is never deleted. The error grows until the gauge no longer follows the
data at all. Any alert on this metric gives false alerts, and the metric
cannot report that the data became fresh again.
How this was found
A feature view that materializes once a day showed a staleness just under
24 hours that stopped moving, although every materialization ran on time. Two
observations separate a leftover file from a monitoring thread that hangs:
same moment. 3.9 h was the true age of the data.
real time. The one that advanced was the view whose live value was above
its own leftover value. A thread that hangs freezes all of them.
The frozen values also fell into clusters, one cluster per worker that had
exited.
Fix
Use
livemax.mark_process_deadthen removes the file of each worker thatexits, and the gauge reports the worst case over the live workers. The
resource gauges above it already use
liveallfor the same reason.Which issue(s) this PR fixes:
None filed.
Checks
git commit -s)Testing Strategy
Added
TestFeatureFreshnessMultiprocessModeinsdk/python/tests/unit/test_metrics.py. It sets a value, callsmark_process_dead, and asserts that the collector no longer returns it. Itreads the mode off the real gauge, so it fails on
maxand passes onlivemax. It runs the check in a subprocess, becauseprometheus_clientbinds the directory of a gauge file prefix on first use, and another test in
the same file binds it first.
Misc
Release note:
The two documentation pages that list this metric,
docs/reference/feature-servers/python-feature-server.mdanddocs/how-to-guides/online-server-performance-tuning.md, describe it as"seconds since last materialization". This change makes that description
correct, so they need no edit.