Repository navigation
Shift ingestion feature metrics to after store write #489
Description
Is your feature request related to a problem? Please describe.
Existing metrics collection for ingestion is done in parallel to store writes:
+
|
+-------+--------+
| |
+------+-----+ +------+-----+
| | | |
| store | | metrics |
| write | | write |
| | | |
+------------+ +------------+
While this does give a rough gauge as to what data looks like in flight, it doesn't actually reflect the status of the data in the store, which is ultimately what we are concerned about, since write failures are common.
Describe the solution you'd like
Ideally we want to write metrics POST store writes, as so:
...
+
|
+-----+-----+
| |
| store |
| write |
| |
+-----+-----+
|
+--------+--------+
+ +
success failure
+ +
| |
+-----+-----+ +-----+-----+
| write | | write |
| sucess | | failure |
| metrics | | metrics |
+-----------+ +-----------+
Metrics collected post store writes would more accurately reflect the data in the stores.
Reacted by Willem Pienaar and Ches Martin
Activity