Repository navigation
Remote materialization #4526
Description
Is your feature request related to a problem? Please describe.
We already have an option to run online/offline store queries remotely through feature server and offline server, respectively. This way rbac rules will be applied on operations. One piece that's missing is materialization. There are several ways to do this:
-
Keep materialization local, but rely on remote online/offline engines to apply rbac rules. This is currently impossible because remote online client doesn't implement
online_write_batchmethod. Even if we did implement it in the feature server itself, we would essentially be using a lightweight fastapi server to transport batches and batches of potentially huge datasets. -
Create remote materialization engine that will defer the whole materialization call to a backend server and apply rbac rules there. We can create another server component MaterializationServer that will receive these requests.
-
The same as above but instead of creating a new component, we can reuse OfflineServer to do the request handling. This is slightly awkward from the naming perspective, but probably makes the most sense in term of usage/maintenance.
I'd probably go with option 3 as a starting point.
Activity
dmartinol commentedon Sep 18, 2024
ATM the
OfflineServerwas designed to implement theOfflineStoreinterface only. Adding a method to write to the online stores would introduce an unplanned dependency and raise some concerns.remote)online_store, which would raise again the original issue?OfflineServer, then we'd need an actualonline_storeconfiguration as well, which was not planned to have in such server (aremoteonline store was designed instead). Not sure about any possible side effects.Why aren't we using the
/materialize-incrementalendpoint on theFeatureServerinstead? (and add a new endpoint for non-incremental jobs)This would avoid any "transport batches and batches of potentially huge datasets." as it would work on the server itself (and would use the remote
offline_storetopull_latest_from_table_or_queryusing the flight protocol) .Otherwise, I'd be in favour of a dedicated
MaterializationServer(with remoteoffline_storeand providedonline_store), which can still be designed as a "lightweight fastapi server" if I understood the materialization flow.dmartinol commentedon Sep 23, 2024
@tokoko do we want to evaluate this one? Any further comments on what solution to apply?
tmihalac commentedon Dec 17, 2024
@tokoko we would like to see how do we progress with the remote materialization, how do we progress from here ?
stale commentedon Apr 26, 2025
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
feat: Add remote materialization support via feature server
feat(server): Add async materialization via ?async=true query param
feat(server): Remote Materialization (#6649)
chore(release): release 0.66.0
chore(release): release 0.66.0