Repository navigation
feat: Add PolarisSource and Polaris governance sync on top of IcebergSource #7005
Copy link
Copy link
Closed
Labels
kind/featureNew feature or requestNew feature or request
Description
Is your feature request related to a problem? Please describe.
Follow-up to #6499. IcebergSource (catalog_type="rest") already works against any Iceberg REST catalog, and UnityCatalogSource + UnityCatalogProvider add Unity Catalog governance on top. There is no equivalent for Apache Polaris, and a few gaps make plain IcebergSource awkward to use against Polaris today:
- Auth. Only a static bearer token via
token_env_varis supported. Polaris uses OAuth2 client credentials with short-lived tokens, so long-runningfeast materialize/ serving jobs break when the token expires. The Polaris/v1/oauth/tokensendpoint is also marked deprecated for removal in the spec, so external IdP tokens need to be supported too. - Defaults. Users must hand-write
endpoint(<host>/api/catalog) and the token env var for every source. - Read path. In
duckdb.py::_read_iceberg_catalog_sourcethe REST branch callspyiceberg.load_catalogwithoutX-Iceberg-Access-Delegation, so credential vending from Polaris is not used, and the table is read with a fullscan().to_arrow()(no column projection / predicate pushdown). - Governance. Nothing records which Polaris tables are backing Feast feature views (project, feature view, owner, tags), unlike UC.
- Discovery. No helper to enumerate Polaris namespaces/tables and build sources from them.
Describe the solution you'd like
Mirror the UC layering, keeping Polaris-specific code out of the generic IcebergSource:
PolarisSource(IcebergSource): defaults fromPOLARIS_HOST/POLARIS_TOKEN(endpoint<host>/api/catalog,warehouse= Polaris catalog name), plusregister_as_feature_table/sync_lineage-style flags, serialized viacustom_options.PolarisClient: governance calls that are available on Polaris, using the standard IcebergupdateTablewithset-propertiesto writefeast.project,feast.feature_view,feast.owner,feast.tag.*,feast.is_feature_table,feast.deleted_at. Lineage has no Polaris API, so it is out of scope (or an optionalfeast.upstreamproperty).PolarisProvider(PassthroughProvider): onfeast apply/ delete, sync those properties for views whose batch source is aPolarisSource(same shape asUnityCatalogProvider; never drops the table).- OAuth2 client-credentials token acquisition and refresh for the REST client (usable by
IcebergSourceas well). - Route the DuckDB (and Spark) read path through credential vending. (Column projection / filter pushdown was split out into feat: Push column projection and time-range filters down to Iceberg scans in the DuckDB and Spark offline stores #7019.)
- A small discovery helper that lists namespaces/tables and returns
PolarisSourceobjects for the user to wire into feature views.
Describe alternatives you've considered
- Use plain
IcebergSourceagainst Polaris: works for reads with a static token, but no refresh, no vended credentials on the DuckDB path, and no governance sync. - Use Polaris as the Feast registry: not a fit. Polaris is a table catalog (Iceberg REST, Generic Tables, Policies), not a transactional object/blob store, so it does not provide the atomic-update semantics the registry needs.
- Policy API for governance: possible later, but properties on the table are the minimal, standards-based first step.
Additional context
- Depends on the
iceberg_catalogcontrib package already on master (feat: Extend Feast's DataSource to natively support Iceberg REST Catalog-backed tables #6499). - Based on reading the Polaris REST specs and the existing
iceberg_catalogcode; not yet validated end to end against a running Polaris. Happy to put up a PR if the direction looks right.
Metadata
Metadata
Assignees
Labels
kind/featureNew feature or requestNew feature or request
Activity
haoxu0 commentedon Oct 15, 2026
Implemented in #7006 (merged):
PolarisSource,PolarisClient(OAuth client-credentials with refresh, table properties via IcebergupdateTable),PolarisProvider(provider: polaris), credential vending on the DuckDB/Spark read paths, andPolarisSource.discover(). The shared PyIceberg catalog config was also consolidated intoIcebergSource.Column projection / filter pushdown was intentionally left out and is tracked in #7019.