Skip to content

feat: Add PolarisSource and Polaris governance sync on top of IcebergSource #7005

Closed

Description

@haoxu0

Is your feature request related to a problem? Please describe.

Follow-up to #6499. IcebergSource (catalog_type="rest") already works against any Iceberg REST catalog, and UnityCatalogSource + UnityCatalogProvider add Unity Catalog governance on top. There is no equivalent for Apache Polaris, and a few gaps make plain IcebergSource awkward to use against Polaris today:

  1. Auth. Only a static bearer token via token_env_var is supported. Polaris uses OAuth2 client credentials with short-lived tokens, so long-running feast materialize / serving jobs break when the token expires. The Polaris /v1/oauth/tokens endpoint is also marked deprecated for removal in the spec, so external IdP tokens need to be supported too.
  2. Defaults. Users must hand-write endpoint (<host>/api/catalog) and the token env var for every source.
  3. Read path. In duckdb.py::_read_iceberg_catalog_source the REST branch calls pyiceberg.load_catalog without X-Iceberg-Access-Delegation, so credential vending from Polaris is not used, and the table is read with a full scan().to_arrow() (no column projection / predicate pushdown).
  4. Governance. Nothing records which Polaris tables are backing Feast feature views (project, feature view, owner, tags), unlike UC.
  5. Discovery. No helper to enumerate Polaris namespaces/tables and build sources from them.

Describe the solution you'd like

Mirror the UC layering, keeping Polaris-specific code out of the generic IcebergSource:

  • PolarisSource(IcebergSource): defaults from POLARIS_HOST / POLARIS_TOKEN (endpoint <host>/api/catalog, warehouse = Polaris catalog name), plus register_as_feature_table / sync_lineage-style flags, serialized via custom_options.
  • PolarisClient: governance calls that are available on Polaris, using the standard Iceberg updateTable with set-properties to write feast.project, feast.feature_view, feast.owner, feast.tag.*, feast.is_feature_table, feast.deleted_at. Lineage has no Polaris API, so it is out of scope (or an optional feast.upstream property).
  • PolarisProvider(PassthroughProvider): on feast apply / delete, sync those properties for views whose batch source is a PolarisSource (same shape as UnityCatalogProvider; never drops the table).
  • OAuth2 client-credentials token acquisition and refresh for the REST client (usable by IcebergSource as well).
  • Route the DuckDB (and Spark) read path through credential vending. (Column projection / filter pushdown was split out into feat: Push column projection and time-range filters down to Iceberg scans in the DuckDB and Spark offline stores #7019.)
  • A small discovery helper that lists namespaces/tables and returns PolarisSource objects for the user to wire into feature views.

Describe alternatives you've considered

  • Use plain IcebergSource against Polaris: works for reads with a static token, but no refresh, no vended credentials on the DuckDB path, and no governance sync.
  • Use Polaris as the Feast registry: not a fit. Polaris is a table catalog (Iceberg REST, Generic Tables, Policies), not a transactional object/blob store, so it does not provide the atomic-update semantics the registry needs.
  • Policy API for governance: possible later, but properties on the table are the minimal, standards-based first step.

Additional context

Activity

  1. haoxu0 commented on Oct 15, 2026

    @haoxu0
    CollaboratorAuthor

    Implemented in #7006 (merged): PolarisSource, PolarisClient (OAuth client-credentials with refresh, table properties via Iceberg updateTable), PolarisProvider (provider: polaris), credential vending on the DuckDB/Spark read paths, and PolarisSource.discover(). The shared PyIceberg catalog config was also consolidated into IcebergSource.

    Column projection / filter pushdown was intentionally left out and is tracked in #7019.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions