Feast 0.5 release #527

We're going to try and make a habit of discussing and evolving our roadmaps using GitHub issues. For each release this will allow folks to see what changes are upcoming, propose new changes to include in the release, and also understand why certain decisions have been made.

I have moved the current roadmap for Feast 0.5 to Gitbook. It is available over here https://docs.feast.dev/introduction/roadmap

The roadmap as committed above is already out of date, and this is part of the reason why I am creating this issue. We are planning to increase the scope of the 0.5 release. The original Feast 0.5 release was mostly focused on

• Batch and streaming feature statistics and validation (Add Feature Validation to Feast #172)
• User authentication/authorization (Authentication and authorization #504)

The release was scheduled for early March

However, we are planning to include some pretty significant changes to the Feast code base

• We want to include the following operational improvements
  • Support for Redis Clusters (Add Support for Redis Cluster #502)
  • Support for handling out of order ingestion (Filter out of order feature rows #273 )
  • Support for compression or storage of features without field names to reduce the memory footprint in Redis (Optimize Redis memory foot print for ingestion / serving #515)
• Remove feature versions and enable automatic data migration. This is a big one. It would be a massive QOL improvement for our users. They wouldnt have to worry about feature versions any more, and any changes they make to a feature set would not drop their existing data. However, this requires a significant amount of development (Feature set versioning #386, Remove feature set versions from Feast serving API #462)
• Given the above change, we would need to refactor the existing storage code to be more modular. This modularity would come in the form of a storage module that is more cleanly separated from the existing codebase. This would make adding new stores possible. (Factor out storage key generation logic, to eliminate code duplication #402)
• Refactoring the current configuration files for both Feast Serving and Feast Core. (Clean up Feast configuration #525)
• Inclusion of a basic job management API. This would allow us to query/stop/restart jobs that population jobs that are currently being managed by Feast Core. (Add job management APIs #302)

The above won't be the only changes we propose, but they are the most significant ones. Given the scope of work, we don't expect 0.5 will land in March. Best estimate is mid-April.

The good news is that a lot of the functionality our users would have been waiting for in Feast 0.6 will now land in Feast 0.5.

Comments welcome!