Repository navigation
Source data model in Feast Core should be generalized #632
Description
Expected Behavior
Sources are generic and extensible.
Current Behavior
Currently the sources data model in Feast Core is hardcoded specifically to KafkaSource.
Possible Solution
Make sure none of the columns in our data model is specific to a single source. The pattern we have followed thus far is to use Map<String,String> for Source, Runner, Store configurations. One way to solve this problem is to have source configuration stored as a serialized JSON map in a single column in the Source table.
Activity
[-]Sources should be generic[/-][+]Sources should be generalized[/+]ches commentedon Apr 18, 2020
While I get the utility and convenience especially for deploying Feast with more distributed ownership in the organization, we ignore / work around a lot of the Source and Store configuration kept in the registry RDBMS, in favor of service discovery.
I also think that asking data scientists and data engineers to be concerned with operational infra configuration like Kafka broker addresses when registering feature sets is not an elegant separation of concerns.
Maybe we can take this as an opportunity to consider design alternatives as well.
[-]Sources should be generalized[/-][+]Source data model in Feast Core should be generalized[/+]woop commentedon Apr 19, 2020
Hi @ches, would love to continue the discussion. I've created an issue specifically for the topic over here: #633
That allows this card to stay on track separately since I don't see us making fundamental changes to sources in 0.5