Repository navigation
Get_historical_features() Does Not Have Option To Return Distributed Dataframe Like A Spark DF #2504
Description
When doing get_historical_features().to_df on a large training dataset in databricks I am hitting memory full errors. Since to_df is returning the data as a pandas dataframe it is not able to use the full capacity of the databricks cluster and distribute it across the different nodes the way a spark dataframe might.
Activity
adchia commentedon Apr 12, 2022
I believe this is actually possible. See https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py#L252
adamluba1 commentedon Apr 12, 2022
@adchia but this is only available if you are using a spark offline store correct? for example, you cannot query a snowflake table and return it as a spark df, correct?
adchia commentedon May 1, 2022
Ah correct. Right now it only allows returning it in arrow batches. Is the intention to be able to materialize through a spark df? Or to do training e.g. with MLLib.
In either case, might make sense for the SparkOfflineStore (this naming doesn't really make too much sense...) to be able to query from data warehouses (but maybe run the join within the DWH)
adamluba1 commentedon May 3, 2022
@adchia The intention is to just return snowflake data offline as a training dataset with the results being a distributed spark df. Is it possible to use snowflake as the source with the SparkOfflineStore? I found a way to hack around it by manipulating the to_sql() string and issuing that as a spark query using snowflakes spark connector, but wondering if there is a cleaner way
adchia commentedon May 5, 2022
That works as a start and I don't think it's that hacky. Would be happy to check that in.
An alternative would be to flush the output of Snowflakes query to S3 and read that, but that also seems pretty hacky
feat: Get Snowflake Query Output As Pyspark Dataframe (#2504) (#3358)
chore(release): release 0.27.0