Repository navigation
Spark Materialization Engine Cannot Infer Schema #5594
Description
Expected Behavior
Spark engine when enabled should materialize features to the online store.
Current Behavior
Materialization fails with the following error:
pyspark.errors.exceptions.base.PySparkTypeError: [CANNOT_INFER_SCHEMA_FOR_TYPE] Can not infer schema for type: `ChunkedArray`.
Steps to reproduce
See this repo that should reproduce the error locally for you. Ensure your JAVA_HOME is pointing to a java 17 installation and run make run to run the workflow. Optionally inspect the Makefile and run the commands yourself.
The repo has the default feast init repo created and will run a feast plan, feast apply and finally a materialize.py script which should trigger a Spark materialization job.
Specifications
- Version: 0.53.0
- Platform: Python 3.12, PySpark 3.5.5, Java 17
- Subsystem: MacOS 15.6.1
Possible Solution
I believe this happens when an arrow dataframe is converted to a Spark dataframe. Why this happens I am unsure, it may be my configuration or it may be a bug. If someone could advise on a solution that would be great.
Thanks!
Reacted by Dan Baron
Activity
danbaron63 commentedon Sep 3, 2025
Note: I've tested this with
pyspark==4.0.0and it works, I believe this is due to changes in the way Spark now natively uses arrow. Unfortunately we're still blocked though as Spark 4 isn't widely available on most data compute platforms (e.g. AWS EMR) yet.I believe converting the pyarrow table to a pandas DataFrame as an intermediary should resolve this which shouldn't be too memory intensive if
spark.sql.execution.arrow.enabledis set totruehaoxu0 commentedon Sep 3, 2025
That make sense. Feel free to submit a pr
jfw-ppi commentedon Dec 29, 2025
I will look into this as it blocks my solution for #5671
fix: build Spark DataFrame from Arrow with schema and empty handling(f…
fix: build Spark DataFrame from Arrow with schema and empty handling(f…
jfw-ppi commentedon Jan 1, 2026
The reproduction repository passes now, using your suggested fix @danbaron63
fix: build Spark DataFrame from Arrow with schema and empty handling(f…
fix: build Spark DataFrame from Arrow with schema and empty handling(f…
fix: build Spark DataFrame from Arrow with schema and empty handling(f…
fix: build Spark DataFrame from Arrow with schema and empty handling(f…
fix: build Spark DataFrame from Arrow with schema and empty handling(f…
fix: Spark Materialization Engine Cannot Infer Schema (#5806)
chore(release): release 0.59.0
chore(release): release 0.59.0
chore(release): release 0.59.0
chore(release): release 0.59.0
fix: Spark Materialization Engine Cannot Infer Schema (feast-dev#5806)
chore(release): release 0.59.0