Vortex Benchmarks

Compression17 chartsencode 2.0× · decode 5.1× faster than Parquet

Write Throughput

#1vortex2.04x1992 MB/s

#2parquet1.00x1134 MB/s

#3lance0.73x761 MB/s

Scan Throughput

#1vortex5.15x51.23 GB/s

#2parquet1.00x9.90 GB/s

#3lance0.95x8.27 GB/s

Geomean throughput ratio to Parquet | Aggregate throughput (higher is better)

Compression Size26 charts14% larger than Parquet

Vs Arrow

#1parquet6.54x

#2vortex5.75x

#3lance3.09x

#4arrow-ipc1.1x

Vs ParquetMinMeanMax

Geometric means of compressed sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)

String Encoding21 charts

Write Throughput

#1fsst1.00x222 MB/s

#2onpair-120.64x153 MB/s

#3onpair-160.63x149 MB/s

Scan Throughput

#1onpair-161.23x4.60 GB/s

#2onpair-121.21x4.61 GB/s

#3fsst1.00x3.72 GB/s

Geomean throughput ratio to FSST | Aggregate throughput (higher is better)

String Encoding Size21 charts

Vs Arrow

#1onpair-162.76x

#2onpair-122.76x

#3fsst2.14x

Vs FSSTMinMeanMax

Geometric means of file sizes versus Arrow (higher is better) and versus FSST (lower is better)

Random Access (NVMe)9 chartshot 102× · cold 32× faster than Parquet

Hot

#1vortex1.00x780.98 us

#2litdata2.40x1.72 ms

#3lance3.08x3.07 ms

#4arrow-ipc4.81x10.49 ms

#5parquet102.39x79.76 ms

Cold

#1vortex1.17x2.67 ms

#2lance1.90x4.56 ms

#3arrow-ipc1.94x10.69 ms

#4litdata20.37x43.73 ms

#5parquet38.09x79.22 ms

Geomean of take time ratio to fastest across every dataset (lower is better)

Random Access (S3)9 chartshot 85× · cold 2.4× faster than Parquet

Hot

#1vortex1.00x20.85 ms

#2lance29.67x173.44 ms

#3parquet84.92x471.47 ms

Cold

#1vortex1.00x266.79 ms

#2lance2.22x582.68 ms

#3parquet2.45x636.39 ms

Geomean of take time ratio to fastest across every dataset (lower is better)

Clickbench43 chartsDuckDB 1.8× · DataFusion 1.6× faster than Parquet

Performance Summary

#1duckdb:vortex1.10x3.66 s

#2datafusion:vortex1.26x3.93 s

#3duckdb:vortex-compact1.31x4.22 s

#4datafusion:vortex-compact1.56x4.94 s

#5datafusion:parquet1.96x5.91 s

#6duckdb:duckdb1.98x5.37 s

#7duckdb:parquet2.02x5.65 s

#8datafusion:lance16.89x47.68 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=1)22 chartsDuckDB 3.7× · DataFusion 3.9× faster than Parquet

Performance Summary

#1duckdb:duckdb1.11x335.77 ms

#2duckdb:vortex1.28x368.82 ms

#3duckdb:vortex-compact1.41x402.76 ms

#4datafusion:vortex1.71x525.48 ms

#5datafusion:vortex-compact1.80x543.17 ms

#6datafusion:arrow3.29x1.08 s

#7duckdb:parquet4.69x1.39 s

#8datafusion:lance6.20x2.01 s

#9datafusion:parquet6.61x1.90 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=1)22 chartsDuckDB 1.1× faster · DataFusion 1.1× slower vs Parquet

Performance Summary

#1datafusion:parquet1.17x5.59 s

#2datafusion:vortex-compact1.18x5.66 s

#3datafusion:vortex1.29x6.20 s

#4duckdb:vortex-compact1.48x6.98 s

#5datafusion:lance1.54x7.69 s

#6duckdb:vortex1.59x7.57 s

#7duckdb:parquet1.67x7.97 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=10)22 chartsDuckDB 2.5× · DataFusion 2.3× faster than Parquet

Performance Summary

#1duckdb:duckdb1.14x1.02 s

#2duckdb:vortex1.31x1.18 s

#3duckdb:vortex-compact1.48x1.30 s

#4datafusion:vortex1.79x1.82 s

#5datafusion:vortex-compact1.98x1.93 s

#6duckdb:parquet3.22x2.90 s

#7datafusion:parquet4.18x3.66 s

#8datafusion:arrow14.51x16.39 s

#9datafusion:lance20.96x22.43 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=10)22 chartsDuckDB 1.5× slower · DataFusion ≈ same vs Parquet

Performance Summary

#1datafusion:vortex-compact1.15x11.29 s

#2datafusion:parquet1.24x12.25 s

#3datafusion:vortex1.28x12.55 s

#4duckdb:parquet1.40x13.35 s

#5duckdb:vortex-compact2.04x20.07 s

#6duckdb:vortex2.12x20.91 s

#7datafusion:lance3.75x43.02 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=100)22 chartsDuckDB 1.7× · DataFusion 1.8× faster than Parquet

Performance Summary

#1duckdb:vortex1.13x7.23 s

#2datafusion:vortex1.66x13.94 s

#3duckdb:parquet1.89x11.34 s

#4datafusion:parquet3.06x19.83 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=100)22 chartsDuckDB 4.0× slower · DataFusion ≈ same vs Parquet

Performance Summary

#1duckdb:parquet1.06x36.60 s

#2datafusion:parquet1.34x48.56 s

#3datafusion:vortex1.38x50.05 s

#4duckdb:vortex4.22x167.58 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=1)99 chartsDuckDB 1.3× · DataFusion 1.5× faster than Parquet

Performance Summary

#1duckdb:duckdb1.04x1.70 s

#2duckdb:vortex1.39x2.15 s

#3duckdb:vortex-compact1.63x2.47 s

#4duckdb:parquet1.83x2.82 s

#5datafusion:vortex2.23x4.28 s

#6datafusion:vortex-compact2.42x4.77 s

#7datafusion:parquet3.33x6.04 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=10)99 chartsDuckDB 1.6× · DataFusion 1.4× faster than Parquet

Performance Summary

#1duckdb:vortex1.09x6.26 s

#2datafusion:vortex1.53x22.92 s

#3duckdb:parquet1.78x8.84 s

#4datafusion:parquet2.08x14.05 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=100)99 chartsDuckDB ≈ same · DataFusion 1.3× faster vs Parquet

Performance Summary

#1duckdb:parquet1.20x27.08 s

#2duckdb:vortex1.21x29.39 s

#3datafusion:vortex1.78x151.64 s

#4datafusion:parquet2.37x148.10 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (S3) (SF=10)99 chartsDuckDB ≈ same · DataFusion 1.2× slower vs Parquet

Performance Summary

#1datafusion:parquet1.38x69.03 s

#2duckdb:parquet1.40x61.18 s

#3duckdb:vortex1.44x74.59 s

#4datafusion:vortex1.62x88.67 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (S3) (SF=100)99 chartsDuckDB 4.8× slower · DataFusion 1.7× faster vs Parquet

Performance Summary

#1datafusion:vortex1.25x256.91 s

#2duckdb:parquet1.28x149.62 s

#3datafusion:parquet2.08x342.81 s

#4duckdb:vortex6.20x965.41 s

Geomean of query time ratio to fastest (lower is better)

Statistical and Population Genetics11 chartsDuckDB 4.2× · DataFusion 3.4× faster than Parquet

Performance Summary

#1datafusion:vortex1.14x1.50 s

#2datafusion:vortex-compact1.23x1.68 s

#3duckdb:vortex2.30x2.49 s

#4duckdb:vortex-compact2.44x2.56 s

#5datafusion:parquet3.85x3.81 s

#6duckdb:parquet9.65x6.72 s

Geomean of query time ratio to fastest (lower is better)

PolarSignals Profiling10 chartsVortex only

Performance Summary

#1datafusion:vortex1.00x375.89 ms

Geomean of query time ratio to fastest (lower is better)

fineweb [nvme]9 chartsDuckDB 5.2× · DataFusion 3.3× faster than Parquet

Performance Summary

#1duckdb:vortex1.01x210.42 ms

#2datafusion:vortex1.34x226.41 ms

#3duckdb:vortex-compact3.18x762.22 ms

#4datafusion:parquet4.44x629.72 ms

#5duckdb:parquet5.27x788.57 ms

#6datafusion:vortex-compact5.81x1.11 s

Geomean of query time ratio to fastest (lower is better)

fineweb [s3]9 chartsDuckDB 1.3× · DataFusion 2.1× faster than Parquet

Performance Summary

#1datafusion:vortex-compact1.08x3.50 s

#2datafusion:vortex1.09x3.60 s

#3duckdb:vortex-compact1.52x5.27 s

#4duckdb:vortex1.56x5.53 s

#5duckdb:parquet1.99x5.52 s

#6datafusion:parquet2.32x6.11 s

Geomean of query time ratio to fastest (lower is better)

appian [nvme]8 chartsDuckDB ≈ same · DataFusion 1.3× faster vs Parquet

Performance Summary

#1duckdb:duckdb1.13x1.62 s

#2duckdb:parquet1.24x1.67 s

#3duckdb:vortex1.30x1.80 s

#4duckdb:vortex-compact1.32x1.85 s

#5datafusion:vortex1.87x10.99 s

#6datafusion:vortex-compact1.89x11.03 s

#7datafusion:parquet2.37x11.42 s

Geomean of query time ratio to fastest (lower is better)

clickbench-sorted [nvme]10 chartsDuckDB 2.2× · DataFusion 2.7× faster than Parquet

Performance Summary

#1duckdb:vortex1.01x174.57 ms

#2duckdb:vortex-compact1.21x214.73 ms

#3datafusion:vortex1.31x319.95 ms

#4datafusion:vortex-compact1.63x375.11 ms

#5duckdb:duckdb2.07x343.66 ms

#6duckdb:parquet2.19x442.48 ms

#7datafusion:parquet3.56x1.28 s

#8datafusion:lance75.50x19.89 s

Geomean of query time ratio to fastest (lower is better)

vortex [nvme]3 chartsDuckDB 1.4× faster · DataFusion ≈ same vs Parquet

Performance Summary

#1datafusion:parquet1.18x491.18 ms

#2datafusion:vortex1.21x433.11 ms

#3duckdb:vortex1.52x547.34 ms

#4duckdb:parquet2.11x780.16 ms

Geomean of query time ratio to fastest (lower is better)