After this post from Matt Martin, I was curious if his findings would hold for Parquet files. I ran two reproducible, Docker-based experiments, using identical resources and the same 1,000-file Parquet dataset, with correctness checks to ensure both engines produced matching results. In the analytical query test, DuckDB’s median processing time was 3.1 seconds versus Spark’s 17.1 seconds, making DuckDB 5.52× faster. In the compaction test, which read, sorted, and merged the source files into a new compressed Parquet file, DuckDB finished in a median of 9.5 seconds versus Spark’s 60.4 seconds, a 6.35× advantage.

https://www.linkedin.com/feed/update/urn:li:activity:7494050617556987905/

https://github.com/sergiorgiraldo/tst1/tree/master/duck