{"id":491,"date":"2026-08-16T12:06:19","date_gmt":"2026-08-16T10:06:19","guid":{"rendered":"https:\/\/www.giral.do\/wp-en\/sem-categoria\/duckdb-and-spark-parquet\/"},"modified":"2026-08-16T12:06:19","modified_gmt":"2026-08-16T10:06:19","slug":"duckdb-and-spark-parquet","status":"publish","type":"post","link":"http:\/\/www.giral.do\/wp-en\/engineering\/duckdb-and-spark-parquet\/","title":{"rendered":"DuckDB and Spark &#8211; Parquet"},"content":{"rendered":"<p>After this post from Matt Martin, I was curious if his findings would hold for Parquet files. I ran two reproducible, Docker-based experiments, using identical resources and the same 1,000-file Parquet dataset, with correctness checks to ensure both engines produced matching results. In the analytical query test, DuckDB\u2019s median processing time was 3.1 seconds versus Spark\u2019s 17.1 seconds, making DuckDB 5.52\u00d7 faster. In the compaction test, which read, sorted, and merged the source files into a new compressed Parquet file, DuckDB finished in a median of 9.5 seconds versus Spark\u2019s 60.4 seconds, a 6.35\u00d7 advantage. <\/p>\n<p>https:\/\/www.linkedin.com\/feed\/update\/urn:li:activity:7494050617556987905\/<\/p>\n<p>https:\/\/github.com\/sergiorgiraldo\/tst1\/tree\/master\/duck<\/p>\n","protected":false},"excerpt":{"rendered":"<p>After this post from Matt Martin, I was curious if his findings would hold for Parquet files. I ran two reproducible, Docker-based experiments, using identical resources and the same 1,000-file Parquet dataset, with correctness checks to ensure both engines produced &hellip; <a href=\"http:\/\/www.giral.do\/wp-en\/engineering\/duckdb-and-spark-parquet\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32,31,33],"tags":[],"class_list":["post-491","post","type-post","status-publish","format-standard","hentry","category-duckdb","category-engineering","category-spark"],"_links":{"self":[{"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/posts\/491","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/comments?post=491"}],"version-history":[{"count":0,"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/posts\/491\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/media?parent=491"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/categories?post=491"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.giral.do\/wp-en\/wp-json\/wp\/v2\/tags?post=491"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}