Measuring Format Effects on Scan Efficiency: A Minimal, Reproducible Methodology
Keywords:
Parquet, ORC, CSV, NDJSON, predicate pushdown, scan efficiency, columnar storage, DuckDB, reproducibilityAbstract
File and record formats are often treated as interchangeable substrates for analytics, yet query cost is strongly shaped by format-level mechanisms such as predicate pushdown, column projection, statistics, and parse overhead. This article presents a compact, reproducible methodology for measuring how format choice influences scan efficiency and tail latency on a single node. Using one dataset in multiple encodings (Parquet [3], ORC [8], CSV, NDJSON [5]), the method varies selectivity, column width, and stripe/row-group size and compression, reporting P50/P95 latency, bytes scanned, and row-group skip rate while detecting pushdown hits and misses. The contribution is a format-centric measurement recipe that others can re-run, extend, and compare across environments, with public artifacts [9] enabling full reproduction. In the refreshed runs (TPC-H SF1, 15 repetitions), sorted Parquet with 8 MB row-groups and ZSTD scanned ~23 MB at ≈1% selectivity versus 0.77 GB (CSV) and 2.35 GB (NDJSON), with P95 latency of 2–3 ms versus 226–239 ms; skip rate ≈94%. ORC (sorted, 8 MB stripes, ZSTD) scanned approximately 0.28 GB - fewer bytes than CSV — yet produced a P95 latency of approximately 850 ms, exceeding all other formats, including CSV. This result isolates a critical ecosystem constraint: columnar byte savings are only realized as latency savings when the analytical engine has a native scanner with predicate pushdown support.
Downloads
References
M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, and M. Ferreira, “C-Store: A column-oriented DBMS,” in Proc. VLDB, 2005, pp. 553–564.
S. Melnik, A. Gubarev, J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis, “Dremel: Interactive analysis of web-scale datasets,” in Proc. VLDB, 2010, pp. 330–339.
Apache Software Foundation, “Apache Parquet file format,” 2024. [Online]. Available: https://parquet.apache.org/docs/file-format/
Apache Software Foundation, “Apache Avro schema resolution,” 2024. [Online]. Available: https://avro.apache.org/docs/current/specification/
JSON Lines, “NDJSON format specification,” 2024. [Online]. Available: https://jsonlines.org/
J. Jain, A. Deshpande, and A. Gupta, “Analyzing and comparing lakehouse storage systems,” in Proc. CIDR, 2023.
H. Pirk, M. Rigger, and M. Kersten, “DuckDB: An analytical database system in-process,” in Proc. VLDB (systems overview), 2017.
Apache Software Foundation, “Apache ORC format specification,” 2024. [Online]. Available: https://orc.apache.org/docs/
A. Joshi, “FormatBench (companion repository),” 2024. [Online]. Available: https://github.com/ankitjoshi14/FormatBench
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
