A Goodput-Centric Benchmark for Large Language Model Inference under Realistic Online Load

Authors

  • Veera Ravindra Divi

Keywords:

large language models, inference serving, systems for machine learning, throughput, tail latency, quantization, benchmark, goodput

Abstract

Peak throughput is the number most public reports use to compare large language model (LLM) serving stacks. It is also the wrong number: it is measured offline, at saturation, with an unbounded queue, while production serving is an online problem governed by tail latency under bursty load. There, time-to-first-token (TTFT) and inter-token latency (TPOT) bound interactivity, and what matters is goodput—the request rate served within a latency service-level objective (SLO). We present InferBench, an open benchmark and harness for LLM inference serving that (i) defines a realistic multi-workload request distribution, (ii) drives systems with an open-loop, Poisson/bursty load generator, and (iii) reports a standardized metric suite—throughput, TTFT, TPOT, and SLO-bounded goodput—plus a quantization track for the accuracy/efficiency trade-off. Evaluating six 2023-era serving systems on a 13B model, InferBench shows that continuous-batching, paged-KV systems sustain up to  the goodput of a static-batching baseline at equal latency SLO, and that 4-bit weight quantization buys up to  throughput at a sub-point accuracy cost. We release the harness, the workloads, and all scripts as a reproducible baseline for serving-system evaluation

Downloads

Download data is not yet available.

References

Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017.

T. B. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems (NeurIPS), 2020.

H. Touvron, L. Martin, K. Stone et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-LM: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019.

R. Y. Aminabadi, S. Rajbhandari, A. A. Awan et al., “DeepSpeed-Inference: Enabling efficient inference of transformer models at unprecedented scale,” in International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2022.

T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.

T. Dao, “FlashAttention-2: Faster attention with better parallelism and work partitioning,” arXiv preprint arXiv:2307.08691, 2023.

G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022.

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023.

V. J. Reddi, C. Cheng, D. Kanter et al., “MLPerf inference benchmark,” in ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020.

D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A low-latency online prediction serving system,” in 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017.

Gujarati, R. Karimi, S. Alzayat et al., “Serving DNNs like clockwork: Performance predictability from the bottom up,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2020.

D. Narayanan, M. Shoeybi, J. Casper et al., “Efficient large-scale language model training on GPU clusters using Megatron-LM,” in International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2021.

Y. Sheng, L. Zheng, B. Yuan et al., “FlexGen: High-throughput generative inference of large language models with a single GPU,” in International Conference on Machine Learning (ICML), 2023.

Hugging Face, “Text generation inference,” 2023, open-source LLM serving toolkit.

NVIDIA, “FasterTransformer,” 2021, optimized transformer inference library.

R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” in Proceedings of Machine Learning and Systems (MLSys), 2023.

N. Shazeer, “Fast transformer decoding: One write-head is all you need,” 2019.

J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” in Proceedings of EMNLP, 2023.

Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, “SARATHI: Efficient LLM inference by piggybacking decodes with chunked prefills,” arXiv preprint arXiv:2308.16369, 2023.

T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.

G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning (ICML), 2023.

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training quantization for generative pre-trained transformers,” in International Conference on Learning Representations (ICLR), 2023.

J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, “AWQ: Activation-aware weight quantization for LLM compression and acceleration,” arXiv preprint arXiv:2306.00978, 2023.

Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” arXiv preprint arXiv:2103.13630, 2021.

P. Liang, R. Bommasani, T. Lee et al., “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110, 2022.

T. Wolf, L. Debut, V. Sanh et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of EMNLP: System Demonstrations, 2020.

Downloads

Published

18.04.2024

How to Cite

Veera Ravindra Divi. (2024). A Goodput-Centric Benchmark for Large Language Model Inference under Realistic Online Load. International Journal of Intelligent Systems and Applications in Engineering, 12(21s), 5274–5280. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/8406

Issue

Vol. 12 No. 21s (2024)

Section

Research Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.

IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.