arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.23503

HBFSim: A Simulator for Exploring HBF in Memory Hierarchies for LLMs

Jinheng Li·Guosheng Wang·Hongchao Du·Shangyu Wu·Qiao Li·Chun Jason Xue
View original source ↗Read source document ↗Export metadata ↓

Abstract

High Bandwidth Flash (HBF) has recently emerged as a promising technology that provides terabyte-scale accelerator-local capacity at bandwidths approaching HBM, with a potential cost-per-bit advantage over HBM. However, its NAND foundation brings long latency, out-of-place updates, and finite write endurance directly into the accelerator memory hierarchy, making its effects on performance and lifetime difficult to predict. To study this emerging design space, we present HBFSim, an LLM-aware event-driven simulator for heterogeneous HBM+HBF memory systems that preserves application-level memory semantics while modeling flash-internal behavior. HBFSim models HBM, HBF with host-managed flash translation, and conventional capacity tiers in one simulation driven by a calibrated serving frontend. Using HBFSim, we study decoding of Llama-3.1-70B and Qwen3-235B-A22B under three workloads that differ in their input and output lengths. HBF serves data that does not fit in HBM 11.0--19.0$\times$ faster than host DRAM; HBM is better used to hold the running sequences' KV cache than as a cache for HBF; and at a published peak load of Llama-3.1-70B, HBF lasts 2.0 to 6.9 years, in proportion to the stacks that share the KV writes. The artifact is available at https://github.com/FlashLabAE/HBFSim and will track public HBF device data as it becomes available.

Explore related subjects

cs.ARcs.ET

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Jinheng Li, Guosheng Wang, Hongchao Du, Shangyu Wu, Qiao Li, Chun Jason Xue. 2026-10-16. HBFSim: A Simulator for Exploring HBF in Memory Hierarchies for LLMs. https://doi.org/10.1145/3848591.3856870

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

DaPPA: A Unified Data-Parallel Programming Framework for Efficient Processing-in-Memory Execution

Processing-in-memory (PIM) alleviates data-movement bottlenecks by placing computation near or inside memory. Two general-purpose PIM paradigms are now practical: processing-near-memory (PNM) and processing-using-memory (PUM). Although both target regular, data-intensive, data-parallel workloads, they remain difficult to program because data organization is tightly coupled to computation and differs across substrates. Thus, programmers must manually manage substrate-specific data partitioning, movement, layout, and tiling, preventing program portability across PIM systems. We propose DaPPA (data-parallel processing-in-memory architecture), a programming framework that exposes a single high-level data-parallel pattern interface and retargets it to multiple PIM substrates. The key idea of DaPPA is to decouple the expression of data parallelism from substrate-specific data organization: programmers express computation once using data-parallel patterns and a dataflow interface, while specialized backends automatically generate data orchestration for PNM and multi-level tiling for PUM. We evaluate DaPPA on representative substrates from both PIM paradigms. On the UPMEM PNM system, DaPPA outperforms hand-tuned implementations by 1.6x on average with 95% fewer lines-of-code, and outperforms the state-of-the-art SimplePIM framework by 1.4x on average with 62% fewer lines-of-code. On a SIMDRAM-like PUM substrate, DaPPA outperforms the state-of-the-art MIMDRAM compiler by 2.0x on average (up to 6.4x) across 12 workloads. DaPPA can deploy all ten of our PNM workloads, whereas CINM, the closest framework targeting both paradigms, can deploy only three, on which DaPPA is 2.7--6.8x faster. We open-source DaPPA at https://github.com/CMU-SAFARI/DaPPA.

cs.AR↗
arXiv2026

GRAU: A Generic Runtime-Reconfigurable Activation-Requantization Unit for Quantized Neural Network Accelerators

As neural networks continue to scale, low-precision quantization is widely adopted in edge accelerators, but conventional multi-threshold activation hardware requires 2^n-1 thresholds for n-bit integer outputs, leading to rapidly increasing hardware costs. We propose GRAU, a Generic Runtime-Reconfigurable Activation-Requantization Unit that fuses batch normalization, nonlinear activation, and requantization through piecewise-linear fitting with Power-of-Two slopes. GRAU requires only basic comparators and 1-bit right shifters, while supporting mixed-precision quantization and runtime switching among post-accumulation functions, including modern nonlinear activations such as SiLU and GELU. Compared with multi-threshold activation-requantization units, GRAU reduces LUT consumption by approximately 90% or more while improving hardware efficiency, flexibility, and scalability. We further explore the trade-off between accuracy and hardware cost across different approximation settings and provide an open-source toolchain for GRAU-aware network training.

cs.AR↗
arXiv2026

Exposing Embedded Matrix Multiplication with polyhedral transformations for CGRA compilation

Matrix-matrix multiplication (mmul) is a dominant computing pattern in edge workloads, but in real applications it is often embedded within larger affine programs and intertwined with other operations. Coarse-Grained Reconfigurable Arrays (CGRAs) can flexibly and efficiently support it, since they combine operation-level reconfigurability and high energy efficiency. However, mapping computational kernels that include mmul with state-of-the-art compilation strategies often leads to suboptimal results, since its multi-dimensional structure hampers the uncovering of its inherent parallelism and, ultimately, runtime performance. This paper presents Kernel-CGRA, a compiler flow that uses polyhedral transformations to expose legal affine mmul subspaces, lower each accepted region to a back-end-independent operation, and preserve the surrounding computation in the generic CDFG path. For CGRA execution, the operation is instantiated as a parametric output-stationary kernel, while residual regions are reconnected through context generation and live-value preservation. The matcher is intentionally strict: only regions proven to match the supported affine mmul form are rewritten. Across canonical, fused, and application-level benchmarks, Kernel-CGRA achieves runtime speedups of up to 10.2x over state-of-the-art CGRA CDFG compilation with modulo scheduling and loop unrolling.

cs.AR↗