arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.07094

Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads

Abbas Raza Ali·Muhammad Ajmal Siddiqui·Moona Zahid
View original source ↗Read source document ↗Export metadata ↓

Abstract

LLM deployment is shifting from single-turn completion to agentic trajectories in which a model plans, calls tools, reads results and reasons at test time before acting. This inverts the economics of inference hardware: chat serving amortises weight reads across large batches, whereas agent trajectories are sequentially dependent, run at effective batch one, and make per-token decode latency (TPOT) the dominant term in task completion time. Using a roofline analysis and a closed-form episode-latency model, we show why this regime favours accelerators that keep weights in on-die SRAM (Cerebras WSE-3/3T, Groq/NVIDIA LPU) or compiler-managed tiered memory (SambaNova SN40L/SN50), and why three vendor ecosystems converged in 2026 on disaggregated prefill/decode serving. We show that per-step reliability compounds exponentially in trajectory length-a 2% per-step failure rate erases a 2x decode advantage for a 20-step agent-so determinism and tail latency are first-order performance variables. We then propose a four-layer evaluation framework (silicon, serving system, agent episode, enterprise) with a metric set built on goodput at an agentic SLO and cost per successful episode, a six-axis benchmark protocol over six task families, a paired-bootstrap statistical design, an attestation protocol for vendor-run benchmarks, and TCO, availability and adoption-timing models with explicit break-even conditions. All performance figures are public and labelled by evidence class; we state seven falsifiable hypotheses and the experiments that test them, and argue that the most likely original result is that token-throughput rankings diverge from cost-per-successful-task rankings on long-horizon work.

Explore related subjects

cs.ARcs.DCcs.LGcs.PF

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid. 2026-10-05. Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads. https://arxiv.org/abs/2610.07094

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Breaking Memory Bottlenecks in Quantum Control Systems for More Precise Experiments and Higher Throughput Computing

As quantum computing matures, there is an increasing need for deeper and longer experiments to characterize quantum devices precisely and higher circuit throughput to validate more domain applications. However, this need is hindered by a memory bottleneck at the quantum control system layer, arising from the limited capacity of on-chip block random-access memory (BRAM) and the non-deterministic latency of dynamic random-access memory (DRAM). To break this bottleneck, we present Ant-Q, a memory hierarchy design for the QubiC 2.0 control system that integrates DRAM with BRAM to support pipelined quantum circuit execution while keeping inter-circuit timing deterministic whenever each circuit outlasts the refill of the next. A switch between command buffers then takes a constant 134 ns and, inside a circuit, acts on the qubit as a 144 ns idle. On the bench, Ant-Q runs deep one- and two-qubit randomized benchmarking beyond the command capacity of the baseline. For circuits with at least 1000 shots, it reduces the time the board spends beyond the circuit execution from 18.91%-104.54% to at most 0.53% of the execution time, and it improves the board-level throughput of 30-circuit batches by 51.06% against a baseline with the same shared tables (59.90% against the stock baseline). On a transmon qubit, Ant-Q executes a 5101-Clifford sequence and acquires uninterrupted 100 s Ramsey records with 0.01 Hz frequency spacing. In simulations, these records detect 1/f frequency noise of 1.37 and 1.62 times lower spectral density than the same records split into the 2.048 s runs of the baseline. The measured spectra resolve no 1/f component and match the measurement floor from 0.1 to 100 Hz. Ant-Q has been merged into the open-source QubiC gateware and software.

cs.AR↗
arXiv2026

CacheFlex: Direct Software-Managed Access to Higher-Level Cache for Scalable Vector Support

Wider vector/SIMD lanes multiply compute throughput, but cache improvements remain challenging. We identify L1 data cache capacity as the most impactful parameter for performance and efficiency. L1 capacity is constrained by pipeline coupling and latency requirements; private L2 offers an order of magnitude more capacity but is inaccessible from software. This work proposes CacheFlex, an ISA and microarchitecture co-design that enables software-managed, direct access to private L2 SRAM by allowing runtime conversion of parts of L2 into a scratchpad memory (SPM). Through a lightweight ISA extension, software explicitly controls data placement and lifetime within the L2. CacheFlex integrates into an out-of-order pipeline with cross-domain memory consistency and OS context-switch support, with merely 2.02% area overhead relative to the L2 in a post-layout 12 nm implementation. We prototype CacheFlex based on the Armv8-A SVE ISA, model it in gem5, and evaluate it using optimized general matrix multiplication (GEMM) kernels with diverse model-derived shapes, FlashAttention-based fused attention, and full-layer workloads from LLaMA-3.2-1B and BERT-Base in the prefill phase. As the vector length (VL) scales from VL = 4 (512-bit) to VL = 16 (2048-bit), CacheFlex's kernel-level geometric-mean speedup and memory-subsystem energy reduction grow from 1.12x and 26% to 1.44x and 40%, respectively, bridging the widening gap between per-instruction data demand and memory performance. On full transformer layers combining CacheFlex with FlashAttention, the design delivers up to 1.78x speedup and 45% memory-subsystem energy savings (BERT-Base, sequence length T=4096, VL = 16; LLaMA-3.2-1B: 1.64x and 38%). CacheFlex is complementary to hardware prefetching, and its benefit grows as vector width and peak compute throughput increase for compute-intensive AI kernels.

cs.AR↗
arXiv2026

TPCS: A Cycle-Level Simulation Paradigm with Native Execution Semantics for RTL Modeling and Generation

The growing diversity of domain-specific architectures and microarchitectural designs demands fast and accurate evaluation methods during early design stages. However, existing approaches suffer from a fundamental trade-off: high-level simulators offer speed but lack fidelity, while RTL-based tools provide accuracy at the cost of significant simulation overhead. This gap limits efficient pre-RTL microarchitecture exploration. Therefore, we propose Transaction-Pruned Cycle Simulation (TPCS), a novel modeling and simulation framework. It serves as a unified intermediate abstraction spanning microarchitecture modeling, cycle-accurate simulation, and synthesizable RTL implementation. TPCS organizes combinational logic as transaction-based function invocations and leverages host CPU control flow to naturally prune inactive logic, eliminating the need for explicit event scheduling or signal tracking. This enables cycle-accurate simulation with significantly improved performance. TPCS preserves structural correspondence with RTL and supports automatic generation of synthesizable designs. TPCS achieves 36$\times$-42$\times$ speedups over parallel Verilator on SMP systems and 3.4$\times$-7.2$\times$ over single-threaded Verilator on accelerators, with cycle-level waveform agreement on the evaluated traces. The generated RTL demonstrates comparable timing and area to handcrafted implementations, with relative changes of -4.7\% to +2.6\% in clock period and -3.3\% to +4.0\% in standard-cell area. By bridging the abstraction gap between high-level modeling and RTL design, TPCS enables a unified and agile pre-RTL workflow, allowing efficient microarchitecture exploration with both high performance and implementation fidelity.

cs.AR↗