arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.16127

Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry

Le Qiu·Zhuan Anne Zhang·Hok Ming Adeline Chan·Emmanuele Chersoni
View original source ↗Read source document ↗Export metadata ↓

Abstract

Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regulatory compliance remains uncertain. In this study, we evaluate the performance of LLMs on Hong Kong fire safety codes. In collaboration with domain experts, we create a localized benchmark dataset comprising both multiple-choice questions (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used proprietary and open-source models. Our findings show that top-performing proprietary models achieve 80-90% accuracy on MCQs, whereas open-source models generally lag behind. However, on open-ended evaluation, expert human assessments reveal an accuracy range of only 50-70%, demonstrating that LLMs still have substantial room for improvement in specialized compliance tasks.

Explore related subjects

cs.CEcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Le Qiu, Zhuan Anne Zhang, Hok Ming Adeline Chan, Emmanuele Chersoni. 2026-10-12. Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry. https://arxiv.org/abs/2610.16127

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A bifidelity sparse-grid surrogate model driven by a discrepancy indicator

Sparse-grid approximations based on Lagrange polynomials are one of the staple methods for approximating functions that are high-dimensional and expensive to evaluate, e.g. in the context of PDE-based parametric design exploration. They are, however, known to have limited granularity: refining these sparse-grid approximations may require multiple additional function evaluations, potentially leading to computational inefficiency. To address this issue, we propose a selection criterion for the candidate collocation points introduced by the refinement procedure: the function is evaluated only at those points for which a suitable (and crucially zero-cost) discrepancy indicator suggests that the additional evaluation would significantly change the resulting approximation. The remaining collocation points are instead assigned values predicted by the already available sparse-grid approximation, i.e., following a bifidelity approach that reduces costs while preserving accuracy. The effectiveness of this methodology is demonstrated on several benchmark functions and on an engineering application concerning flashback phenomena in hydrogen-fueled perforated burners.

cs.CE↗
arXiv2026

FinTradeBench: A Financial Reasoning Benchmark for LLMs

Financial analysis requires reasoning over heterogeneous signals, including company fundamentals from regulatory filings and trading signals derived from market prices. However, existing financial benchmarks focus almost exclusively on regulatory filings, neglecting market dynamics and cross-signal interactions. We introduce FinTradeBench, a benchmark of 1,400 questions covering NASDAQ-100 stocks over a ten-year window. To holistically evaluate financial reasoning, FinTradeBench includes questions in three reasoning categories: fundamentals-focused, trading-signal-focused, and hybrid questions that require cross-signal reasoning. Our benchmark starts with 150 expert-authored seed questions, which are then scaled to 1400. To ensure reliable evaluation, we create a Core400 subset in which every item is expert-reviewed. In zero-shot (No-RAG) response generation, we observe that all models perform substantially worse on trading-focused questions compared to fundamental-focused questions. To identify failure causes, we evaluate LLMs under controlled settings including retrieval-based baselines, tool-assisted computation, oracle conditions, and time-series-specialized pipelines. LLMs' capabilities to do computation and evidence selection contribute most to reasoning failures. Additionally, retrieval significantly improves trading-signal reasoning but does not improve fundamental reasoning, where identifying relevant signals remains the dominant failure mode. Finally, rather than relying on a small, fine-tuned model for time-series reasoning, it is better to summarize the time-series data and leverage LLMs' reasoning capabilities.

cs.CE↗
arXiv2026

Non-linear mechanical field reconstruction coupling recurrent neural networks with physics-informed graph neural networks

Reconstructing local stress fields in heterogeneous microstructures under non-linear, history-dependent loading remains a major computational bottleneck in multi-scale simulations. We propose a coupled LSTM-GNN framework that links the temporal and spatial aspects of local stress field reconstruction. A Long Short-Term Memory network encodes macroscopic stress-strain sequences into a compact hidden state that captures the path-dependent constitutive response, while a physics-informed Graph Neural Network reconstructs the spatially-resolved stress field at each time step. We introduce a relative weighting strategy with linear warm-up to balance the data-driven reconstruction loss and a discrete divergence-based equilibrium penalty. This resolves the scale mismatch that prevents fixed-weight formulations from converging in the elasto-plastic regime. The model is trained on 10,000 non-proportional loading paths applied to a periodic plate-with-a-hole microstructure and von Mises elasto-plasticity. The model achieves three orders of magnitude speedup over finite element simulations and generalizes to loading sequences twice the training length, with 1.9% cumulative error. Because the graph relies on mesh connectivity instead of the specific element type, one trained surrogate can be applied directly without retraining to meshes with different element types and to both coarser and finer resolutions, while in all cases reproducing the high-fidelity quad-element FE field used during training. Indeed, the message passing characteristics inherent to GNN and MeshGraphNet architecture render the model mesh-agnostic. Analysis of the LSTM hidden states suggests a low-dimensional structure related to the internal state variables of the constitutive model.

cs.CE↗