arXiv subjects

Taylan G. Topcu

Publications and source records attributed to Taylan G. Topcu.

10 recordsLinked to original sources

arXiv2026

Two Truths and A Lie? Benchmarking Off-the-Shelf LLMs for Requirements Quality Assessment: Performance, False Alarms, and Misses

Requirements engineering (RE) governs the quality of everything downstream in systems engineering (SE); defective requirements that survive review cycles propagate into design rework, schedule delays, and cost overruns. Because requirements are often written in natural language, recent advances in generative AI have raised expectations that large language models (LLMs) can absorb requirement quality assessment, a task otherwise slow and human expertise-intensive. Yet empirical evidence on whether LLMs can be trusted to do so remains scarce. This study presents the first benchmarking analysis of off-the-shelf LLM performance for requirement quality evaluation. Against an expert-derived ground truth built on INCOSE quality criteria, we evaluate ten models spanning two families (OpenAI and Anthropic) and five generations each, across one hundred independent runs, two requirement sets, and five sampling temperatures. Four contributions follow. First, we quantify a strongly asymmetric error profile: across all models and runs, the best-performing Anthropic model detects a median of only 47% of expert-identified issues while false-flagging 11%. Second, performance degrades significantly where SE judgment is required, as necessity and correctness issues are almost always missed. Third, generational progress is non-monotonic, so newer models cannot be assumed better. Fourth, this error behavior shifts only modestly and non-monotonically across sampling temperatures, indicating characteristic model deficiencies rather than inherent stochasticity. Off-the-shelf LLMs are therefore not yet trustworthy autonomous evaluators. Findings also warrant caution for Agentic AI developers: orchestrating these LLM modules in specialized architectures risks compounding these deficiencies rather than correcting them. Their defensible near-term role is human-in-the-loop decision support.

cs.SE↗

arXiv2026

Natural-Language to SysMLv2 Translation via Conformance-Driven Iterative Refinement

Model-Based Systems Engineering (MBSE) relies on formal system models as primary technical artifacts for representing requirements, structure, and behavior across the system lifecycle. With the standardization of SysMLv2 as a textual language, interest is increasing in translating natural-language descriptions directly into executable models. For practical deployment, generated models must be accepted by industrial modeling environments, not merely satisfy grammar constraints. We present a conformance-checker-driven framework for reliable natural-language-to-SysMLv2 translation that enforces production-level acceptance as the termination condition. The system embeds a SysMLv2 conformance checker within a generate-check-repair loop. Each model is evaluated using the checker, and deterministic diagnostics are incorporated into revisions until zero conformance errors are achieved. Using the production checker as the oracle ensures the framework targets deployability rather than grammar plausibility. We evaluate the approach on the full SysMBench prompt set of 151 prompts across four large language model backends, yielding 604 prompt-model cases. Single-shot generation achieves 51.16% production-conformance acceptance, while our approach achieves 100.00% conformance. By elevating production conformance from a post-processing check to a control mechanism within generation, the framework converts probabilistic outputs into production-accepted SysMLv2 artifacts suitable for loading, visualization, and engineering use.

cs.SE↗

arXiv2025

Leveraging High-Fidelity Digital Models and Reinforcement Learning for Mission Engineering: A Case Study of Aerial Firefighting Under Perfect Information

As systems engineering (SE) objectives evolve from design and operation of monolithic systems to complex System of Systems (SoS), the discipline of Mission Engineering (ME) has emerged which is increasingly being accepted as a new line of thinking for the SE community. Moreover, mission environments are uncertain, dynamic, and mission outcomes are a direct function of how the mission assets will interact with this environment. This proves static architectures brittle and calls for analytically rigorous approaches for ME. To that end, this paper proposes an intelligent mission coordination methodology that integrates digital mission models with Reinforcement Learning (RL), that specifically addresses the need for adaptive task allocation and reconfiguration. More specifically, we are leveraging a Digital Engineering (DE) based infrastructure that is composed of a high-fidelity digital mission model and agent-based simulation; and then we formulate the mission tactics management problem as a Markov Decision Process (MDP), and employ an RL agent trained via Proximal Policy Optimization. By leveraging the simulation as a sandbox, we map the system states to actions, refining the policy based on realized mission outcomes. The utility of the RL-based intelligent mission coordinator is demonstrated through an aerial firefighting case study. Our findings indicate that the RL-based intelligent mission coordinator not only surpasses baseline performance but also significantly reduces the variability in mission performance. Thus, this study serves as a proof of concept demonstrating that DE-enabled mission simulations combined with advanced analytical tools offer a mission-agnostic framework for improving ME practice; which can be extended to more complicated fleet design and selection problems in the future from a mission-first perspective.

cs.CY↗

arXiv2025

What is the Return on Investment of Digital Engineering for Complex Systems Development? Findings from a Mixed-Methods Study on the Post-production Design Change Process of Navy Assets

Complex engineered systems routinely face schedule and cost overruns, along with poor post-deployment performance. Championed by both INCOSE and the U.S. Department of Defense (DoD), the systems engineering (SE) community has increasingly looked to Digital Engineering (DE) as a potential remedy. Despite this growing advocacy, most of DE's purported benefits remain anecdotal, and its return on investment (ROI) remains poorly understood. This research presents findings from a case study on a Navy SE team responsible for the preliminary design phase of post-production design change projects for Navy assets. Using a mixed-methods approach, we document why complex system sustainment projects are routinely late, where and to what extent schedule slips arise, and how a DE transformation could improve schedule adherence. This study makes three contributions. First, it identifies four archetypical inefficiency modes that drive schedule overruns and explains how these mechanisms unfold in their organizational context. Second, it quantifies the magnitude and variation of schedule slips. Third, it creates a hypothetical digitally transformed version of the current process, aligned with DoD DE policy, and compares it to the current state to estimate potential schedule gains. Our findings suggest that a DE transformation could reduce the median project duration by 50.1% and reduce the standard deviation by 41.5%, leading to faster and more predictable timelines. However, the observed gains are not uniform across task categories. Overall, this study provides initial quantitative evidence of DE's potential ROI and its value in improving the efficiency and predictability of complex system sustainment projects.

cs.CY↗

arXiv2025

An Analysis of Early-Stage Functional Safety Analysis Methods and Their Integration into Model-Based Systems Engineering

As systems become increasingly complex, conducting effective safety analysis in the earlier phases of a system's lifecycle is essential to identify and mitigate risks before they escalate. To that end, this paper investigates the capabilities of key safety analysis techniques, namely: Failure Mode and Effects Analysis (FMEA), Functional Hazard Analysis (FHA), and Functional Failure Identification and Propagation (FFIP), along with the current state of the literature in terms of their integration into Model-Based Systems Engineering (MBSE). A two-phase approach is adopted. The first phase is focused on contrasting FMEA, FHA, and FFIP techniques, examining their procedures, along with a documentation of their relative strengths and limitations. Our analysis highlights FFIP's capability in identifying emergent system behaviors, second-order effects, and fault propagation; thus, suggesting it is better suited for the safety needs of modern interconnected systems. Second, we review the existing research on the efforts to integrate each of these methods into MBSE. We find that MBSE integration efforts primarily focus on FMEA, and integration of FHA and FFIP is nascent. Additionally, FMEA-MBSE integration efforts could be organized into four categories: model-to-model transformation, use of external customized algorithms, built-in MBSE packages, and manual use of standard MBSE diagrams. While our findings indicate a variety of MBSE integration approaches, there is no universally established framework or standard. This leaves room for an integration approach that could support the ongoing Digital Engineering transformation efforts by enabling a more synergistic lifecycle safety management methods and tools.

cs.SE↗

arXiv2025

Digital Engineering Transformation as a Sociotechnical Challenge: Categorization of Barriers and Their Mapping to DoD's Policy Goals

Digital Engineering (DE) transformation represents a paradigm shift in systems engineering (SE), aiming to integrate diverse analytical models and digital artifacts into an authoritative source of truth for improved traceability and more efficient system lifecycle management. Despite institutional support, many DE initiatives underperform or fail to realize their intended benefits. We argue that this often results from a limited understanding of the social and technical barriers, and particularly how their interplay shapes transformation outcomes. To address this gap, we document barriers identified in the literature and grounded in sociotechnical systems theory, organized into six dimensions: people, processes, culture, goals, infrastructure, and technology. We then map these barriers to the U.S. Department of Defense's DE policy goals. Our analysis shows that technological investments alone are insufficient, as failures frequently arise from social factors such as workforce readiness, leadership support, and cultural alignment. The mapping also demonstrates that sociotechnical barriers often cascade across dimensions, making their impact on policy goals difficult to trace and complicating implementation. These insights carry practical implications: managers may use the mapping as a diagnostic tool to identify risks and prioritize resources; policymakers may complement strategic mandates with sustained investments and long-term change management; and engineers may view DE not as a threat to job security but as an opportunity for more effective collaboration.

cs.ET↗

arXiv2025

Stakeholder Perspectives on Digital Twin Implementation Challenges in Healthcare: Insights from a Provider Digital Twin Case Study

Digital twin (DT) technology holds immense potential for transforming healthcare systems through real-time monitoring, predictive analysis, and agile interventions to support various decision-making needs. However, its successful implementation depends on addressing a range of complex sociotechnical challenges. Using a case study of provider workload DT, this research investigates DT implementation challenges in healthcare by capturing the perspectives of four distinct stakeholders: family medicine specialists (FMS), organizational psychologists (OP), engineers (EE), and implementation scientists (IS). We conducted semi-structured interviews guided by the updated Consolidated Framework for Implementation Research (CFIR 2.0), a widely used implementation science framework for understanding factors that influence implementation outcomes. We then mapped each stakeholder group's preferences and concerns, revealing a nuanced landscape of converging and diverging perspectives that highlight both shared and group-specific implementation barriers. Through thematic coding, the 66 identified challenges were categorized into seven domains: data-related, financial and economic, operational, organizational, personnel, regulatory and ethical, and technological. Our findings reveal shared concerns such as data privacy and security, interoperability, and regulatory compliance. However, divergences also emerged, reflecting each group's functional focus. These findings emphasize the need for a multidisciplinary, stakeholder-sensitive approach that addresses both functional and practical concerns, highlighting the importance of tailored implementation strategies to support successful DT adoption in healthcare.

cs.CY↗

arXiv2025

Exploring Core and Periphery Precepts in Biological and Artificial Intelligence: An Outcome-Based Perspective

Engineering methodologies predominantly revolve around established principles of decomposition and recomposition. These principles involve partitioning inputs and outputs at the component level, ensuring that the properties of individual components are preserved upon composition. However, this view does not transfer well to intelligent systems, particularly when addressing the scaling of intelligence as a system property. Our prior research contends that the engineering of general intelligence necessitates a fresh set of overarching systems principles. As a result, we introduced the "core and periphery" principles, a novel conceptual framework rooted in abstract systems theory and the Law of Requisite Variety. In this paper, we assert that these abstract concepts hold practical significance. Through empirical evidence, we illustrate their applicability to both biological and artificial intelligence systems, bridging abstract theory with real-world implementations. Then, we expand on our previous theoretical framework by mathematically defining core-dominant vs periphery-dominant systems.

cs.AI↗

arXiv2025

Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a Characterization of Failure Modes

Multi-purpose Large Language Models (LLMs), a subset of generative Artificial Intelligence (AI), have recently made significant progress. While expectations for LLMs to assist systems engineering (SE) tasks are paramount; the interdisciplinary and complex nature of systems, along with the need to synthesize deep-domain knowledge and operational context, raise questions regarding the efficacy of LLMs to generate SE artifacts, particularly given that they are trained using data that is broadly available on the internet. To that end, we present results from an empirical exploration, where a human expert-generated SE artifact was taken as a benchmark, parsed, and fed into various LLMs through prompt engineering to generate segments of typical SE artifacts. This procedure was applied without any fine-tuning or calibration to document baseline LLM performance. We then adopted a two-fold mixed-methods approach to compare AI generated artifacts against the benchmark. First, we quantitatively compare the artifacts using natural language processing algorithms and find that when prompted carefully, the state-of-the-art algorithms cannot differentiate AI-generated artifacts from the human-expert benchmark. Second, we conduct a qualitative deep dive to investigate how they differ in terms of quality. We document that while the two-material appear very similar, AI generated artifacts exhibit serious failure modes that could be difficult to detect. We characterize these as: premature requirements definition, unsubstantiated numerical estimates, and propensity to overspecify. We contend that this study tells a cautionary tale about why the SE community must be more cautious adopting AI suggested feedback, at least when generated by multi-purpose LLMs.

cs.CL↗

arXiv2025

An Empirical Exploration of ChatGPT's Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of its Performance Variability

Systems engineering (SE) is evolving with the availability of generative artificial intelligence (AI) and the demand for a systems-of-systems perspective, formalized under the purview of mission engineering (ME) in the US Department of Defense. Formulating ME problems is challenging because they are open-ended exercises that involve translation of ill-defined problems into well-defined ones that are amenable for engineering development. It remains to be seen to which extent AI could assist problem formulation objectives. To that end, this paper explores the quality and consistency of multi-purpose Large Language Models (LLM) in supporting ME problem formulation tasks, specifically focusing on stakeholder identification. We identify a relevant reference problem, a NASA space mission design challenge, and document ChatGPT-3.5's ability to perform stakeholder identification tasks. We execute multiple parallel attempts and qualitatively evaluate LLM outputs, focusing on both their quality and variability. Our findings portray a nuanced picture. We find that the LLM performs well in identifying human-focused stakeholders but poorly in recognizing external systems and environmental factors, despite explicit efforts to account for these. Additionally, LLMs struggle with preserving the desired level of abstraction and exhibit a tendency to produce solution specific outputs that are inappropriate for problem formulation. More importantly, we document great variability among parallel threads, highlighting that LLM outputs should be used with caution, ideally by adopting a stochastic view of their abilities. Overall, our findings suggest that, while ChatGPT could reduce some expert workload, its lack of consistency and domain understanding may limit its reliability for problem formulation tasks.

cs.SE↗