arXiv Science⌕ Search

arXiv · 2611.09896

From Prompt Engineering to Programmatic Prompting: Toward a Software Engineering Discipline for Large Language Models

Abstract

The practice of instructing large language models (LLMs) through natural-language prompts has, in the span of a few years, migrated from an interactive curiosity to a load-bearing component of production software. Yet the vocabulary, tooling, and mental models of the field remain anchored in its interactive origins: we still speak of "prompt engineering" as though the crafting of a single well-worded instruction were the central problem. This paper argues for a categorical distinction. Prompt engineering, we contend, is an interaction technique - a human-computer interaction skill for eliciting desired behavior from a model in a conversational or exploratory setting. Programmatic prompting, by contrast, is an engineering discipline governing the behavioral interface between software systems and language models: the construction, composition, versioning, testing, and maintenance of prompts as first-class software artifacts embedded in larger systems. Conflating the two has produced brittle applications, unrepeatable evaluations, and an accumulation of prompt debt. We make three contributions: (1) a definition of programmatic prompting; (2) a systematic differentiation of programmatic prompting from prompt engineering, prompt tuning, fine-tuning, and general LLM application development; and (3) an account of prompt debt that distinguishes the self-admitted debt measurable in code from the silent debt that surfaces only as incidents, and identifies its economic cause. Our thesis is simple to state and consequential to accept: prompt engineering is an interaction technique; programmatic prompting is an engineering discipline.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Sebastian Dingler. 2026-07-19. From Prompt Engineering to Programmatic Prompting: Toward a Software Engineering Discipline for Large Language Models. https://arxiv.org/abs/2611.09896

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

Understanding Developer Information Needs for Software with Embedded Natural Language Prompts

Developers use foundation models (FMs) to create novel forms of intelligent software by writing natural language prompts embedded within a software's implementation---referred to as embedded prompts---that are also passed as a string within an FM's API. Developing software with embedded prompts requires developers to engage in iterative embedded prompt changes to obtain the desired behavior. While the information needs of traditional software have been investigated, the information needs of modern developers for embedded prompt changes are underexplored, despite these needs shaping how developers plan and justify embedded prompt changes. In this work, we perform a mixed-method exploratory study on the information needs of developers making changes to embedded prompts. We create, refine, validate, and apply a list of 50 information needs that developers have while making a change to an embedded prompt by interviewing 16 developers, observing 8 developers make prompt changes, surveying 50 developers, and evaluating 72 research and commercial tools. We find that embedded prompts are not well-supported by existing tools: 16 of the 50 information needs, including four of the most important ones, are unaddressed. We conclude with important design opportunities that should be addressed by future developer tools for embedded prompts.

cs.SE↗

Enhancing Neural Code Representation with Additional Context

Automated program comprehension underpins many software engineering tasks, from code summarisation to clone detection. While recent deep learning models have achieved strong results, they typically rely on source code alone, overlooking contextual signals such as version history or structural relationships. We conduct an empirical study of how enriching code representations with such signals affects model performance on four tasks: code clone detection, code classification, code summarisation, and vulnerability detection, using three datasets (SeSaMe, CodeSearchNet-Java, Vul4J). We evaluate five foundational code models (CodeBERT, GraphCodeBERT, CodeT5, PLBART, ASTNN) and seven code-specialised LLMs (CodeLlama, Qwen2.5-Coder, DeepSeek-Coder), comparing prompt-based context injection against fine-tuned use of context. Context improves classification for all five foundational models and, after fine-tuning, for most LLMs. For summarisation, the effect depends on the model family: version history improves all four foundational models on every metric, whereas prompt-level context helps two of the seven LLMs and hurts the other five. Combining context types reaches up to +21.48% macro-F1 in code classification and +15.04% F1 in clone detection. A human evaluation of CodeT5 summaries on 100 Java snippets ranks context-augmented summaries significantly higher for Accuracy and Content Adequacy. An exploratory vulnerability detection evaluation provides initial evidence of transfer to security-oriented tasks.

cs.SE↗

Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems

LLM agents are explored for automating root cause analysis (RCA) in cloud-native systems, creating a need to evaluate both diagnostic correctness and the quality of the supporting investigation. Existing static benchmarks offer repeatable inputs but limited system-facing interaction, while live testbeds expose tools but hinder controlled comparison because incident evidence varies across runs; both paradigms focus primarily on final answers. To address these limitations, we present Cloud-OpsBench, an evaluation infrastructure for interactive and evidence-grounded cloud RCA. It comprises 754 runtime-verified cases across 57 fault types on two microservice workloads spanning application services and Kubernetes platform layers. Each fault is captured as a state snapshot and replayed through standard diagnostic interfaces, with outcome labels and diagnostic evidence graphs that enable matched comparisons and process-level analysis. Evaluating eleven LLMs on Cloud-OpsBench reveals that strong outcome accuracy often masks incomplete investigations: while Joint RCA Accuracy (JRA) reaches up to 0.78 on OnlineBoutique and 0.68 on TrainTicket, models identify the correct root cause without achieving evidence closure (e.g., in 33.1% and 29.4% of evaluated cases for GPT-5). These results demonstrate that final-answer accuracy alone cannot establish whether an agent has traced the failure and collected the evidence needed to confirm its root cause.

cs.SE↗