arXiv subjects

Florian Angermeir

Publications and source records attributed to Florian Angermeir.

11 recordsLinked to original sources

arXiv2026

RefVerifier: Semi-Automated Reference Claim Verification for Scientific Manuscripts

As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.

cs.SE↗

arXiv2026

From Backlog Items to Security Guidance: Towards Continuous Security Compliance

Continuous software engineering in regulated domains requires engineering teams to address security throughout the development lifecycle. Yet making security requirements explicit in backlog items is still problematic. Engineers must instead infer security relevance of backlog items from brief, free-form descriptions and often lack timely guidance on applicable requirements. We present an NLP-based backlog enrichment system that detects security-relevant backlog items and links them to relevant security requirements. The approach combines a security-relevance classifier with a retrieval-augmented generation (RAG) pipeline over security requirements documents. The approach was developed and evaluated in the context of a large enterprise in highly regulated domains. We present three contributions. First, we release a dataset of 288 backlog items labeled for security relevance by nine security practitioners, with substantial agreement (Fleiss' $\kappa=0.787$). Second, a recall-oriented classifier achieving $F2=0.774$ in-distribution and mean zero-shot G-measure $\approx 0.65$ across five established benchmarks, matching or outperforming most published classical-ML and open-source GPT baselines. Third, we preliminarily evaluated a four-stage security requirements document-grounded RAG pipeline with two practitioners on industrial backlogs using company-internal security policies and CIS Benchmarks. Of the retrieved 24 clauses, 12 were rated at least 4/5 for relevance. Our findings provide first indicators that NLP-based product backlog enrichment can support engineers in identifying security requirements early in the development process. With this work we aim to facilitate continuous security compliance through proactive introduction of security requirements in continuous software engineering.

cs.SE↗

arXiv2026

An Agentic Approach Towards Replication Package Quality Evaluation

Reproducibility in empirical software engineering relies on complete, accessible, and reusable research artifacts, yet artifact evaluation remains largely manual and difficult to scale. This emerging results paper explores an agentic approach for assessing replication package quality by translating open-science guidelines into machine-verifiable criteria. We consolidate 380 requirements from 34 sources into 51 reproducibility criteria, of which 31 are operationalized for automated artifact-based evaluation. Based on these criteria, we implement a multi-agent prototype that inspects replication packages and produces evidence-grounded improvement reports. A preliminary evaluation on five replication packages shows high inter-run consistency of 91.4% and 75.4% correctness, through micro-averaged agreement with a manual baseline. The agent performs best on structural criteria such as code, environment, and artifact availability, but struggles with qualitative or mixed-method studies. A pilot survey with seven software engineering researchers indicates well perceived usefulness and adoption potential, while revealing cognitive load in the human-in-the-loop planning step. Overall, within these small samples, the results indicate that agentic research artifact evaluation has the potential to support authors and reviewers by automating selected routine checks.

cs.SE↗

arXiv2025

Aligning Security Compliance and DevOps: A Longitudinal Study

Companies adopt agile methodologies and DevOps to facilitate efficient development and deployment of software-intensive products. This, in turn, introduces challenges in relation to security standard compliance traditionally following a more linear workflow. This is especially a challenge for the engineering of products and services associated with critical infrastructures. To support companies in their transition towards DevOps, this paper presents an adaptation of DevOps according to security regulations and standards. We report on our longitudinal study at Siemens AG, consisting of several individual sub-studies in the inception, validation, and initial adoption of our framework based on RefA as well as the implications for practice. RefA is a prescriptive model of a security compliant DevOps lifecycle based on the IEC 62443-4-1 standard. The overall framework is aimed at professionals, not only security experts, being able to use it on implementing DevOps processes while remaining compliant with security norms. We demonstrate how RefA facilitates the transfer of security compliance knowledge to product development teams. This knowledge transfer supports the agility aim of ensuring that cross-functional teams have all the skills needed to deliver the compliant products.

cs.SE↗

arXiv2025

Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies

Large Language Models have gained remarkable interest in industry and academia. The increasing interest in LLMs in academia is also reflected in the number of publications on this topic over the last years. For instance, alone 78 of the around 425 publications at ICSE 2024 performed experiments with LLMs. Conducting empirical studies with LLMs remains challenging and raises questions on how to achieve reproducible results, for both researchers and practitioners. One important step towards excelling in empirical research on LLM and their application is to first understand to what extent current research results are eventually reproducible and what factors may impede reproducibility. This investigation is within the scope of our work. We contribute an analysis of the reproducibility of LLM-centric studies, provide insights into the factors impeding reproducibility, and discuss suggestions on how to improve the current state. In particular, we studied the 85 articles describing LLM-centric studies, published at ICSE 2024 and ASE 2024. Of the 85 articles, 18 provided research artefacts and used OpenAI models. We attempted to replicate those 18 studies. Of the 18 studies, only five were sufficiently complete and executable. For none of the five studies, we were able to fully reproduce the results. Two studies seemed to be partially reproducible, and three studies did not seem to be reproducible. Our results highlight not only the need for stricter research artefact evaluations but also for more robust study designs to ensure the reproducible value of future publications.

cs.SE↗

arXiv2025

Guidelines for Empirical Studies in Software Engineering involving Large Language Models

Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommendations (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the system and prompt design beyond the model; (4) report session traces, i.e., interaction logs and runtime traces; (5) use suitable baselines, benchmarks, and metrics; (6) include an open LLM as a baseline; (7) validate LLM outputs against human judgment; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines$.$org).

cs.SE↗

arXiv2024

Towards Automated Continuous Security Compliance

Context: Continuous Software Engineering is increasingly adopted in highly regulated domains, raising the need for continuous compliance. Adherence to especially security regulations -- a major concern in highly regulated domains -- renders Continuous Security Compliance of high relevance to industry and research. Problem: One key barrier to adopting continuous software engineering in the industry is the resource-intensive and error-prone nature of traditional manual security compliance activities. Automation promises to be advantageous. However, continuous security compliance is under-researched, precluding an effective adoption. Contribution: We have initiated a long-term research project with our industry partner to address these issues. In this manuscript, we make three contributions: (1) We provide a precise definition of the term continuous security compliance aligning with the state-of-art, (2) elaborate a preliminary overview of challenges in the field of automated continuous security compliance through a tertiary literature study, and (3) present a research roadmap to address those challenges via automated continuous security compliance.

cs.SE↗

arXiv2024

No Free Lunch: Research Software Testing in Teaching

Software is at the core of most scientific discoveries today. Therefore, the quality of research results highly depends on the quality of the research software. Rigorous testing, as we know it from software engineering in the industry, could ensure the quality of the research software but it also requires a substantial effort that is often not rewarded in academia. Therefore, this research explores the effects of research software testing integrated into teaching on research software. In an in-vivo experiment, we integrated the engineering of a test suite for a large-scale network simulation as group projects into a course on software testing at the Blekinge Institute of Technology, Sweden, and qualitatively measured the effects of this integration on the research software. We found that the research software benefited from the integration through substantially improved documentation and fewer hardware and software dependencies. However, this integration was effortful and although the student teams developed elegant and thoughtful test suites, no code by students went directly into the research software since we were not able to make the integration back into the research software obligatory or even remunerative. Although we strongly believe that integrating research software engineering such as testing into teaching is not only valuable for the research software itself but also for students, the research of the next generation, as they get in touch with research software engineering and bleeding-edge research in their field as part of their education, the uncertainty about the intellectual properties of students' code substantially limits the potential of integrating research software testing into teaching.

cs.SE↗

arXiv2024

Industrial Challenges in Secure Continuous Development

The intersection between security and continuous software engineering has been of great interest since the early years of the agile development movement, and it remains relevant as software development processes are more frequently guided by agility and the adoption of DevOps. Several authors have contributed studies about the framing of secure agile development and secure DevOps, motivating academic contributions to methods and practices, but also discussions around benefits and challenges. Especially the challenges captured also our interest since, for the last few years, we are conducting research on secure continuous software engineering from a more applied, practical perspective with the overarching aim to introduce solutions that can be adopted at scale. The short positioning at hands summarizes a relevant part of our endeavors in which we validated challenges with several practitioners of different roles. More than framing a set of challenges, we conclude by presenting four key research directions we identified for practitioners and researchers to delineate future work.

cs.SE↗

arXiv2024

Automated Security Findings Management: A Case Study in Industrial DevOps

In recent years, DevOps, the unification of development and operation workflows, has become a trend for the industrial software development lifecycle. Security activities turned into an essential field of application for DevOps principles as they are a fundamental part of secure software development in the industry. A common practice arising from this trend is the automation of security tests that analyze a software product from several perspectives. To effectively improve the security of the analyzed product, the identified security findings must be managed and looped back to the project team for stakeholders to take action. This management must cope with several challenges ranging from low data quality to a consistent prioritization of findings while following DevOps aims. To manage security findings with the same efficiency as other activities in DevOps projects, a methodology for the management of industrial security findings minding DevOps principles is essential. In this paper, we propose a methodology for the management of security findings in industrial DevOps projects, summarizing our research in this domain and presenting the resulting artifact. As an instance of the methodology, we developed the Security Flama, a semantic knowledge base for the automated management of security findings. To analyze the impact of our methodology on industrial practice, we performed a case study on two DevOps projects of a multinational industrial enterprise. The results emphasize the importance of using such an automated methodology in industrial DevOps projects, confirm our approach's usefulness and positive impact on the studied projects, and identify the communication strategy as a crucial factor for usability in practice.

cs.SE↗

arXiv2021

Enterprise-Driven Open Source Software: A Case Study on Security Automation

Agile and DevOps are widely adopted by the industry. Hence, integrating security activities with industrial practices, such as continuous integration (CI) pipelines, is necessary to detect security flaws and adhere to regulators' demands early. In this paper, we analyze automated security activities in CI pipelines of enterprise-driven open source software (OSS). This shall allow us, in the long-run, to better understand the extent to which security activities are (or should be) part of automated pipelines. In particular, we mine publicly available OSS repositories and survey a sample of project maintainers to better understand the role that security activities and their related tools play in their CI pipelines. To increase transparency and allow other researchers to replicate our study (and to take different perspectives), we further disclose our research artefacts. Our results indicate that security activities in enterprise-driven OSS projects are scarce and protection coverage is rather low. Only 6.83% of the analyzed 8,243 projects apply security automation in their CI pipelines, even though maintainers consider security to be rather important. This alerts industry to keep the focus on vulnerabilities of 3rd Party software and it opens space for other improvements of practice which we outline in this manuscript.

cs.SE↗