arXiv subjects

Rick Rabiser

Publications and source records attributed to Rick Rabiser.

9 recordsLinked to original sources

arXiv2026

Performance Optimization in Stream Processing Systems: Experiment-Driven Configuration Tuning for Kafka Streams

Configuring stream processing systems for efficient performance, especially in cloud-native deployments, is a challenging and largely manual task. We present an experiment-driven approach for automated configuration optimization that combines three phases: Latin Hypercube Sampling for initial exploration, Simulated Annealing for guided stochastic search, and Hill Climbing for local refinement. The workflow is integrated with the cloud-native Theodolite benchmarking framework, enabling automated experiment orchestration on Kubernetes and early termination of underperforming configurations. In an experimental evaluation with Kafka Streams and a Kubernetes-based cloud testbed, our approach identifies configurations that improve throughput by up to 23% over the default. The results indicate that Latin Hypercube Sampling with early termination and Simulated Annealing are particularly effective in navigating the configuration space, whereas additional fine-tuning via Hill Climbing yields limited benefits.

cs.PF↗

arXiv2025

Should I Run My Cloud Benchmark on Black Friday?

Benchmarks and performance experiments are frequently conducted in cloud environments. However, their results are often treated with caution, as the presumed high variability of performance in the cloud raises concerns about reproducibility and credibility. In a recent study, we empirically quantified the impact of this variability on benchmarking results by repeatedly executing a stream processing application benchmark at different times of the day over several months. Our analysis confirms that performance variability is indeed observable at the application level, although it is less pronounced than often assumed. The larger scale of our study compared to related work allowed us to identify subtle daily and weekly performance patterns. We now extend this investigation by examining whether a major global event, such as Black Friday, affects the outcomes of performance benchmarks.

cs.SE↗

arXiv2025

When Should I Run My Application Benchmark?: Studying Cloud Performance Variability for the Case of Stream Processing Applications

Performance benchmarking is a common practice in software engineering, particularly when building large-scale, distributed, and data-intensive systems. While cloud environments offer several advantages for running benchmarks, it is often reported that benchmark results can vary significantly between repetitions -- making it difficult to draw reliable conclusions about real-world performance. In this paper, we empirically quantify the impact of cloud performance variability on benchmarking results, focusing on stream processing applications as a representative type of data-intensive, performance-critical system. In a longitudinal study spanning more than three months, we repeatedly executed an application benchmark used in research and development at Dynatrace. This allows us to assess various aspects of performance variability, particularly concerning temporal effects. With approximately 591 hours of experiments, deploying 789 Kubernetes clusters on AWS and executing 2366 benchmarks, this is likely the largest study of its kind and the only one addressing performance from an end-to-end, i.e., application benchmark perspective. Our study confirms that performance variability exists, but it is less pronounced than often assumed (coefficient of variation of < 3.7%). Unlike related studies, we find that performance does exhibit a daily and weekly pattern, although with only small variability (<= 2.5%). Re-using benchmarking infrastructure across multiple repetitions introduces only a slight reduction in result accuracy (<= 2.5 percentage points). These key observations hold consistently across different cloud regions and machine types with different processor architectures. We conclude that for engineers and researchers focused on detecting substantial performance differences (e.g., > 5%) in...

cs.SE↗

arXiv2024

High-level Stream Processing: A Complementary Analysis of Fault Recovery

Parallel computing is very important to accelerate the performance of software systems. Additionally, considering that a recurring challenge is to process high data volumes continuously, stream processing emerged as a paradigm and software architectural style. Several software systems rely on stream processing to deliver scalable performance, whereas open-source frameworks provide coding abstraction and high-level parallel computing. Although stream processing's performance is being extensively studied, the measurement of fault tolerance--a key abstraction offered by stream processing frameworks--has still not been adequately measured with comprehensive testbeds. In this work, we extend the previous fault recovery measurements with an exploratory analysis of the configuration space, additional experimental measurements, and analysis of improvement opportunities. We focus on robust deployment setups inspired by requirements for near real-time analytics of a large cloud observability platform. The results indicate significant potential for improving fault recovery and performance. However, these improvements entail grappling with configuration complexities, particularly in identifying and selecting the configurations to be fine-tuned and determining the appropriate values for them. Therefore, new abstractions for transparent configuration tuning are also needed for large-scale industry setups. We believe that more software engineering efforts are needed to provide insights into potential abstractions and how to achieve them. The stream processing community and industry practitioners could also benefit from more interactions with the high-level parallel programming community, whose expertise and insights on making parallel programming more productive and efficient could be extended.

cs.DC↗

arXiv2024

A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks

Nowadays, several software systems rely on stream processing architectures to deliver scalable performance and handle large volumes of data in near real-time. Stream processing frameworks facilitate scalable computing by distributing the application's execution across multiple machines. Despite performance being extensively studied, the measurement of fault tolerance-a key feature offered by stream processing frameworks-has still not been measured properly with updated and comprehensive testbeds. Moreover, the impact that fault recovery can have on performance is mostly ignored. This paper provides a comprehensive analysis of fault recovery performance, stability, and recovery time in a cloud-native environment with modern open-source frameworks, namely Flink, Kafka Streams, and Spark Structured Streaming. Our benchmarking analysis is inspired by chaos engineering to inject failures. Generally, our results indicate that much has changed compared to previous studies on fault recovery in distributed stream processing. In particular, the results indicate that Flink is the most stable and has one of the best fault recovery. Moreover, Kafka Streams shows performance instabilities after failures, which is due to its current rebalancing strategy that can be suboptimal in terms of load balancing. Spark Structured Streaming shows suitable fault recovery performance and stability, but with higher event latency. Our study intends to (i) help industry practitioners in choosing the most suitable stream processing framework for efficient and reliable executions of data-intensive applications; (ii) support researchers in applying and extending our research method as well as our benchmark; (iii) identify, prevent, and assist in solving potential issues in production deployments.

cs.DC↗

arXiv2024

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Distributed stream processing frameworks help building scalable and reliable applications that perform transformations and aggregations on continuous data streams. This paper introduces ShuffleBench, a novel benchmark to evaluate the performance of modern stream processing frameworks. In contrast to other benchmarks, it focuses on use cases where stream processing frameworks are mainly employed for shuffling (i.e., re-distributing) data records to perform state-local aggregations, while the actual aggregation logic is considered as black-box software components. ShuffleBench is inspired by requirements for near real-time analytics of a large cloud observability platform and takes up benchmarking metrics and methods for latency, throughput, and scalability established in the performance engineering research community. Although inspired by a real-world observability use case, it is highly configurable to allow domain-independent evaluations. ShuffleBench comes as a ready-to-use open-source software utilizing existing Kubernetes tooling and providing implementations for four state-of-the-art frameworks. Therefore, we expect ShuffleBench to be a valuable contribution to both industrial practitioners building stream processing applications and researchers working on new stream processing approaches. We complement this paper with an experimental performance evaluation that employs ShuffleBench with various configurations on Flink, Hazelcast, Kafka Streams, and Spark in a cloud-native environment. Our results show that Flink achieves the highest throughput while Hazelcast processes data streams with the lowest latency.

cs.SE↗

arXiv2024

On the Challenges of Transforming UVL to IVML

Software product line techniques encourage the reuse and adaptation of software components for creating customized products or software systems. These different product variants have commonalities and differences, which are managed by variability modeling. Over the past three decades, both academia and industry have developed numerous variability modeling methods, each with its own advantages and disadvantages. Many of these methods have demonstrated their utility within specific domains or applications. However, comprehending the capabilities and differences among these approaches to pinpoint the most suitable one for a particular use case remains challenging. Thus, new modeling techniques and tailored tools for handling variability are frequently created. Transitioning between variability models through transformations from different approaches can help in understanding the benefits and drawbacks of different modeling approaches. However, implementing such transformations presents challenges, such as semantic preservation and avoiding information loss. TRAVART is a tool that helps with transitioning between different approaches by enabling the transformation of variability models into other variability models of different types. This paper discusses the challenges for such transformations between UVL and IVML. It also presents a one-way transformation from the UVL to IVML with as little information loss as possible.

cs.SE↗

arXiv2024

Variability Modeling of Products, Processes, and Resources in Cyber-Physical Production Systems Engineering

Cyber-Physical Production Systems (CPPSs), such as automated car manufacturing plants, execute a configurable sequence of production steps to manufacture products from a product portfolio. In CPPS engineering, domain experts start with manually determining feasible production step sequences and resources based on implicit knowledge. This process is hard to reproduce and highly inefficient. In this paper, we present the Extended Iterative Process Sequence Exploration (eIPSE) approach to derive variability models for products, processes, and resources from a domain-specific description. To automate the integrated exploration and configuration process for a CPPS, we provide a toolchain which automatically reduces the configuration space and allows to generate CPPS artifacts, such as control code for resources. We evaluate the approach with four real-world use cases, including the generation of control code artifacts, and an observational user study to collect feedback from engineers with different backgrounds. The results confirm the usefulness of the eIPSE approach and accompanying prototype to straightforwardly configure a desired CPPS.

cs.SE↗

arXiv2020

Comparing Constraints Mined From Execution Logs to Understand Software Evolution

Complex software systems evolve frequently, e.g., when introducing new features or fixing bugs during maintenance. However, understanding the impact of such changes on system behavior is often difficult. Many approaches have thus been proposed that analyze systems before and after changes, e.g., by comparing source code, model-based representations, or system execution logs. In this paper, we propose an approach for comparing run-time constraints, synthesized by a constraint mining algorithm, based on execution logs recorded before and after changes. Specifically, automatically mined constraints define the expected timing and order of recurring events and the values of data elements attached to events. Our approach presents the differences of the mined constraints to users, thereby providing a higher-level view on software evolution and supporting the analysis of the impact of changes on system behavior. We present a motivating example and a preliminary evaluation based on a cyber-physical system controlling unmanned aerial vehicles. The results of our preliminary evaluation show that our approach can help to analyze changed behavior and thus contributes to understanding software evolution.

cs.SE↗