arXiv subjects

Michael Fromm

Publications and source records attributed to Michael Fromm.

At least 19 recordsLinked to original sources

arXiv2026

A Sovereign, Open-Source Foundation Model for German and English

We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.

cs.CL↗

arXiv2026

KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report

High-quality pretraining data is a central ingredient in modern language models, but German-language resources remain far less developed than their English counterparts: they are often smaller, less carefully curated, weakly documented, and rarely validated through controlled training experiments. We introduce KletterMix, a high-quality German corpus for language model pretraining and annealing, designed as a reusable dataset artifact for the natural language processing and modeling community. KletterMix is built by translating a state-of-the-art English pretraining corpus into German while preserving document boundaries, metadata, source structure, and topical diversity. This construction yields a German corpus with the scale and diversity of a modern pretraining dataset, while enabling direct comparison to its English source. We document the dataset through a broad set of corpus-level analyses, including translation quality, document length distributions, topic coverage, source composition, and geographic metadata. Using COMETKiwi, we show that the translated documents achieve strong quality across diverse domains, suggesting that careful translation can preserve much of the semantic and stylistic richness of the original corpus. Beyond dataset construction, we evaluate KletterMix as training data. Through controlled pretraining and annealing ablations against established German corpora, we show that models trained on KletterMix achieve measurable improvements on German-language downstream evaluations. These results demonstrate that carefully curated translated data can substantially strengthen the German pretraining data ecosystem.

cs.CL↗

arXiv2026

Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research

Today's LLM (pre-) training and research workflows typically allocate a significant amount of compute to large-scale ablation studies. Despite the substantial compute costs of these ablations, existing open-source frameworks provide limited tooling for these experiments, often forcing researchers to write their own wrappers and scripts. We propose Modalities, an end-to-end PyTorch-native framework that integrates data-driven LLM research with large-scale model training from two angles. Firstly, by integrating state-of-the-art parallelization strategies, it enables both efficient pretraining and systematic ablations at trillion-token and billion-parameter scale. Secondly, Modalities adopts modular design with declarative, self-contained configuration, enabling reproducibility and extensibility levels that are difficult to achieve out-of-the-box with existing LLM training frameworks.

cs.LG↗

arXiv2025

On the nature of the QCD chiral phase transition with imaginary chemical potential

The order of the thermal chiral phase transition in lattice QCD is known to be strongly cutoff-dependent. A previous study using $N_\mathrm{f}\in[2,6]$ mass-degenerate, unimproved staggered quark flavours on $N_\tau\in\{4,6,8\}$ lattices found that the bare mass regions displaying explicit first-order transitions shrink to zero, with their critical boundary line terminating in a tricritical point before the continuum limit is reached. Here we perform an analogous study for fixed imaginary baryon chemical potential and find the same behaviour: first-order regions observed on coarse lattices disappear in tricritical points with diminishing lattice spacing. These observations are consistent with currently available results from improved staggered discretisations, both at zero and non-zero imaginary chemical potential. Unless additional first-order transitions are found on finer lattices or with chiral lattice actions, this implies a second-order transition in the continuum chiral limit for all these cases, at zero and imaginary chemical potential. Implications for the $N_\mathrm{f}=2+1$ QCD phase diagram at the physical point are discussed.

hep-lat↗

arXiv2025

Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models

High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets remains limited. Existing state-of-the-art datasets mostly rely on heuristic filtering methods, restricting both their cross-lingual transferability and scalability. Here, we introduce JQL, a systematic approach that efficiently curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. JQL distills LLMs' annotation capabilities into lightweight annotators based on pretrained multilingual embeddings. These models exhibit robust multilingual and cross-lingual performance, even for languages and scripts unseen during training. Evaluated empirically across 35 languages, the resulting annotation pipeline substantially outperforms current heuristic filtering methods like Fineweb2. JQL notably enhances downstream model training quality and increases data retention rates. Our research provides practical insights and valuable resources for multilingual data curation, raising the standards of multilingual dataset development.

cs.CL↗

arXiv2024

Data Processing for the OpenGPT-X Model Family

This paper presents a comprehensive overview of the data preparation pipeline developed for the OpenGPT-X project, a large-scale initiative aimed at creating open and high-performance multilingual large language models (LLMs). The project goal is to deliver models that cover all major European languages, with a particular focus on real-world applications within the European Union. We explain all data processing steps, starting with the data selection and requirement definition to the preparation of the final filtered data. We distinguish between curated data and web data, as each of these categories is handled by distinct pipelines, with curated data undergoing minimal filtering and web data requiring extensive filtering and deduplication. This distinction guided the development of specialized algorithmic solutions for both pipelines. In addition to describing the processing methodologies, we provide an in-depth analysis of the datasets, increasing transparency and alignment with European data regulations. Finally, we share key insights and challenges faced during the project, offering recommendations for future endeavors in large-scale multilingual data preparation for LLMs.

cs.CL↗

arXiv2024

Towards Multilingual LLM Evaluation for European Languages

The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and meaningful way across multiple European languages remains challenging, especially due to the scarcity of language-parallel multilingual benchmarks. We introduce a multilingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages. Our contributions include examining the effectiveness of translated benchmarks, assessing the impact of different translation services, and offering a multilingual evaluation framework for LLMs that includes newly created datasets: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, and EU20-GSM8K. The benchmarks and results are made publicly available to encourage further research in multilingual LLM evaluation.

cs.CL↗

arXiv2024

Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs

We present two multilingual LLMs, Teuken 7B-base and Teuken 7B-instruct, designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. Trained on a dataset comprising around 60% non-English data and utilizing a custom multilingual tokenizer, our models address the limitations of existing LLMs that predominantly focus on English or a few high-resource languages. We detail the models' development principles, i.e., data composition, tokenizer optimization, and training methodologies. The models demonstrate strong performance across multilingual benchmarks, as evidenced by their performance on European versions of ARC, HellaSwag, and TruthfulQA.

cs.CL↗

arXiv2024

Do Multilingual Large Language Models Mitigate Stereotype Bias?

While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training on bias mitigation, is lacking. This study addresses this gap by systematically training six LLMs of identical size (2.6B parameters) and architecture: five monolingual models (English, German, French, Italian, and Spanish) and one multilingual model trained on an equal distribution of data across these languages, all using publicly available data. To ensure robust evaluation, standard bias benchmarks were automatically translated into the five target languages and verified for both translation quality and bias preservation by human annotators. Our results consistently demonstrate that multilingual training effectively mitigates bias. Moreover, we observe that multilingual models achieve not only lower bias but also superior prediction accuracy when compared to monolingual models with the same amount of training data, model architecture, and size.

cs.CL↗

arXiv2024

Quantum computational resources for lattice QCD in the strong-coupling limit

We consider the strong coupling limit of lattice QCD with massless staggered quarks and study the resource requirements for quantum simulating the theory in its Hamiltonian formulation. The bosonic Hilbert space of the color-singlet degrees of freedom grows quickly with the number of quark flavors, making it a suitable testing ground for resource considerations across different platforms. In particular, in addition to the standard model of computation with qubits, we consider mapping the theory to qudits $(d>2)$ and qumodes, as used on trapped-ion systems and photonic devices, respectively.

hep-lat↗

arXiv2024

Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?

The adaption of multilingual pre-trained LLMs into eloquent and helpful assistants is essential to facilitate their use across different language regions. In that spirit, we are the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages. We systematically examine the effects of language and instruction dataset size on a mid-sized and a large, multilingual LLMs by instruction-tuning them on parallel instruction-tuning datasets. Our results demonstrate that instruction-tuning on parallel instead of monolingual corpora benefits cross-lingual instruction following capabilities by up to 9.9%. Furthermore, we show that the Superficial Alignment Hypothesis does not hold in general, as the investigated multilingual 7B parameter model presents a counter-example requiring large-scale instruction-tuning datasets. Finally, we conduct a human annotation study to understand the alignment between human-based and GPT-4-based evaluation within multilingual chat scenarios.

cs.CL↗

arXiv2023

Tokenizer Choice For LLM Training: Negligible or Crucial?

The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. Shedding light on this underexplored area, we conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating different tokenizer algorithms and parameterizations. Our studies highlight that the tokenizer choice can significantly impact the model's downstream performance and training costs. In particular, we find that the common tokenizer evaluation metrics fertility and parity are not always predictive of model downstream performance, rendering these metrics a questionable proxy for the model's downstream performance. Furthermore, we show that multilingual tokenizers trained on the five most frequent European languages require vocabulary size increases of factor three in comparison to English. While English-centric tokenizers have been applied to the training of multi-lingual LLMs in the past, we find that this approach results in a severe downstream performance degradation and additional training costs of up to 68%, due to an inefficient tokenization vocabulary.

cs.LG↗

arXiv2023

Quantum Gate Sets for Lattice QCD in the strong coupling limit: $N_f=1$

We derive the primitive quantum gate sets to simulate lattice quantum chromodynamics (LQCD) in the strong-coupling limit with one flavor of massless staggered quarks. This theory is of interest for studies at non-zero density as the sign problem can be overcome using Monte Carlo methods. In this work, we use it as a testing ground for quantum simulations. The key point is that no truncation of the bosonic Hilbert space is necessary as the theory is formulated in terms of color-singlet degrees of freedom (``baryons'' and ``mesons''). The baryons become static in the limit of continuous time and decouple, whereas the dynamics of the mesonic theory involves two qubits per lattice site. Lending dynamics also to the ``baryons'' simply requires to use the derived gate set in its controlled version.

hep-lat↗

arXiv2023

Simulating $\mathbb{Z}_2$ Lattice Gauge Theory with the Variational Quantum Thermalizer

The properties of strongly-coupled lattice gauge theories at finite density as well as in real time have largely eluded first-principles studies on the lattice. This is due to the failure of importance sampling for systems with a complex action. An alternative to evade the sign problem is quantum simulation. Although still in its infancy, a lot of progress has been made in devising algorithms to address these problems. In particular, recent efforts have addressed the question of how to produce thermal Gibbs states on a quantum computer. In this study, we apply a variational quantum algorithm to a low-dimensional model which has a local abelian gauge symmetry. We demonstrate how this approach can be applied to obtain information regarding the phase diagram as well as unequal-time correlation functions at non-zero temperature.

hep-lat↗

arXiv2022

Dihedral Lattice Gauge Theories on a Quantum Annealer

We study lattice gauge theory with discrete, non-Abelian gauge groups. We extend the formalism of previous studies on D-Wave's quantum annealer as a computing platform to finite, simply reducible gauge groups. As an example, we use the dihedral group $D_n$ with $n=3,4$ on a two plaquette ladder for which we provide proof-of-principle calculations of the ground-state and employ the known time evolution formalism with Feynman clock states.

hep-lat↗

arXiv2022

Towards a Holistic View on Argument Quality Prediction

Argumentation is one of society's foundational pillars, and, sparked by advances in NLP and the vast availability of text data, automated mining of arguments receives increasing attention. A decisive property of arguments is their strength or quality. While there are works on the automated estimation of argument strength, their scope is narrow: they focus on isolated datasets and neglect the interactions with related argument mining tasks, such as argument identification, evidence detection, or emotional appeal. In this work, we close this gap by approaching argument quality estimation from multiple different angles: Grounded on rich results from thorough empirical evaluations, we assess the generalization capabilities of argument quality estimation across diverse domains, the interplay with related argument mining tasks, and the impact of emotions on perceived argument strength. We find that generalization depends on a sufficient representation of different domains in the training part. In zero-shot transfer and multi-task experiments, we reveal that argument quality is among the more challenging tasks but can improve others. Finally, we show that emotions play a minor role in argument quality than is often assumed.

cs.CL↗

arXiv2021

Conifer Seedling Detection in UAV-Imagery with RGB-Depth Information

Monitoring of reforestation is currently being considerably streamlined through the use of drones and image recognition algorithms, which have already proven to be effective on colour imagery. In addition to colour imagery, elevation data is often also available. The primary aim of this work was to improve the performance of the faster-RCNN object detection algorithm by integrating this height information, which showed itself to notably improve performance. Interestingly, the structure of the network played a key role, with direct addition of the height information as a fourth image channel showing no improvement, while integration after the backbone network and before the region proposal network led to marked improvements. This effect persisted with very long training regimes. Increasing the resolution of this height information also showed little effect.

cs.CV↗

arXiv2021

Active Learning for Argument Strength Estimation

High-quality arguments are an essential part of decision-making. Automatically predicting the quality of an argument is a complex task that recently got much attention in argument mining. However, the annotation effort for this task is exceptionally high. Therefore, we test uncertainty-based active learning (AL) methods on two popular argument-strength data sets to estimate whether sample-efficient learning can be enabled. Our extensive empirical evaluation shows that uncertainty-based acquisition functions can not surpass the accuracy reached with the random acquisition on these data sets.

cs.LG↗