arXiv Science⌕ Search

arXiv · 2611.10144

Dataset Collections: Challenges of Large-Scale Data Aggregation in 3D Medical Image Datasets

Abstract

The development of deep learning-based medical AI depends on high-quality annotated data. Yet, dataset creation remains constrained by the need for expert radiologist annotations, limiting curated datasets to relatively small cohorts. To overcome this limitation, recent efforts increasingly aggregate disparate public datasets into large-scale dataset collections. However, combining datasets at scale often obscures data provenance, introducing substantial risks of reuse propagation, unintended leakage, bias amplification and distorted data distributions. In this work, we study these problems on a massive collection of 565k 3D radiological images across 1034 public datasets and provide practical resources to address them. Firstly, we formalize dataset collections as a distinct paradigm in medical image analysis and characterize their structural challenges, including reuse propagation, duplication, and provenance fragmentation. Secondly, we perform a large-scale empirical study of duplication on this collection and demonstrate extensive exact and near-duplicate reuse across public datasets via perceptual hashing and manual tracing, identifying a lower bound of 118k duplicate pairs. We make the perceptual hashes publicly accessible for comparison through our website, providing researchers with a standardized reference for identifying potential overlaps and shared provenance. Thirdly, since hashing alone cannot detect all duplicates, we introduce MAP (Metadata for Aggregation and Provenance), a lightweight, portable metadata profile for documenting provenance, known overlaps, and deduplication decisions in dataset collections. Our contributions help dataset collections shift from coarse aggregations to reliable and provenance-aware data scaling mechanisms in medical image analysis. A web tool supporting our work is available at https://mic-dct.dkfz.de/.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yannick Kirchhoff, Saikat Roy, Elisa Stegmeier, Hamideh Haghiri, Constantin Ulrich, Maximilian Rouven Rokuss, Tassilo Wald, Karol Gotkowski, Benjamin Hamm, Raphael Stock, Nico Disch, Michael Baumgartner, Dimitrios Bounias, Anand Deshpande, Stefan Dinkelacker, Stefan Dvoretskii, Katharina Eckstein, Selen Erkan, Jessica Kächele, Kim-Celine Kahl, Balint Kovacs, Lucas Kulla, Moritz Langenberg, Philipp Schader, Stephen Schaumann, Dasha Trofimova, Shuhan Xiao, Sebastian Ziegler, David Zimmerer, Marco Nolden, Fabian Isensee, Klaus Maier-Hein, Ralf Floca. 2026-11-06. Dataset Collections: Challenges of Large-Scale Data Aggregation in 3D Medical Image Datasets. https://arxiv.org/abs/2611.10144

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

A Cytology Dataset for Early Detection of Oral Squamous Cell Carcinoma

Oral squamous cell carcinoma OSCC is a major global health burden, particularly in several regions across Asia, Africa, and South America, where it accounts for a significant proportion of cancer cases. Early detection dramatically improves outcomes, with stage I cancers achieving up to 90 percent survival. However, traditional diagnosis based on histopathology has limited accessibility in low-resource settings because it is invasive, resource-intensive, and reliant on expert pathologists. On the other hand, oral cytology of brush biopsy offers a minimally invasive and lower cost alternative, provided that the remaining challenges, inter observer variability and unavailability of expert pathologists can be addressed using artificial intelligence. Development and validation of robust AI solutions requires access to large, labeled, and multi-source datasets to train high capacity models that generalize across domain shifts. We introduce the first large and multicenter oral cytology dataset, comprising annotated slides stained with Papanicolaou(PAP) and May-Grunwald-Giemsa(MGG) protocols, collected from ten tertiary medical centers in India. The dataset is labeled and annotated by expert pathologists for cellular anomaly classification and detection, is designed to advance AI driven diagnostic methods. By filling the gap in publicly available oral cytology datasets, this resource aims to enhance automated detection, reduce diagnostic errors, and improve early OSCC diagnosis in resource-constrained settings, ultimately contributing to reduced mortality and better patient outcomes worldwide.

eess.IV↗

StreamAZ: Line-by-Line SAR Focusing with State Space Models

Conventional focusing methods for Synthetic Aperture Radar (SAR) employ block-wise processing efficiently but remain latency-heavy processes that prevent the realisation of a closed-loop cognitive SAR system. We present an online SAR image-formation framework, StreamAZ, that treats sensing as a stream and produces focused images in a line by line fashion. Our method uses a tiny state-space surrogate model trained with teacher-student distillation and multi-stage losses. We evaluate the method on real SAR data from Maya4, a Sentinel-1-derived dataset containing raw, range-compressed, range-cell-migration-corrected, and azimuth-compressed products. Relative to a linewise digital-signal-processing baseline, our framework delivers approximately 70$\times$ lower latency and 130$\times$ lower memory use; on a single AMD CPU core it processes one row in 16 ms with a memory footprint of 6 MB whilst maintaining a focusing quality high enough to support downstream decisions, which we illustrate with vessel detection and flood-mapping tasks.

eess.IV↗

Decoder-Guided Lossy Contour Coding Via Anchor Refinement

Object contours serve as compact structural priors for many receiver-side vision tasks such as image super-resolution, edge-conditioned generation, and machine vision. When such tasks are deployed over a bandwidth-limited channel, the sender transmits the high-quality object contour as structural side information to guide reconstruction at the receiver, while-to save bandwidth-only a low-quality reference such as a downsampled image or base-layer reconstruction is delivered. As a result, the decoder can already extract a coarse contour from this reference at no transmission cost, creating an encoder-decoder asymmetry: the fine contour must be coded and sent, yet a free coarse version is available at the decoder. This asymmetry is ignored by existing contour codecs such as JBIG2 and chain coding, which are lossless, symmetric, and offer no rate-distortion control, leading to high bitrates. In this paper, we propose a coarse-to-fine contour coding framework that models a high-quality contour as a structured geometric refinement of the decoder-available coarse contour. The encoder extracts ordered anchors along the fine contour and performs adaptive anchor skipping under a distortion constraint. The decoder then reconstructs the contour by using the coarse prior to guide anchor connectivity. This formulation enables lossy contour compression with an explicit rate-distortion trade-off. Experiments show 54.5%-66.9% bitrate reduction over methods without decoder-side guidance, and up to 5 times savings over JBIG2, while preserving high geometric accuracy.

eess.IV↗