Creating SquashFS Images to Support HPC and AI Datasets

Julie Mullen; Albert Reuther; LaToya Anderson; Michael Jones; William Arcand; William Bergeron; David Bestor; Alex Bonn; Daniel Burrill; Chansup Byun; Vijay Gadepally; Michael Houle; Matthew Hubbell; Hayden Jananthan; Piotr Luszczek; Peter Michaleas; Guillermo Morales; Andrew Prout; Antonio Rosa; Chuck Yee; Jeremy Kepner

arXiv · 2610.20578

Creating SquashFS Images to Support HPC and AI Datasets

View original source ↗Read source document ↗Export metadata ↓

Abstract

HPC users often download datasets on HPC systems for analysis and processing. When these datasets are comprised of hundreds of thousands (or more) of small files, accessing them in parallel can put a tremendous burden on the metadata servers of central parallel file systems. If the dataset is read-only, an effective solution is to compress the dataset into a SquashFS image. There are, however, a number of workflows that can be used to create such a dataset image, and we observed that some were more efficient than others. To provide more effective guidance for our userbase, we benchmarked eleven different workflows for creating SquashFS dataset image files of datasets with thousands to millions of very small files. We took into account whether the data arrived as tar.gz, zip, or files-and-directories downloads, whether the image creation was on the central file system or local compute node storage, and the number of SquashFS file compression threads used. This paper shares the results of these benchmarks and provides recommendations to HPC system users toward the best workflows to use for SquashFS image creation.

Explore related subjects

cs.IR

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Julie Mullen, Albert Reuther, LaToya Anderson, Michael Jones, William Arcand, William Bergeron, David Bestor, Alex Bonn, Daniel Burrill, Chansup Byun, Vijay Gadepally, Michael Houle, Matthew Hubbell, Hayden Jananthan, Piotr Luszczek, Peter Michaleas, Guillermo Morales, Andrew Prout, Antonio Rosa, Chuck Yee, Jeremy Kepner. 2026-10-14. Creating SquashFS Images to Support HPC and AI Datasets. https://arxiv.org/abs/2610.20578

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

EviRerank: Adaptive Evidence Construction for Long-Document LLM Reranking

Decoder-only LLM rerankers struggle with long documents: inference is costly and relevance signals can be diluted by irrelevant context. Motivated by an attention analysis indicating a consistent degradation trend when non-relevant text is appended, we propose EviRerank, an evidence-based long-document reranking framework for decoder-only LLMs. EviRerank (i) scores document blocks with a lightweight selector (BM25, bi-encoder, or cross-encoder), (ii) constructs a compact reranking context under a hard token cap by dynamically budgeting evidence blocks with Adaptive Evidence Budgeting (AEB) and adding a global summary cue via Summary Augmentation (SA), and (iii) reranks with a decoder-only LLM. Across TREC DL'19, DL'22, DL'23, and MLDR-zh, EviRerank consistently outperforms full-document LLM reranking and strong block-selection baselines while substantially reducing the required input length. On TREC DL'19, EviRerank reaches up to 0.744 nDCG@10 while substantially reducing reranking latency. Additional RankZephyr-7B experiments confirm transfer to listwise reranking under matched context budgets.

cs.IR↗

arXiv2026

On the Impact of Graph Neural Networks in Recommender Systems: A Topological Perspective

In recommender systems, user-item interactions can be modeled as a bipartite graph, where user and item nodes are connected by undirected edges. This graph-based view has motivated the rapid adoption of graph neural networks (GNNs), which often outperform collaborative filtering (CF) methods such as latent factor models, deep neural networks, and generative strategies. Yet, despite their empirical success, the reasons why GNNs offer systematic advantages over other CF approaches remain only partially understood. This monograph advances a topology-centered perspective on GNN-based recommendation. We argue that a comprehensive understanding of these models' performance should consider the structural properties of user-item graphs and their interaction with GNN architectural design. To support this view, we introduce a formal taxonomy that distills common modeling patterns across eleven representative GNN-based recommendation approaches and consolidates them into a unified conceptual pipeline. We further formalize thirteen classical and topological characteristics of recommendation datasets and reinterpret them through the lens of graph machine learning. Using these definitions, we analyze the considered GNN-based recommender architectures to assess how and to what extent they encode such properties. Building on this analysis, we derive an explanatory framework that links measurable dataset characteristics to model behavior and performance. Taken together, this monograph re-frames GNN-based recommendation through its topological underpinnings and outlines open theoretical, data-centric, and evaluation challenges for the next generation of topology-aware recommender systems.

cs.IR↗

arXiv2026

Relevance Supervision Can Teach Neural Retrievers to Prefer LLM-Generated Text

Recent studies show that neural retrievers often exhibit source bias, ranking LLM-generated passages above semantically similar human-written ones, which threatens the reliability of modern information access systems. While this bias is often viewed as a model-level property, we show that relevance supervision itself can induce it. Empirically, source bias is clearest in relevance-supervised retrievers, and the same encoders become more LLM-preferring after relevance fine-tuning. We trace this effect to surface-level artifacts in supervised retrieval data: annotated positives and negatives differ systematically in properties such as fluency and lexical specificity, and these differences parallel those between LLM-generated and human-written passages. As a result, retrievers can learn artifact-based shortcuts for relevance. We further show theoretically that, under contrastive learning, artifacts that reliably separate positives from negatives can be encoded into learned relevance scores. Motivated by this mechanism, we propose complementary strategies that reduce source bias during training and at inference: balancing artifact distributions in negative sampling and removing the artifact-aligned embedding component, respectively. Overall, our findings support a data-centric view of source bias: retrievers may prefer LLM-generated text not merely because of inherent model properties, but because relevance supervision teaches them to rely on surface-level artifacts.

cs.IR↗