arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2110.00275

SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Polyphonic Sound Event Localization and Detection

Thi Ngoc Tho Nguyen·Karn N. Watcharasupat·Ngoc Khanh Nguyen·Douglas L. Jones·Woon-Seng Gan
View original source ↗Read source document ↗Export metadata ↓

Abstract

Sound event localization and detection (SELD) consists of two subtasks, which are sound event detection and direction-of-arrival estimation. While sound event detection mainly relies on time-frequency patterns to distinguish different sound classes, direction-of-arrival estimation uses amplitude and/or phase differences between microphones to estimate source directions. As a result, it is often difficult to jointly optimize these two subtasks. We propose a novel feature called Spatial cue-Augmented Log-SpectrogrAm (SALSA) with exact time-frequency mapping between the signal power and the source directional cues, which is crucial for resolving overlapping sound sources. The SALSA feature consists of multichannel log-spectrograms stacked along with the normalized principal eigenvector of the spatial covariance matrix at each corresponding time-frequency bin. Depending on the microphone array format, the principal eigenvector can be normalized differently to extract amplitude and/or phase differences between the microphones. As a result, SALSA features are applicable for different microphone array formats such as first-order ambisonics (FOA) and multichannel microphone array (MIC). Experimental results on the TAU-NIGENS Spatial Sound Events 2021 dataset with directional interferences showed that SALSA features outperformed other state-of-the-art features. Specifically, the use of SALSA features in the FOA format increased the F1 score and localization recall by 6% each, compared to the multichannel log-mel spectrograms with intensity vectors. For the MIC format, using SALSA features increased F1 score and localization recall by 16% and 7%, respectively, compared to using multichannel log-mel spectrograms with generalized cross-correlation spectra.

Explore related subjects

eess.AScs.SD

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Thi Ngoc Tho Nguyen, Karn N. Watcharasupat, Ngoc Khanh Nguyen, Douglas L. Jones, Woon-Seng Gan. 2022-06-06. SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Polyphonic Sound Event Localization and Detection. https://doi.org/10.1109/taslp.2022.3173054

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

LLMs and Speech: Integration vs. Combination

In this work, we study different approaches to utilize large language models (LLMs) for automatic speech recognition (ASR). Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion and provide ablations on the effect of different label units and LLM sizes. For tight integration, we further examine the effect of attention interfaces, encoder downsampling, and length normalization. Furthermore, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations. We train and evaluate on LibriSpeech and Loquacious and additionally evaluate on the HuggingFace ASR leaderboard. Across model sizes, we find that shallow fusion consistently outperforms tight integration of AM and LLM on in-domain data, highlighting the importance of strong shallow-fusion baselines when evaluating speech LLMs for ASR. On the more heterogeneous HuggingFace ASR leaderboard, however, the integrated prefix LLM achieves lower average WER than shallow fusion, with gains concentrated on out-of-domain corpora.

eess.AS↗
arXiv2026

Vaani Benchmark V1.0: An Inclusive Multimodal Benchmark Dataset for Hindi

Benchmarking is critical for the systematic evaluation of machine learning systems. While several open-source datasets are available for Hindi, existing benchmarks remain limited in terms of modality, geographic diversity, demographic representation, and transcription robustness. We introduce an inclusive, multimodal Hindi benchmark dataset collected from 102 districts across India. The dataset consists of spontaneous speech elicited using image prompts and recorded under real-world acoustic conditions across diverse demographic groups. Each audio segment is associated with the image prompt that elicited it and is annotated with three independent transcriptions, enabling multi-reference evaluation that accounts for permissible orthographic and lexical variations. We show that single-reference evaluation overstates errors, while multi-reference evaluation provides a more robust, inclusive, and realistic assessment of automatic speech recognition (ASR) systems. The pairing of each utterance with its eliciting image further enables image retrieval evaluation using both speech and text queries. We evaluate multiple multimodal embedding models for image retrieval and report their performance on the combined speech--image and text--image retrieval tasks. The results reveal a performance gap between text-based and audio-based retrieval, with text-based retrieval consistently achieving higher performance.

eess.AS↗
arXiv2026

Cover First, Disagree Softly: Rethinking Mismatch-First Active Learning for Frame-Level Audio Classification

Sound event detection relies on frame-level strong labels whose annotation is expensive. Active learning addresses this problem by selecting the audio segments whose labels help the classifier most. One of the prevailing acquisition strategies for this task, mismatch-first farthest-traversal (MFFT), combines the disagreement between two classifiers and the diversity of the selected segments through hard sequential decisions. It selects whole groups of high-disagreement segments first and spreads only the remaining budget by farthest traversal. On two multi-label datasets we show that this design is blind to the similarity among the selected segments and fails under low budgets, with every mismatch-first variant ending below the plain geometric strategy it builds on. We propose mismatch-weighted facility location (MW-FL), which spends the entire budget through a disagreement-weighted coverage objective that penalizes similarity among the selected segments. The disagreement signal from MFFT is used to obtain the nonnegative weights of this facility-location objective, using fixed smoothing without dataset-specific tuning. Experiments across two geometric mechanisms with three ways of using disagreement show that coverage of the selected segments is the dominant factor, hard disagreement gating of selection is harmful on both mechanisms, and soft disagreement weighting helps on top of coverage. MW-FL attains the best area under the learning curve on both datasets.

eess.AS↗