arXiv Science⌕ Search

arXiv · 2611.13814

Swanada: Split-Encoder Capability-Routed Conformer CTC and Sub-30 MB On-Device Speech Recognition Across Twelve Scripts

Abstract

Client-side automatic speech recognition (ASR) across Indic and Classical scripts eliminates cloud transcription costs, network tail latency, and server-side audio retention, yet neural exports face four systems bottlenecks in browsers and mobile runtimes: (1) monolithic per-language Conformer checkpoints require ~177 MiB per script (> 2.1 GiB across 12 scripts), exhausting CacheStorage quotas; (2) static relative positional encoding tables embed 10.2-20.5 MB of FP32 constants while 1x1 Conv1d layers trigger slow or unsupported ConvInteger kernels in WebAssembly SIMD; (3) sliding-window CTC flickers on boundary tokens; and (4) mobile WebAudio pipelines suffer 44.1/48 kHz sample-rate mismatches and silent AudioWorklet stalls under iOS WKWebView. We present Swanada, an open-source 16 kHz streaming ASR system spanning Classical Sanskrit, Indian English, and ten Indian languages (sa, en, hi, kn, mr, bn, ta, te, ml, gu, pa, or) across browser WebAssembly, Node.js, and Python. Swanada pairs capability-routed shared INT8 Conformer encoders (a 17-layer 177.38 MiB Studio tier at $d = 512$ for desktop/servers and a distilled 14-layer 25.35 MiB Mobile tier at $d = 256$ for phones) with twelve hot-swappable 515.0 KiB native-script linear CTC heads ($257 \times 512$), cutting 12-language storage by $11.6\times$ (183.4 MiB) and $67.8\times$ (31.39 MiB; 25.85 MiB first load). Runtime protobuf tail surgery exposes the shared representation in $< 2$ ms, while dynamic positional synthesis, MatMul pointwise convolutions, and post-quantization QInt8 head calibration preserve sub-30 MB accuracy. Across $N = 480$ held-out and $N = 1,280$ open public test clips (FLEURS, LibriSpeech, GLOBE, AMI, wild Sanskrit), Swanada achieves 5.43% held-out and 8.87% 12-language open CER on Studio (18.56% and 19.23% on Mobile) at $0.0097\times$-$0.0264\times$ RTF, with 30.6 ms median TTFT and 0.00% committed-prefix churn.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Hemanth HM. 2026-11-10. Swanada: Split-Encoder Capability-Routed Conformer CTC and Sub-30 MB On-Device Speech Recognition Across Twelve Scripts. https://arxiv.science/abs/2611.13814

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Structure Across Voices: Comparing acoustic-event type accumulation and sequence dependence across four vocal corpora using frozen audio encoders

Comparing acoustic structure across very different vocal corpora is difficult because the recordings use different native event units, contain different amounts of sequence, and were collected under different conditions. We use the high-dimensional representations of frozen, pretrained audio encoders to measure acoustic diversity and temporal organization. Matched sampling gives each corpus the same opportunities to observe new acoustic forms and relationships between events, without changing its native event definition. We compare sperm-whale codas, human speech phones, Bengalese-finch syllables, and common-marmoset calls using 1,952 events and 1,700 adjacent pairs per comparison. Whale shows the fastest accumulation of encoder-defined acoustic groups, while Finch has the strongest repeated-sequence recurrence against ordinary within-block shuffles. Their neighboring-event dependence is closely matched under a source-and-position control. A second preceding event changes held-out prediction more favorably for Whale than for the other corpora, although Whale's standalone improvement remains uncertain. Waveform measurements recover several differences, but duration and acquisition-sensitive properties also contribute. Together, the findings show how pretrained representations can measure distinct forms of vocal organization, while waveform and source controls define the scope of the comparison.

cs.SD↗

Music of the Deep: An acoustic blueprint of sperm whale codas

Sperm-whale codas are usually described by click count and spacing. Earlier work found additional structure in click waveforms and across coda sequences. We ask how these two levels compare with performed music. We examine 1,952 codas alongside violin performances, drum grooves, and sung melodies. Using frozen audio encoders, we describe constituent sounds and complete units separately, comparing composition, order, and sequence context. In codas, click composition improves prediction of whole-coda acoustic form beyond timing. Click-form order carries additional conditional information, and rearranging click spacing changes the acoustic representations across six encoders. Knowing the second preceding coda improves held-out prediction beyond knowing only the last one. The musical references show different combinations of these relationships, so similarities across sequences do not imply similar organization within units. We then examine acoustic properties that sequence measurements cannot capture. Codas span broad, reproducible ranges along dimensions validated using human emotion ratings. WhAM-generated clips retain measurable acoustic differences from matched natural codas. Together, these findings show why a coda cannot be described by timing, constituents, or sequence context alone. They motivate an acoustic blueprint that brings these measurements together and provides a reference for comparing natural and generated sound.

cs.SD↗

High-Fidelity Binarized Neural Vocoder

Neural vocoders have achieved remarkable progress in high-fidelity waveform generation in recent years, but most existing methods still rely on full-precision operators, leading to non-negligible storage and computational costs during inference. Binary neural networks (BNNs) provide an alternative direction by quantizing weights and activations to 1 bit. However, directly applying binarization to neural vocoders is rather challenging as it inherently incurs severe information loss. To this end, we propose BiVoC, a BNN-oriented time-frequency neural vocoder for high-fidelity waveform generation. Building on our previous full-precision RNDVoC framework, BiVoC adapts the inverse-problem formulation, range-null decomposition, and fineto-coarse non-uniform subband modeling to the 1-bit setting. Specifically, we develop a simple binarized convolutional unit to provide stable time-frequency feature modeling under binary representations. Furthermore, we analyze the quality-complexity sensitivity of different modules and propose a mixed-precision design to retain quality-sensitive components in full precision while binarizing computation-dominant parts. Extensive experiments are conducted on LibriTTS and multiple out-of-distribution benchmarks. Both objective and subjective results demonstrate that BiVoC substantially outperforms BNN variants obtained by directly binarizing existing vocoders, while achieving competitive performance against several mainstream full-precision vocoder baselines under much lower theoretical computational cost. These results demonstrate the promising potential of BNNs in the audio generation field. Codes and pre-trained weights are available at https://github.com/Andong-Li-speech/BiVoC.

cs.SD↗