Speech Technology

Topic of today - autoregression vs diffusion in TTS First of all the presentation from Meta. Claims diffusion/autoregression decision could be dynamic https://www.youtube.com/watch?v=kMimQxIJLos Second paper from today on the similar topic https://arxiv.org/abs/2607.04140 DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech Junwon Moon, Yejin Lee, Seungbeom Kim, Hoseong Ahn, Sewoong Park, Heeseung Kim, Kyuhong Shim

Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness, since local errors propagate to later positions and can escalate into hallucination. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, since the model receives the full input text before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a 1/t-weighted training objective and a time-shifted inference schedule that together defer low-confidence positions to later steps. Trained on only 585 hours of LibriTTS, DELTA-TTS achieves a 1.75% WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens 3.3x faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates the hallucinations observed in AR generation.

Deepmind also releases something https://github.com/google-deepmind/phasecoder https://arxiv.org/abs/2601.21124 PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar

Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We demonstrate that Gemma 3n LLM can be fine-tuned to reason over "Spatial Audio Tokens" produced by PhaseCoder. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.

Piotr is a reincarnation of Lennart https://x.com/PiotrZelasko/status/2085709891604254788

There is a big interest in full duplex as I see, here is a nice collection of papers https://github.com/Ruiqi-Yan/Awesome-Full-Duplex-SDM

Things move on in openai as well. Interesting that voice model is separate. And no turn detector anymore.

One more https://github.com/AnXMuy/AgenticASR

https://huckiyang.github.io/voice-memory/ from NVIDIA https://arxiv.org/abs/2607.26410 Voice Memory for Agentic Speech Recognition Chao-Han Huck Yang, Zih-Ching Chen, Piotr Zelasko, Zhehuai Chen, Jagadeesh Balam, Boris Ginsburg

We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain this http URL and decides per utterance whether to act on the hypothesis or abstain and keep the 1-best. Asynchronously, a score-gated optimizer revises that file through bounded edits, accepting an edit only when it strictly improves a held-out score. Extended from classical ASR-LM framework, we refer this split the listener-thinker architecture; the two roles are coupled only through the memory, so no weights change and the learned skill stays auditable and portable. Restraint turns out to be the operative skill this loop discovers: unconstrained generative error correction (GER) over-corrects, breaking correct tokens on up to 64% of its edits on financial news, and Voice Memory, reduces this rate to 35%. Across ten HyPoradise domains with an open corrector, Voice Memory, lowers weighted word error rate from 8.36% to 7.52% (7.47% with three added in-context examples) without regressing any dataset below its 1-best baseline; gains concentrate where recoverable headroom is largest, including air-travel commands (8.40% to 3.40%) and noisy far-field speech (CHiME-4, 12.69% to 10.46%). The me

https://huggingface.co/nyralabs/CrisperWhisper2.0_large Most speech-to-text systems never actually decide whether to write down what was said or what was meant. They inherit that choice from their training data and apply it inconsistently. CrisperWhisper 2.0 makes it an explicit, controllable choice. One recording, two transcripts:

Verbatim, exactly what was said, in one consistent format: 

[um] so we we need to, to reschedule the th- thursday meeting to [uh] march third at nine thirty [laughter]

Intended, the clean version the speaker meant, with numbers, dates, and emails formatted the way you'd write them: 

So we need to reschedule the Thursday meeting to March 3 at 9:30. On top of that: Word-level timings. Around 30 ms mean boundary error on read speech and 41 ms on conversational speech, the most precise word timing of any system we benchmarked, on both. Verbatimize. Upgrade transcripts you already have: given audio plus a trusted clean transcript, the model reproduces your content word-for-word and inserts only the disfluencies and vocal events actually present in the audio (rare-word recall jumps from 6.8% to 96.1% vs. re-transcribing). This turns the world's abundant clean corpora into verbatim ones, ready for TTS data, clinical speech analysis, and dataset construction. Multilingual. Verbatim and intended modes work across most languages Whisper supports. CrisperWhisper 2.0 tops the Nyra Verbatim Speech Benchmark leaderboard for disfluency F1 across ten languages, ahead of every closed-source alternative we tested. Seamless longform. Audio of any length, transcribed without the usual chunk-boundary artifacts: each window continues from the words already transcribed (conditional continuation), so there are no duplicated or dropped words at the seams and no fragile timestamp-token bookkeeping. Production inference. A CTranslate2 runtime with speculative decoding and built-in mitigation of Whisper's looping-hallucination failure mode.

Some recent Uzbek things https://huggingface.co/datasets/k2speech/FeruzaSpeech - single speaker 40 hours TTS dataset https://huggingface.co/collections/navai-uz/navai-whisper-collection - recently trained Whisper models from Navai https://navai.pro https://huggingface.co/instinct-org/collections - some loosely organized data https://huggingface.co/datasets/OvozifyLabs/asr_evaluate_set - evaluation dataset with Telegram messages https://huggingface.co/datasets/openbank-uz/youtube_transcriptions - large autotranscribed dataset (300k rows, gemini transcribed) https://huggingface.co/uzinfocom-edu-ai/asr-uz-fastconformer-large - recently trained fastconformer (cv + issai + uzvoice + islomov) https://huggingface.co/Abduqayum/whisper-uzbek-medium-callcenter - recently trained whisper https://huggingface.co/datasets/Abduqayum/Uzbek-STT-Dataset-780h - dataset from the above, gemini transribed

Everyone builds self-improvement loops in LLMs, I wonder how they could look like in ASR/TTS. Not many publications on that yet.

We compared three LALM judges against a calibrated human panel across 15 dimensions of speech quality. The LALMs tracked humans closely on relevance, answer quality, and instruction following—what was said—but were much less reliable on naturalness, emotion, pronunciation, and overall feel—how it was said. https://research.withdavid.ai/blog/lalm-as-judge-vs-hitl

Interesting math on speech LLM https://arxiv.org/abs/2604.08003v1 Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs Yuan Xie, Jiaqi Song, Guang Qiu, Xianliang Wang, Ming Lei, Jie Gao, Jie Wu

Integrating large language models (LLMs) into automatic speech recognition (ASR) has become a dominant paradigm. Although recent LLM-based ASR models have shown promising performance on public benchmarks, it remains challenging to balance recognition quality with latency and overhead, while hallucinations further limit real-world deployment. In this study, we revisit LLM-based ASR from an entropy allocation perspective and introduce three metrics to characterize how training paradigms allocate entropy reduction between the speech encoder and the LLM. To remedy entropy-allocation inefficiencies in prevailing approaches, we propose a principled multi-stage training strategy grounded in capability-boundary awareness, optimizing parameter efficiency and hallucination robustness. Specifically, we redesign the pretraining strategy to alleviate the speech-text modality gap, and further introduce an iterative asynchronous SFT stage between alignment and joint SFT to preserve functional decoupling and constrain encoder representation drift. Experiments on Mandarin and English benchmarks show that our method achieves competitive performance with state-of-the-art models using only 2.3B parameters, while also effectively mitigating hallucinations through our decoupling-oriented design.

Interesting project https://github.com/Xiaobin-Rong/unipase UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations Xiaobin Rong, Zheng Wang, Yushi Wang, Jun Gao, Jing Lu

Universal speech enhancement (USE) aims to restore speech signals from diverse distortions across multiple sampling rates. We propose UniPASE, an extension of the low-hallucination PASE framework tailored for USE. At its core is DeWavLM-Omni, a unified representation-level enhancement module fine-tuned from WavLM via knowledge distillation on a large-scale supervised multi-distortion dataset. This module directly converts degraded waveforms into clean and linguistically faithful phonetic representations, ensuring robust enhancement with minimal linguistic hallucination. Based on these enhanced phonetic representations, an Adapter generates enhanced acoustic representations containing rich acoustic details, which a neural Vocoder uses to reconstruct corresponding high-fidelity 16-kHz waveforms. A PostNet then converts the waveforms to 48~kHz before resampling them to their original rates, enabling seamless handling of inputs and outputs at multiple sampling rates. Experimental results on several evaluation datasets, covering sub-tasks and full tasks, demonstrate that UniPASE achieves superior or competitive performance compared with existing state-of-the-art models. The proposed model also serves as the backbone of our submission to the URGENT 2026 Challenge, which achieved 1st place in the objective evaluation. The source code and audio demos are available at this https URL.

SGLang-Omni does serious job on optimizing speech models (Higgs TTS too) https://x.com/YichiZ03/status/2078588932191895976

Interesting codec https://huggingface.co/Scicom-intl/WideCodec

Outstanding paper on ACL 2026 https://github.com/HITsz-TMG/Lychee-FD Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLM

https://huggingface.co/AutoArk-AI/Audio8-ASR-0.1B Audio8-ASR-0.1B is a compact autoregressive ASR model whose language-model component has only 0.1B parameters. It supports multilingual speech recognition for languages including Chinese, English, French, German, Japanese, Korean, and Cantonese. We position it as one of the smallest usable performance ASR models in the LLM era. The audio encoder backbone is based on Qwen3-ASR-0.6B, with the audio adapter and projector trained as part of Audio8-ASR. The language-model backbone is based on Ref-Pretrain-Qwen-104M.

Part of the news on Inkling is that it actually handles audio https://x.com/huckiyang/status/2077625513384841679