Speech Technology

Quite an obvious but so ignored by industry before. There is certainly no need to clone from 3 seconds https://www.linkedin.com/posts/soniox_soniox-texttospeech-voiceai-activity-7501594413660401664-p9Cq

https://www.nature.com/articles/srep12881 Human starts to plan answer 2 seconds before question end

We know human scores are useless but anyway https://x.com/datapointai/status/2094829412625654141 today, we're releasing the largest open-source human audio preferences dataset, focused on the customer support use-case - 300K+ annotations by real people - 15 SOTA TTS models ranked (Sonic 3.6, Grok TTS, Simba 3.2, Eleven Labs v3) - 8 categories (IVR menus, empathy, escalations, refunds etc) dataset + benchmark + frontier plot below:

You can download really huge datasets these days https://x.com/ahochlehnert/status/2092648676829413778 LAION-BVD: a 10-million-hour open video dataset for multimodal pre-training. - 1.3B video URLs from CommonCrawl - 80M downloaded videos - 10M video hours - 55M captioned clips - 300M frame-caption pairs This repository contains 1.7 million audio clips taken from BVD-V-55M and sampled for uniqueness of the source video, so that the subset maximises source diversity rather than clip count. Each clip comes with a caption, its language, and the timestamps locating it in the source video. https://huggingface.co/datasets/laion/BVD-A-1.7M

https://huggingface.co/BreezeBlue/Breeze-TTS-2 https://breezeblue.ai/breeze-tts-2 English/Chinese only but really good quality

Kytai released pockettts training code https://github.com/kyutai-labs/pocket-tts/tree/main/training https://x.com/kyutai_labs/status/2092254286772080768

https://www.hume.ai/blog/measuring-benchmark-optimization-in-speech-recognition https://x.com/hume_ai/status/2090813428793319933

Somehow they forgot about leaderboard

Audio in the wild is still very complex https://aslp-lab.github.io/SmartGlasses/ https://x.com/MosiAI_Official/status/2090775763666812997 MOSS takes 1st place across all four tasks in the IEEE SLT 2026 SmartGlasses Challenge. The challenge tests whether AI can truly hear and understand real conversations through smart glasses — handling noise, overlapping speech, multiple speakers, and long-context interactions. Among 77 participating teams, MOSS achieved 1st place across both tracks and all four tasks, covering speech recognition and spoken language understanding in real-world smart glasses scenarios. Results: Two-person TSA-ASR — 5.23% tcpCER Two-person SLU — 88.8% Accuracy Multi-party TSA-ASR — 27.95% tcpCER Multi-party SLU — 93.0% Accuracy The models behind these results: • MOSS-Transcribe-Diarize for TSA-ASR • MOSS-Audio for SLU

Some modern tech for speech processing on earbuds, interesting talk overall https://www.youtube.com/watch?v=zeDkT8EuKao

Alex Smola leading a great research at Boson.AI (Higgs authors). Couple of recent selected papers: https://arxiv.org/search/cs?searchtype=author&query=Smola,+A https://arxiv.org/abs/2603.25727 Back to Basics: Revisiting ASR in the Age of Voice Agents Geeyang Tay, Wentao Ma, Jaewon Lee, Yuzhi Tang, Daniel Lee, Weisu Yin, Dongming Shen, Silin Meng, Yi Zhu, Mu Li, Alex Smola

Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of degradation. We introduce WildASR, a multilingual (four-language) diagnostic benchmark sourced entirely from real human speech that factorizes ASR robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. Evaluating seven widely used ASR systems, we find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions. Critically, models often hallucinate plausible but unspoken content under partial or degraded inputs, creating concrete safety risks for downstream agent behavior. Our results demonstrate that targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production systems. Besides the benchmark itself, we also present three analytical tools that practitioners can use to guide deployment decisions.

https://arxiv.org/abs/2607.20460 Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions? Yuzhi Tang, Wentao Ma, Xiling Zhao, Ahmad Salimi, Sepehr Harfi Moridani, Dongming Shen, Jixuan Wang, Abdulrahman Abdulrazzag, Murdock Aubry, Yu-Hua Chen, Daniel Lee, Jaewon Lee, Jonah Mackey, Silin Meng, Nicholas Stranges, Chenxu Xiong, Hao Yu, Yi Zhu, Mu Li, Alex Smola

Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed. This is critical for real-world deployment, where conversational policies vary across applications (e.g., proactive tutoring vs. passive counseling). We introduce Instruct-FD, an instruction-conditioned benchmark for evaluating controllable turn management in FD systems. To enable this, we develop a human-validated, scalable synthetic pipeline that generates instruction-conditioned conversations, along with a deployment-agnostic multi-turn evaluation protocol and an LLM-based judge. Benchmarking six state-of-the-art full-duplex systems reveals a substantial gap in instruction-following turn management: the best model achieves only 64.4% adherence. Performance is highly uneven across behaviors and scenarios, with proactive behaviors such as model backchanneling and interruption remaining particularly challenging. These findings establish instruction-following turn management as a crucial direction for building adaptable and deployable full-duplex dialogue systems.

When full to end doesn't really work this thing has potential https://arxiv.org/abs/2608.13831 VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor, Shehzeen Hussain, Viacheslav Klimkov, Valentin Mendelev, Mikyas Desta, Paarth Neekhara, Piotr Zelasko, Chen Chen, Elena Rastorgueva, Ke Hu, Ankita Pasad, Xuesong Yang, Aya Alja'fari, Rajarshi Roy, Rohan Badlani, Jason Roche, Jason Li, Zhehuai Chen

Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.

https://github.com/anyreachai/dualturn DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining Shangeth Rajaa

Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.

❇️ dots.tts Fully Open-Sourced RedNote’s Dots Studio team has fully open-sourced dots.tts, a 2B-parameter, fully continuous, end-to-end autoregressive text-to-speech foundation model.   Unlike mainstream TTS systems that generate discrete acoustic tokens, dots.tts models speech directly in a continuous latent space, bringing together high-fidelity zero-shot voice cloning, multilingual speech synthesis, expressive generation, and real-time streaming. 🌟 Key Highlights: 🌊 Fully continuous autoregressive architecture dots.tts removes discrete acoustic tokens from the speech-generation pipeline. It combines a 48 kHz AudioVAE, a semantic encoder, an LLM, and an autoregressive flow-matching acoustic head to generate speech one continuous latent patch at a time. 🧠 Improved long-range consistency Full-history conditioning allows the acoustic head to use the complete generated prefix, while reward-free self-corrective post-training exposes the model to its own inference-time errors. Together, these designs help reduce drift and improve robustness during long autoregressive generation. ⚡️ Ultra-low-latency dual streaming CFG-aware MeanFlow distillation reduces acoustic generation to only 2–4 function evaluations, achieving first-packet latencies of 85 ms in output-streaming mode and 54 ms in dual-streaming mode, making the model suitable for real-time conversational applications. 🌍 Strong multilingual voice cloning and expressiveness Trained on 1.5 million hours of multilingual speech, dots.tts has been evaluated across 24 languages and demonstrates strong multilingual and cross-lingual voice cloning, as well as expressive speech generation. 🏆 Open-source state-of-the-art performance dots.tts achieves strong results on major TTS benchmarks. The release includes pretrained, post-trained SOAR, and MeanFlow-distilled checkpoints, together with training, inference, and fine-tuning code under the Apache 2.0 license. 🚀 Get started: 👉 GitHub: https://lnkd.in/ecSN2-t8 👉 Hugging Face Models: https://lnkd.in/euBvdAcv 👉 Online Demo: https://lnkd.in/egcVMBfw 👉 Paper: https://lnkd.in/eqHWx-gx

https://github.com/xiaomi-research/midashenglm-gen https://arxiv.org/abs/2608.11804 MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching Xingwei Sun, Heinrich Dinkel, Gang Li, Jiahao Mei, Yadong Niu, Zerui Han, Yuepeng Jiang, Jiahao Zhou, Lichun Fan, Jian Luan

Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at this https URL and this https URL, and the demo page is available at this https URL.

One more agentic thing https://github.com/InteractiveASR/AgenticASR https://interactiveasr.github.io/ https://arxiv.org/abs/2604.09121 Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition Peng Wang, Yanqiao Zhu, Zixuan Jiang, Qinyuan Chen, Xingjian Zhao, Xipeng Qiu, Wupeng Wang, Zhifu Gao, Xiangang Li, Kai Yu, Xie Chen

Recent years have witnessed remarkable progress in automatic speech recognition (ASR), driven by advances in model architectures and large-scale training data. However, two important aspects remain underexplored. First, Word Error Rate (WER), the dominant evaluation metric for decades, treats all words equally and often fails to reflect the semantic correctness of an utterance at the sentence level. Second, interactive correction-an essential component of human communication-has rarely been systematically studied in ASR research. In this paper, we integrate these two perspectives under an agentic framework for interactive ASR. We propose leveraging LLM-as-a-Judge as a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy. Furthermore, we design an LLM-driven agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs through semantic feedback. Extensive experiments are conducted on standard benchmarks, including GigaSpeech (English), WenetSpeech (Chinese), the ASRU 2019 code-switching test set. Both objective and subjective evaluations demonstrate the effectiveness of the proposed framework in improving semantic fidelity and interactive correction capability. We will release the code to facilitate future research in interactive and agentic ASR.

Big paper on state of the art in speech translation (87 pages) Speech Translation and Metrics in 2026: Findings of the IWSLT Campaign https://aclanthology.org/2026.iwslt-1.39.pdf This paper reports on the outcomes of the shared tasks organized as part of the 23rd International Workshop on Spoken Language Translation (IWSLT). The workshop covered ten major challenges in spoken language translation, including speech-to-text translation for both high-resource and low-resource language pairs, customized speech translation, speech generation, instruction-following speech processing, and the evaluation of speech translation systems. The shared tasks received strong participation, with more than 30 teams submitting runs. This year’s edition broadened the range of tasks, placing particular emphasis on speech generation and evaluation metrics.

Duplex paper from today https://github.com/duplexgen/duplexgen-code https://arxiv.org/abs/2607.26178

A multilingual joint diarization and ASR benchmark for Indian languages, spanning all 22 scheduled languages of India with approximately 108 hours of natural multi-speaker audio. https://huggingface.co/datasets/sarvamai/indic-diarbench