Speech Technology

Good result https://arxiv.org/abs/2308.08294 The ID R&D VoxCeleb Speaker Recognition Challenge 2023 System Description Nikita Torgashov, Rostislav Makarov, Ivan Yakovlev, Pavel Malov, Andrei Balykin, Anton Okhotnikov This report describes ID R&D team submissions for Track 2 (open) to the VoxCeleb Speaker Recognition Challenge 2023 (VoxSRC-23). Our solution is based on the fusion of deep ResNets and self-supervised learning (SSL) based models trained on a mixture of a VoxCeleb2 dataset and a large version of a VoxTube dataset. The final submission to the Track 2 achieved the first place on the VoxSRC-23 public leaderboard with a minDCF(0.05) of 0.0762 and EER of 1.30%.

https://github.com/facebookresearcmetah/muavic https://arxiv.org/abs/2303.00628 A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation.

https://arxiv.org/abs/2307.08720 ivrit.ai: A Comprehensive Dataset of Hebrew Speech for AI Research and Development Yanir Marmor, Kinneret Misgav, Yair Lifshitz We introduce "this http URL", a comprehensive Hebrew speech dataset, addressing the distinct lack of extensive, high-quality resources for advancing Automated Speech Recognition (ASR) technology in Hebrew. With over 3,300 speech hours and a over a thousand diverse speakers, this http URL offers a substantial compilation of Hebrew speech across various contexts. It is delivered in three forms to cater to varying research needs: raw unprocessed audio; data post-Voice Activity Detection, and partially transcribed data. The dataset stands out for its legal accessibility, permitting use at no cost, thereby serving as a crucial resource for researchers, developers, and commercial entities. this http URL opens up numerous applications, offering vast potential to enhance AI capabilities in Hebrew. Future efforts aim to expand this http URL further, thereby advancing Hebrew's standing in AI research and technology. https://www.ivrit.ai/

Today, we #NICT open-sourced Hi-Fi-CAPTAIN corpus for accelerating speech synthesis research!! 1. 14K utts of one female and one male for English 2. 19K utts of one female and one male for Japanese. 3. ESPnet recipe for E2E-TTS JETS using the corpus https://ast-astrec.nict.go.jp/en/release/hi-fi-captain/ https://twitter.com/okamotocamera/status/1689538465984294912

https://twitter.com/LiuXub/status/1689311290513063937 https://audio-agi.github.io/Separate-Anything-You-Describe/

Nemo also recently released a big English model https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xxlarge

Interesting parts: Token-and-Duration Transducer (TDT) #6536 (2.8x speedup) Graph-RNN-T #6168 WildCard-RNN-T #6168 Confidence Ensembles for ASR Spellchecking ASR #6179 https://github.com/NVIDIA/NeMo/releases/tag/v1.20.0

K2 project from Google from the JSALT talk above https://github.com/google-research/last and the paper https://arxiv.org/abs/2304.13134 LAST: Scalable Lattice-Based Speech Modelling in JAX Ke Wu, Ehsan Variani, Tom Bagby, Michael Riley We introduce LAST, a LAttice-based Speech Transducer library in JAX. With an emphasis on flexibility, ease-of-use, and scalability, LAST implements differentiable weighted finite state automaton (WFSA) algorithms needed for training \& inference that scale to a large WFSA such as a recognition lattice over the entire utterance. Despite these WFSA algorithms being well-known in the literature, new challenges arise from performance characteristics of modern architectures, and from nuances in automatic differentiation. We describe a suite of generally applicable techniques employed in LAST to address these challenges, and demonstrate their effectiveness with benchmarks on TPUv3 and V100 GPU.

Encodec training code is also open sourced https://twitter.com/MetaAI/status/1686776683398127616

ESPnet v.202308 has been released with many exciting movements, including discrete "asr2" recipe, faster CI, stable HuBERT pre-training, VISinger2 from Muskits, and many more! Check the detailed info at https://github.com/espnet/espnet/releases/tag/v.202308 Thanks to our great contributors as always!

AudioLDM 2: text to audio/music/speech generation Code/models/paper coming soon https://twitter.com/_akhaliq/status/1686790597087031305

https://arxiv.org/abs/2307.16430 VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design Jungil Kong, Jihoon Park, Beomjeong Kim, Jeongmin Kim, Dohee Kong, Sangjin Kim Single-stage text-to-speech models have been actively studied recently, and their results have outperformed two-stage pipeline systems. Although the previous single-stage model has made great progress, there is room for improvement in terms of its intermittent unnaturalness, computational efficiency, and strong dependence on phoneme conversion. In this work, we introduce VITS2, a single-stage text-to-speech model that efficiently synthesizes a more natural speech by improving several aspects of the previous work. We propose improved structures and training mechanisms and present that the proposed methods are effective in improving naturalness, similarity of speech characteristics in a multi-speaker model, and efficiency of training and inference. Furthermore, we demonstrate that the strong dependence on phoneme conversion in previous works can be significantly reduced with our method, which allows a fully end-to-end single-stage approach.

Starting with a text prompt, WavJourney can generate story-driven audio content with personalized speakers, lifelike speech, immersive music and sound effects! 🎧 💻GitHub: https://github.com/Audio-AGI/WavJourney 📎 Learn more at: https://twitter.com/LiuXub/status/1684338437934002176

Azure implements really cool features, very strong industrial solution for speech technology https://techcommunity.microsoft.com/t5/azure-ai-services-blog/creating-a-branded-ai-voice-that-conveys-emotions-and-speaks/ba-p/3876336

Interesting non-autoregressive model landed in espnet https://github.com/espnet/espnet/pull/5363 https://arxiv.org/abs/2010.14233 Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment Ethan A. Chi, Julian Salazar, Katrin Kirchhoff Non-autoregressive models greatly improve decoding speed over typical sequence-to-sequence models, but suffer from degraded performance. Infilling and iterative refinement models make up some of this gap by editing the outputs of a non-autoregressive model, but are constrained in the edits that they can make. We propose iterative realignment, where refinements occur over latent alignments rather than output sequence space. We demonstrate this in speech recognition with Align-Refine, an end-to-end Transformer-based model which refines connectionist temporal classification (CTC) alignments to allow length-changing insertions and deletions. Align-Refine outperforms Imputer and Mask-CTC, matching an autoregressive baseline on WSJ at 1/14th the real-time factor and attaining a LibriSpeech test-other WER of 9.0% without an LM. Our model is strong even in one iteration with a shallower decoder.