Speech Technology

https://arxiv.org/abs/2309.04031 Multiple Representation Transfer from Large Language Models to End-to-End ASR Systems Takuma Udagawa, Masayuki Suzuki, Gakuto Kurata, Masayasu Muraoka, George Saon Transferring the knowledge of large language models (LLMs) is a promising technique to incorporate linguistic knowledge into end-to-end automatic speech recognition (ASR) systems. However, existing works only transfer a single representation of LLM (e.g. the last layer of pretrained BERT), while the representation of a text is inherently non-unique and can be obtained variously from different layers, contexts and models. In this work, we explore a wide range of techniques to obtain and transfer multiple representations of LLMs into a transducer-based ASR system. While being conceptually simple, we show that transferring multiple representations of LLMs can be an effective alternative to transferring only a single representation.

To benefit the research community and accelerate the research of speech processing in the driving scenario, we launch the ICASSP2024 In-Car Multi-Channel Automatic Speech Recognition Challenge (ICMC-ASR), which is dedicated to the domain of speech recognition in complex driving conditions. Different from previous challenges, the ICMC-ASR dataset comprises an extensive collection of 1000 hours of real-world recorded, multi-channel, multi-speaker, in-car conversational Mandarin speech data.  http://icmcasr.org/

Recently discussed leaderboard https://huggingface.co/spaces/hf-audio/open_asr_leaderboard is somewhat nice, but it lists Whisper RTF the same as Transducder RTF 12.7e-3 no idea how they got that number. In my experience for short files Whisper Medium is 0.3xRT on GPU and Whisper Large is 0.7xRT.

One more TTS, winner one https://arxiv.org/abs/2309.02743 MuLanTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2023 Zhihang Xu, Shaofei Zhang, Xi Wang, Jiajun Zhang, Wenning Wei, Lei He, Sheng Zhao In this paper, we present MuLanTTS, the Microsoft end-to-end neural text-to-speech (TTS) system designed for the Blizzard Challenge 2023. About 50 hours of audiobook corpus for French TTS as hub task and another 2 hours of speaker adaptation as spoke task are released to build synthesized voices for different test purposes including sentences, paragraphs, homographs, lists, etc. Building upon DelightfulTTS, we adopt contextual and emotion encoders to adapt the audiobook data to enrich beyond sentences for long-form prosody and dialogue expressiveness. Regarding the recording quality, we also apply denoise algorithms and long audio processing for both corpora. For the hub task, only the 50-hour single speaker data is used for building the TTS system, while for the spoke task, a multi-speaker source model is used for target speaker fine tuning. MuLanTTS achieves mean scores of quality assessment 4.3 and 4.5 in the respective tasks, statistically comparable with natural speech while keeping good similarity according to similarity assessment. The excellent quality and similarity in this year's new and dense statistical evaluation.

Matcha-TTS, A fast TTS architecture with conditional flow matching Demo page: https://shivammehta25.github.io/Matcha-TTS/ arXiv preprint: https://arxiv.org/abs/2309.03199 Diffusion models are great but slow to sample from. Lipman et al 2023, introduced conditional flow matching a new simulation-free umbrella paradigm to improve the transport paths taken from noise to data using continuous normalising flows. We used conditional flow matching and made various design choices to improve on various aspects to improve the synthesis and training speed whilst minimising memory consumption. Presenting our latest work, Matcha-TTS, (pronounced as Matcha-TeaTS). Read more about it on our arXiv preprint.

Good features, good system https://arxiv.org/abs/2308.15945 The DeepZen Speech Synthesis System for Blizzard Challenge 2023 Christophe Veaux, Ranniery Maia, Spyridoula Papendreou This paper describes the DeepZen text to speech (TTS) system for Blizzard Challenge 2023. The goal of this challenge is to synthesise natural and high-quality speech in French, from a large monospeaker dataset (hub task) and from a smaller dataset by speaker adaptation (spoke task). We participated to both tasks with the same model architecture. Our approach has been to use an auto-regressive model, which retains an advantage for generating natural sounding speech but to improve prosodic control in several ways. Similarly to non-attentive Tacotron, the model uses a duration predictor and gaussian upsampling at inference, but with a simpler unsupervised training. We also model the speaking style at both sentence and word levels by extracting global and local style tokens from the reference speech. At inference, the global and local style tokens are predicted from a BERT model run on text. This BERT model is also used to predict specific pronunciation features like schwa elision and optional liaisons. Finally, a modified version of HifiGAN trained on a large public dataset and fine-tuned on the target voices is used to generate speech waveform. Our team is identified as O in the the Blizzard evaluation and MUSHRA test results show that our system performs second ex aequo in both hub task (median score of 0.75) and spoke task (median score of 0.68), over 18 and 14 participants, respectively.

https://huggingface.co/blog/audioldm2 In this blog post, we showcased four optimisation methods that are available out of the box with Diffusers, taking the generation time of AudioLDM 2 from 14 seconds down to less than 1 second. We also highlighted how to employ memory saving tricks, such as half-precision and CPU offload, to reduce peak memory usage for long audio samples or large checkpoint sizes.

Related - TTS-RNNT in Nemo https://github.com/NVIDIA/NeMo/commit/33badd437b43133085e70fccf681bea33fa7cec8 paper OverFlow: Putting flows on top of neural transducers for better TTS Shivam Mehta, Ambika Kirkland, Harm Lameris, Jonas Beskow, Éva Székely, Gustav Eje Henter Neural HMMs are a type of neural transducer recently proposed for sequence-to-sequence modelling in text-to-speech. They combine the best features of classic statistical speech synthesis and modern neural TTS, requiring less data and fewer training updates, and are less prone to gibberish output caused by neural attention failures. In this paper, we combine neural HMM TTS with normalising flows for describing the highly non-Gaussian distribution of speech acoustics. The result is a powerful, fully probabilistic model of durations and acoustics that can be trained using exact maximum likelihood. Experiments show that a system based on our proposal needs fewer updates than comparable methods to produce accurate pronunciations and a subjective speech quality close to natural speech. https://www.isca-speech.org/archive/pdfs/interspeech_2023/mehta23_interspeech.pdf

We're releasing the code and @huggingface demo! Experience WavJourney now! 🤩🤩🤩 Code: https://github.com/Audio-AGI/WavJourney Demo: https://huggingface.co/spaces/Audio-AGI/WavJourney

Actually it is not that bad as I originally thought https://alphacephei.com/nsh/2023/08/24/mms-seamless.html

Multilingual and Code-Switching Speech Recognition 2022 Eighth Frederick Jelinek Memorial Summer Workshop https://www.clsp.jhu.edu/wp-content/uploads/2023/08/JSALT2022_CS_Report.pdf

New Seamless thing feels useless, much like previous MMS MichaelSpecter_2010-0089092-0089921 - what MichaelSpecter_2010-0089921-0090664 - what MichaelSpecter_2010-0090664-0091588 - what MichaelSpecter_2010-0091588-0092685 - what MichaelSpecter_2010-0092685-0093129 it s okay MichaelSpecter_2010-0093225-0093551 so after this amazingly fun conversation MichaelSpecter_2010-0093730-0094154 - what MichaelSpecter_2010-0094154-0095334 - what MichaelSpecter_2010-0095334-0096148 - what MichaelSpecter_2010-0096148-0096748 we re going

This thing does ASR too https://ai.meta.com/research/publications/seamless-m4t/ not sure how accurate though