Speech Technology

https://twitter.com/Maureendss/status/1630209732223852544 📢 Exciting news! We just released ProsAudit, a prosodic benchmark for SSL models of speech 🥳 💬 It is now part of the Zero Resource Speech Challenge (track 4). The paper also includes results on a human comparison. 👨‍💻🤖 📰Check out the preprint: https://arxiv.org/pdf/2302.12057.pdf

https://arxiv.org/abs/2302.12369 Factual Consistency Oriented Speech Recognition Naoyuki Kanda, Takuya Yoshioka, Yang Liu This paper presents a novel optimization framework for automatic speech recognition (ASR) with the aim of reducing hallucinations produced by an ASR model. The proposed framework optimizes the ASR model to maximize an expected factual consistency score between ASR hypotheses and ground-truth transcriptions, where the factual consistency score is computed by a separately trained estimator. Experimental results using the AMI meeting corpus and the VoxPopuli corpus show that the ASR model trained with the proposed framework generates ASR hypotheses that have significantly higher consistency scores with ground-truth transcriptions while maintaining the word error rates close to those of cross entropy-trained ASR models. Furthermore, it is shown that training the ASR models with the proposed framework improves the speech summarization quality as measured by the factual consistency of meeting conversation summaries generated by a large language model.

How much smaller can you make your LM with overtraining? This figure from Chinchilla gives you a clue on what to expect. Say, you have C = 6e20. If N = 350M, it performs on par with L_opt of C = 1e20 (N_opt = 900M). => 6x training FLOPS for 2.5x less inference FLOPS https://twitter.com/arankomatsuzaki/status/1630257908238696449

From Phil Woodland https://arxiv.org/abs/2302.08579 Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmax Keqi Deng, Philip C. Woodland End-to-end (E2E) automatic speech recognition (ASR) implicitly learns the token sequence distribution of paired audio-transcript training data. However, it still suffers from domain shifts from training to testing, and domain adaptation is still challenging. To alleviate this problem, this paper designs a replaceable internal language model (RILM) method, which makes it feasible to directly replace the internal language model (LM) of E2E ASR models with a target-domain LM in the decoding stage when a domain shift is encountered. Furthermore, this paper proposes a residual softmax (R-softmax) that is designed for CTC-based E2E ASR models to adapt to the target domain without re-training during inference. For E2E ASR models trained on the LibriSpeech corpus, experiments showed that the proposed methods gave a 2.6% absolute WER reduction on the Switchboard data and a 1.0% WER reduction on the AESRC2020 corpus while maintaining intra-domain ASR results.

https://arxiv.org/abs/2302.10248 VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge Jaesung Huh, Andrew Brown, Jee-weon Jung, Joon Son Chung, Arsha Nagrani, Daniel Garcia-Romero, Andrew Zisserman This paper summarises the findings from the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22), which was held in conjunction with INTERSPEECH 2022. The goal of this challenge was to evaluate how well state-of-the-art speaker recognition systems can diarise and recognise speakers from speech obtained "in the wild". The challenge consisted of: (i) the provision of publicly available speaker recognition and diarisation data from YouTube videos together with ground truth annotation and standardised evaluation software; and (ii) a public challenge and hybrid workshop held at INTERSPEECH 2022. We describe the four tracks of our challenge along with the baselines, methods, and results. We conclude with a discussion on the new domain-transfer focus of VoxSRC-22, and on the progression of the challenge from the previous three editions.

From Google https://arxiv.org/abs/2302.11186 UML: A Universal Monolingual Output Layer for Multilingual ASR Chao Zhang, Bo Li, Tara N. Sainath, Trevor Strohman, Shuo-yiin Chang Word-piece models (WPMs) are commonly used subword units in state-of-the-art end-to-end automatic speech recognition (ASR) systems. For multilingual ASR, due to the differences in written scripts across languages, multilingual WPMs bring the challenges of having overly large output layers and scaling to more languages. In this work, we propose a universal monolingual output layer (UML) to address such problems. Instead of one output node for only one WPM, UML re-associates each output node with multiple WPMs, one for each language, and results in a smaller monolingual output layer shared across languages. Consequently, the UML enables to switch in the interpretation of each output node depending on the language of the input speech. Experimental results on an 11-language voice search task demonstrated the feasibility of using UML for high-quality and high-efficiency multilingual streaming ASR.

Attention ASR developers and researchers! 🚀 Great news, with the latest update of 🤗 PEFT, you can now fine-tune your Whisper-large model faster than ever before! The new update allows you to fit 5X larger batches with less than 10GB GPU VRAM, thanks to LoRA and Tim Dettmers's bnb packaged nicely in 🤗 PEFT. And the best part? You get a comparable WER, but just faster!! ⚡️ But that's not all, you no longer have to compromise on the training speed to maintain WER. In fact, in our experiments with the Marathi language, the WER was comparable with full fine-tuning runs of Whisper-large. Without PEFT, 13.64 WER (full training run) and with PEFT, 14.01 WER (trained on a @googlecolab ). With 🤗 PEFT, you can now train a Whisper-large v2 model in less than 8GB GPU VRAM! 📉 Without 🤗 PEFT, you could experience OOM on a Colab T4, but not anymore! You can easily save on storage and port tiny checkpoints, ~63 MB compared to 6.7 GB fully fine-tuned model. 🐜 And that's not all! For low latency, you can convert the PEFT model to ONNX and use ORT using 🤗 Optimum. Start experimenting today and fine-tune your Whisper using PEFT+INT8 in Colab on a language of your choice! Join our Discord community to get involved in the conversation and discuss your results and questions. 🔬 Check out the Colab notebook examples and start your ASR development journey with 🤗 PEFT today! https://github.com/huggingface/peft

We glad people use Vosk for real-life applications. If you have built something using Vosk please share. Here is a great example Pal Robotics uses Vosk to recognize speech in ARI V2 service robot https://pal-robotics.com/wp-content/uploads/2022/12/ARI-Datasheet.pdf

Coqui's AI voice studio is live! 🪄 Create new voices 🪞 Clone your voice ⚡️ Fuse voices 🎬 Direct your voices 📂 Organize your projects https://twitter.com/coqui_ai/status/1626738634849239042

Whisper optimized for Radeon cards https://github.com/Const-me/Whisper

https://opendata.iisys.de/ Good German TTS dataset (326 hours, 5 speakers) and 610h ASR dataset Learned from https://arxiv.org/abs/2302.06008 ASR Bundestag: A Large-Scale political debate dataset in German We present ASR Bundestag, a dataset for automatic speech recognition in German, consisting of 610 hours of aligned audio-transcript pairs for supervised training as well as 1,038 hours of unlabeled audio snippets for self-supervised learning, based on raw audio data and transcriptions from plenary sessions and committee meetings of the German parliament. In addition, we discuss utilized approaches for the automated creation of speech datasets and assess the quality of the resulting dataset based on evaluations and finetuning of a pre-trained state of the art model. We make the dataset publicly available, including all subsets.

Generally, TRT gives avg ~26.5% speed up. See README.md for details. https://github.com/k2-fsa/sherpa/pull/300

From FunASR Updated onnxruntime today, optimized inference speed, actual measurement, paraformer-large, compared to modelscope pipeline, 100 averages on cpu, 2.8 times faster inference speed, rtf: 0.110 -> 0.0386, deployed using onnx. Users can update the new pipeline: https://github.com/alibaba-damo-academy/FunASR/tree/main/funasr/runtime/python/onnxruntime/paraformer/rapid_paraformer

Some things about state of software from this weekend: Transformers library pipeline API doesn't split long texts for NER yet https://github.com/huggingface/transformers/pull/19735 Fixed bug in Kaldi matrix to load 10Gb matrices, not sure how it was unnoticed for such a long time https://github.com/kaldi-asr/kaldi/pull/4823

Abdelrahman Mohamed, Director of AI at Meta joined Rembrand https://www.rembrand.com/blog/rembrand-announces-8-million-seed-round/

https://github.com/openai/whisper/discussions/937 Whisper model in CTranslate2, which is a fast inference engine for Transformer models. The project supports many useful inference features such as CPU and GPU execution, asynchronous execution, multi-GPU execution, 8-bit quantization, etc. You can find a usage example here. Note that it does not currently implement the full transcription loop, only the model.decode part. So you would still need to implement the transcription logic from transcribe.py on top of it (iterate on each 30-second window, accumulate the context in the prompt, etc.). For example, here's the transcription time of 13 minutes of audio on a V100 for the same accuracy: Implementation Time with "small" model Time with "medium" model Baseline 1m37s 3m16s CTranslate2 0m25s 0m42s