Speech Technology

Not just WER is important, nice idea here: https://arxiv.org/abs/2401.01572 Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models Rita Frieske, Bertram E. Shi Hallucinations are a type of output error produced by deep neural networks. While this has been studied in natural language processing, they have not been researched previously in automatic speech recognition. Here, we define hallucinations in ASR as transcriptions generated by a model that are semantically unrelated to the source utterance, yet still fluent and coherent. The similarity of hallucinations to probable natural language outputs of the model creates a danger of deception and impacts the credibility of the system. We show that commonly used metrics, such as word error rates, cannot differentiate between hallucinatory and non-hallucinatory models. To address this, we propose a perturbation-based method for assessing the susceptibility of an automatic speech recognition (ASR) model to hallucination at test time, which does not require access to the training dataset. We demonstrate that this method helps to distinguish between hallucinatory and non-hallucinatory models that have similar baseline word error rates. We further explore the relationship between the types of ASR errors and the types of dataset noise to determine what types of noise are most likely to create hallucinatory outputs. We devise a framework for identifying hallucinations by analysing their semantic connection with the ground truth and their fluency. Finally, we discover how to induce hallucinations with a random noise injection to the utterance. Related, NVIDIA adds BLEU score to ASR models: https://github.com/NVIDIA/NeMo/pull/8069

Needs a mention. Overall, open source is hard.

Parakeet RNNT & CTC models top the Open ASR Leaderboard! 👑 Brought to you by @NVIDIAAI and @suno_ai_ , parakeet beats Whisper and regains its first place. The models are released under a commercially permissive license! 🥳 The models inherit the same FastConformer architecture and come in 2 flavours: 1. RNNT (1.1B & 0.6B) 2. CTC (1.1B & 0.5B) Each model is trained on 65K hours of English data (40K private proprietary data by Suno & NeMo teams) over several hundred epochs. Key features of the parakeet model: 1. It doesn't hallucinate (if the audio sample has silence, the output is silent). 2. It is quite robust to noisy audio (if the audio sample has non-vocal sounds, it outputs silence). Here's how you can use these models, too: 1. Install NeMo toolkit pip install nemo_toolkit['asr'] 2. Instantiate the model from the Hub import nemo.collections.asr as nemo_asr asr_model = nemo_asr.models.EncDecRNNTBPEModel.from_pretrained(model_name="nvidia/parakeet-rnnt-1.1b") 3. Transcribe away! asr_model.transcribe(['<FILE_NAME.WAV>']) That's it! 🤗 https://twitter.com/reach_vb/status/1742261240141918684

"Sounds emitted by plants under stress are airborne and informative": https://cell.com/cell/fulltext/S0092-8674(23)00262-3 TL;DR: Plants scream. https://twitter.com/samim/status/1641831857804173313

Overall, it is an interesting aspect of speech https://arxiv.org/abs/2312.16599 Relationship between auditory and semantic entrainment using Deep Neural Networks (DNN) Jay Kejriwal, Štefan Beňuš The tendency of people to engage in similar, matching, or synchronized behaviour when interacting is known as entrainment. Many studies examined linguistic (syntactic and lexical structures) and paralinguistic (pitch, intensity) entrainment, but less attention was given to finding the relationship between them. In this study, we utilized state-of-the-art DNN embeddings such as BERT and TRIpLet Loss network (TRILL) vectors to extract features for measuring semantic and auditory similarities of turns within dialogues in two comparable spoken corpora of two different languages. We found people's tendency to entrain on semantic features more when compared to auditory features. Additionally, we found that entrainment in semantic and auditory linguistic features are positively correlated. The findings of this study might assist in implementing the mechanism of entrainment in human-machine interaction (HMI).

Below is the reduction in Real-time factor (RTF): whisper large v3: 10.3 -> 7.45 distil whisper v2: 4.93 -> 2.08 How did we achieve this speed-up? 1. Native SDPA (Scaled Dot Product Attention) integration. 2. Torch backend for STFT (Short-Term Fourier Transform). How can you benefit from this? pip install --upgrade git+https://github. com/huggingface/transformers.git

GoogleDeepMind paper wins Best Paper for unlocking spoken language understanding in frozen LLMs. Dive deeper: https://arxiv.org/abs/2310.00230 https://twitter.com/IzhakShafran/status/1738083120937889992

Retrieval was always a thing https://arxiv.org/abs/2312.13560 kNN-CTC: Enhancing ASR via Retrieval of CTC Pseudo Labels Jiaming Zhou, Shiwan Zhao, Yaqi Liu, Wenjia Zeng, Yong Chen, Yong Qin The success of retrieval-augmented language models in various natural language processing (NLP) tasks has been constrained in automatic speech recognition (ASR) applications due to challenges in constructing fine-grained audio-text datastores. This paper presents kNN-CTC, a novel approach that overcomes these challenges by leveraging Connectionist Temporal Classification (CTC) pseudo labels to establish frame-level audio-text key-value pairs, circumventing the need for precise ground truth alignments. We further introduce a skip-blank strategy, which strategically ignores CTC blank frames, to reduce datastore size. kNN-CTC incorporates a k-nearest neighbors retrieval mechanism into pre-trained CTC ASR systems, achieving significant improvements in performance. By incorporating a k-nearest neighbors retrieval mechanism into pre-trained CTC ASR systems and leveraging a fine-grained, pruned datastore, kNN-CTC consistently achieves substantial improvements in performance under various experimental settings. Our code is available at this https URL.

https://arxiv.org/abs/2312.13585 Speech Translation with Large Language Models: An Industrial Practice Zhichao Huang, Rong Ye, Tom Ko, Qianqian Dong, Shanbo Cheng, Mingxuan Wang, Hang Li Given the great success of large language models (LLMs) across various tasks, in this paper, we introduce LLM-ST, a novel and effective speech translation model constructed upon a pre-trained LLM. By integrating the large language model (LLM) with a speech encoder and employing multi-task instruction tuning, LLM-ST can produce accurate timestamped transcriptions and translations, even from long audio inputs. Furthermore, our findings indicate that the implementation of Chain-of-Thought (CoT) prompting can yield advantages in the context of LLM-ST. Through rigorous experimentation on English and Chinese datasets, we showcase the exceptional performance of LLM-ST, establishing a new benchmark in the field of speech translation. Demo: https://speechtranslation.github.io/llm-st/

https://github.com/open-mmlab/Amphion released naturalspeech checkpoint https://huggingface.co/spaces/amphion/NaturalSpeech2/tree/main/ckpts/ns2

Very important paper, its time to bring phonemes back https://arxiv.org/abs/2312.09582 Phoneme-aware Encoding for Prefix-tree-based Contextual ASR Hayato Futami, Emiru Tsunoo, Yosuke Kashiwagi, Hiroaki Ogawa, Siddhant Arora, Shinji Watanabe In speech recognition applications, it is important to recognize context-specific rare words, such as proper nouns. Tree-constrained Pointer Generator (TCPGen) has shown promise for this purpose, which efficiently biases such words with a prefix tree. While the original TCPGen relies on grapheme-based encoding, we propose extending it with phoneme-aware encoding to better recognize words of unusual pronunciations. As TCPGen handles biasing words as subword units, we propose obtaining subword-level phoneme-aware encoding by using alignment between phonemes and subwords. Furthermore, we propose injecting phoneme-level predictions from CTC into queries of TCPGen so that the model better interprets the phoneme-aware encodings. We conducted ASR experiments with TCPGen for RNN transducer. We observed that proposed phoneme-aware encoding outperformed ordinary grapheme-based encoding on both the English LibriSpeech and Japanese CSJ datasets, demonstrating the robustness of our approach across linguistically diverse languages.

https://twitter.com/chenwanch1/status/1736568496891498552/photo/1 The details of OWSM’s budget was featured in @shinjiw_at_cmu keynote at @ASRU2023 “It would have been a disaster if the paper was rejected”

Hot topic on merging text into ASR training https://arxiv.org/abs/2312.09100 FastInject: Injecting Unpaired Text Data into CTC-based ASR training Keqi Deng, Philip C. Woodland Recently, connectionist temporal classification (CTC)-based end-to-end (E2E) automatic speech recognition (ASR) models have achieved impressive results, especially with the development of self-supervised learning. However, E2E ASR models trained on paired speech-text data often suffer from domain shifts from training to testing. To alleviate this issue, this paper proposes a flat-start joint training method, named FastInject, which efficiently injects multi-domain unpaired text data into CTC-based ASR training. To maintain training efficiency, text units are pre-upsampled, and their representations are fed into the CTC model along with speech features. To bridge the modality gap between speech and text, an attention-based modality matching mechanism (AM3) is proposed, which retains the E2E flat-start training. Experiments show that the proposed FastInject gave a 22\% relative WER reduction (WERR) for intra-domain Librispeech-100h data and 20\% relative WERR on out-of-domain test sets.

From Google. While reduction is interesting, the model size is still very large (10% of 2B parameters is still like 400Mb model) https://arxiv.org/abs/2312.08553 USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models Shaojin Ding, Qiu David, David Rim, Yanzhang He, Oleg Rybakov, Bo Li, Rohit Prabhavalkar, Weiran Wang, Tara N. Sainath, Shivani Agrawal, Zhonglin Han, Jian Li, Amir Yazdanbakhsh End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploying these massive USMs is extremely expensive due to the enormous memory usage and computational cost. Therefore, model compression is an important research topic to fit USM-based ASR under budget in real-world scenarios. In this study, we propose a USM fine-tuning approach for ASR, with a low-bit quantization and N:M structured sparsity aware paradigm on the model weights, reducing the model complexity from parameter precision and matrix topology perspectives. We conducted extensive experiments with a 2-billion parameter USM on a large-scale voice search dataset to evaluate our proposed method. A series of ablation studies validate the effectiveness of up to int4 quantization and 2:4 sparsity. However, a single compression technique fails to recover the performance well under extreme setups including int2 quantization and 1:4 sparsity. By contrast, our proposed method can compress the model to have 9.4% of the size, at the cost of only 7.3% relative word error rate (WER) regressions. We also provided in-depth analyses on the results and discussions on the limitations and potential solutions, which would be valuable for future studies.