Speech Technology

SLT2022 starts tomorrow, here is the technical program: https://slt2022.org/technical-program.php#ASR

Everyone is working on fast training. Here is example for BERT

https://github.com/NVIDIA/NeMo/releases/tag/v1.14.0 Hybrid CTC + Transducer loss ASR #5364 Sampled Softmax RNNT (Enables large vocab RNNT, for speech translation and multilingual ASR) #5216 ASR Adapters hyper parameter search scripts #5159 RNNT {ONNX, TorchScript} x GPU export infer #5248 Exportable MelSpectrogram (TorchScript) #5512 Audio To Audio Dataset Processor #5196 Multi Channel Audio Transcription #5479 Silence Augmentation #5476

I'm always interested in great research by @gnroy https://arxiv.org/abs/2212.08703 Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition Aleksandr Laptev, Boris Ginsburg This paper presents a class of new fast non-trainable entropy-based confidence estimation methods for automatic speech recognition. We show how per-frame entropy values can be normalized and aggregated to obtain a confidence measure per unit and per word for Connectionist Temporal Classification (CTC) and Recurrent Neural Network Transducer (RNN-T) models. Proposed methods have similar computational complexity to the traditional method based on the maximum per-frame probability, but they are more adjustable, have a wider effective threshold range, and better push apart the confidence distributions of correct and incorrect words. We evaluate the proposed confidence measures on LibriSpeech test sets, and show that they are up to 2 and 4 times better than confidence estimation based on the maximum per-frame probability at detecting incorrect words for Conformer-CTC and Conformer-RNN-T models, respectively.

Facebook tries to expriment with with different streaming architectures. However, the effect of low latency is still pretty bad, librispeech clean WER is above 3.0 no matter what you do. The thing here is that you don't always need such a low latency, only in the end of phrase probably. A controllable latency approach would be ideal. More on this later. https://arxiv.org/abs/2212.07650 Improving Fast-slow Encoder based Transducer with Streaming Deliberation Ke Li, Jay Mahadeokar, Jinxi Guo, Yangyang Shi, Gil Keren, Ozlem Kalinli, Michael L. Seltzer, Duc Le This paper introduces a fast-slow encoder based transducer with streaming deliberation for end-to-end automatic speech recognition. We aim to improve the recognition accuracy of the fast-slow encoder based transducer while keeping its latency low by integrating a streaming deliberation model. Specifically, the deliberation model leverages partial hypotheses from the streaming fast encoder and implicitly learns to correct recognition errors. We modify the parallel beam search algorithm for fast-slow encoder based transducer to be efficient and compatible with the deliberation model. In addition, the deliberation model is designed to process streaming data. To further improve the deliberation performance, a simple text augmentation approach is explored. We also compare LSTM and Conformer models for encoding partial hypotheses. Experiments on Librispeech and in-house data show relative WER reductions (WERRs) from 3% to 5% with a slight increase in model size and negligible extra token emission latency compared with fast-slow encoder based transducer. Compared with vanilla neural transducers, the proposed deliberation model together with fast-slow encoder based transducer obtains relative 10-11% WERRs on Librispeech and around relative 6% WERR on in-house data with smaller emission delays.

The statistical process part in the diffusion process doesn’t seem very imporant, what is important is top-down model of the reality when we start with overall shape and go down to smaller and smaller things. With diffusion process we learn to figure out important parts and go step by step to more detailed represenation in an efficient way. It is exciting for me how similar the processes of boosting and diffusion are. Top-down feature engineering seems like a really promising way and we might see more of them in modern efficient deep learning. https://alphacephei.com/nsh/2022/12/24/stable-diffusion.html

The problem statement is a responsibility of a good scientific advisor (like Prof Khudanpur)

Not really speech, but I found it facinating such thing exists. Training convolutional net CIFAR on single GPU in 2 seconds to 94% accuracy: https://github.com/tysam-code/hlb-CIFAR10 Taken from https://news.ycombinator.com/item?id=34096988

CommonVoice case demonstrated that OSS is more about social setup than software itself. This is a nice followup creat by Yandex developers: https://techcrunch.com/2022/12/20/petals-is-creating-a-free-distributed-network-for-running-text-generating-ai/ https://github.com/bigscience-workshop/petals https://arxiv.org/abs/2209.01188 Petals: Collaborative Inference and Fine-tuning of Large Models Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research. In this work, we propose Petals − a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties trusted to process client's data. We demonstrate that this strategy significantly outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with ≈ 1 step per second. Unlike most inference APIs, Petals also natively exposes the hidden states of served models, allowing its users to train and share custom model extensions based on efficient fine-tuning methods

I made this introduction (97 slides) to modern ASR systems and speech SSL. I share the pdf on https://github.com/besacier/ASR2022 Should be usable for 6-8h of lecture. Below the improvements I'd like to have to make it better: https://twitter.com/laurent_besacie/status/1604831966880493568

Facebook released data2vec2.0 recently. Claimed to be better than wav2vec and hubert and wavlm https://ai.facebook.com/research/publications/efficient-self-supervised-learning-with-contextualized-target-representations-for-vision-speech-and-language

Apple has very old GPU cards https://arxiv.org/abs/2211.16270 Neural Transducer Training: Reduced Memory Consumption with Sample-wise Computation Stefan Braun, Erik McDermott, Roger Hsiao The neural transducer is an end-to-end model for automatic speech recognition (ASR). While the model is well-suited for streaming ASR, the training process remains challenging. During training, the memory requirements may quickly exceed the capacity of state-of-the-art GPUs, limiting batch size and sequence lengths. In this work, we analyze the time and space complexity of a typical transducer training setup. We propose a memory-efficient training method that computes the transducer loss and gradients sample by sample. We present optimizations to increase the efficiency and parallelism of the sample-wise method. In a set of thorough benchmarks, we show that our sample-wise method significantly reduces memory usage, and performs at competitive speed when compared to the default batched computation. As a highlight, we manage to compute the transducer loss and gradients for a batch size of 1024, and audio length of 40 seconds, using only 6 GB of memory.

Stable DIffusion for Audio https://www.riffusion.com/about

Whisper finetuning results, mostly small languages https://twitter.com/reach_vb/status/1603451281007202304