Speech Technology

https://arxiv.org/abs/2211.03541 Multi-blank Transducers for Speech Recognition Hainan Xu, Fei Jia, Somshubra Majumdar, Shinji Watanabe, Boris Ginsburg This paper proposes a modification to RNN-Transducer (RNN-T) models for automatic speech recognition (ASR). In standard RNN-T, the emission of a blank symbol consumes exactly one input frame; in our proposed method, we introduce additional blank symbols, which consume two or more input frames when emitted. We refer to the added symbols as big blanks, and the method multi-blank RNN-T. For training multi-blank RNN-Ts, we propose a novel logit under-normalization method in order to prioritize emissions of big blanks. With experiments on multiple languages and datasets, we show that multi-blank RNN-T methods could bring relative speedups of over +90%/+139% to model inference for English Librispeech and German Multilingual Librispeech datasets, respectively. The multi-blank RNN-T method also improves ASR accuracy consistently. We will release our implementation of the method in the NeMo (\url{this https URL}) toolkit.

14 times faster than hifigan https://arxiv.org/abs/2211.06989 Autovocoder: Fast Waveform Generation from a Learned Speech Representation using Differentiable Digital Signal Processing Jacob J Webber, Cassia Valentini-Botinhao, Evelyn Williams, Gustav Eje Henter, Simon King Most state-of-the-art Text-to-Speech systems use the mel-spectrogram as an intermediate representation, to decompose the task into acoustic modelling and waveform generation. A mel-spectrogram is extracted from the waveform by a simple, fast DSP operation, but generating a high-quality waveform from a mel-spectrogram requires computationally expensive machine learning: a neural vocoder. Our proposed ``autovocoder'' reverses this arrangement. We use machine learning to obtain a representation that replaces the mel-spectrogram, and that can be inverted back to a waveform using simple, fast operations including a differentiable implementation of the inverse STFT. The autovocoder generates a waveform 5 times faster than the DSP-based Griffin-Lim algorithm, and 14 times faster than the neural vocoder HiFi-GAN. We provide perceptual listening test results to confirm that the speech is of comparable quality to HiFi-GAN in the copy synthesis task.

Paper for LAION https://t.me/speechtech/1158 https://arxiv.org/abs/2211.06687 Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.

Haha Full Primary Dataset ~7.1 TB Entire Dataset 30+ TB

https://github.com/EGO4D/audio-visual The Audio-Visual Diarization (AVD) benchmark corresponds to characterizing low-level information about conversational scenarios in the EGO4D dataset. This includes tasks focused on detection, tracking, segmentation of speakers and transcirption of speech content. To that end, we are proposing 4 tasks in this benchmark. For more information on Ego4D or to download the dataset, read: Start Here.

https://arxiv.org/abs/2211.08697 PBSM: Backdoor attack against Keyword spotting based on pitch boosting and sound masking Hanbo Cai, Pengcheng Zhang, Hai Dong, Yan Xiao, Shunhui Ji Keyword spotting (KWS) has been widely used in various speech control scenarios. The training of KWS is usually based on deep neural networks and requires a large amount of data. Manufacturers often use third-party data to train KWS. However, deep neural networks are not sufficiently interpretable to manufacturers, and attackers can manipulate third-party training data to plant backdoors during the model training. An effective backdoor attack can force the model to make specified judgments under certain conditions, i.e., triggers. In this paper, we design a backdoor attack scheme based on Pitch Boosting and Sound Masking for KWS, called PBSM. Experimental results demonstrated that PBSM is feasible to achieve an average attack success rate close to 90% in three victim models when poisoning less than 1% of the training data.

https://twitter.com/coqui_ai/status/1592991246414675969 🐸TTS v0.9.0 is out!! 25 new models in 25 different languages. See the release notes for more...

https://twitter.com/reach_vb/status/1592185698903199746 🗣️ A new speech community event is incoming!! 📆 The Whisper fine-tuning sprints will be held from the 5th to the 19th of December. 🌍 Come join us to build better and faster speech recognition systems in 70+ languages. 🔥 Claim SoTA in a language of your choice!

In 2018, Echo and Alexa lost about $5 billion, said a person with knowledge of the finances https://www.nytimes.com/2022/11/14/technology/amazon-layoffs.html

Prompt come to speech as well https://twitter.com/AlexTamkin/status/1586522694463287297

SpeechBrain still looks for sponsors 🙂 https://www.linkedin.com/posts/titouan-parcollet-b233a698_i-am-now-a-research-scientist-for-the-samsung-activity-6998297635019223040-nPYi/?utm_source=share&utm_medium=member_desktop

The core usecase for speech technology. The girl is actually very clever https://www.linkedin.com/posts/alexandrelebrun_i-finally-explained-my-8-year-old-daughter-activity-6997644425069740032-7JTQ I finally explained my 8-year old daughter what I have been working on for so many years, conversational AI etc. - "It's what powers things like Siri" I said. She understood and I thought: she is a genius. It's time to sign her up to Stanford's CS224n: Natural Language Processing with Deep Learning. Five minutes later, she comes to me and rubs my back: - "Poor dad, it must be so tiring to set timers for so many people around the world"

Recently Kaldi project released a pack of models trained on Gigaspeech. You can find them here Models are good, not significantly better than our previous model, but not significantly worse either. We expect them to work better on Youtube-like inputs and podcasts. Some notes: * Unlike model 0.22 this new model is more stable and doesn’t output ‘the’ for long silence regions. * The model has TDNN+LSTM architecture and a bit slow. * We packaged it with a big graph so the archive is larger (2.3Gb) and decoding requires 16Gb. It should not be a problem for modern servers. You can download models packaged for Vosk on https://alphacephei.com/vosk/models. Here are the accuracy results https://alphacephei.com/nsh/2022/11/13/kaldi-gigaspeech.html

A good review of RNNT LM rescoring https://arxiv.org/abs/2203.16776 An Empirical Study of Language Model Integration for Transducer based Speech Recognition Huahuan Zheng, Keyu An, Zhijian Ou, Chen Huang, Ke Ding, Guanglu Wan Utilizing text-only data with an external language model (ELM) in end-to-end RNN-Transducer (RNN-T) for speech recognition is challenging. Recently, a class of methods such as density ratio (DR) and internal language model estimation (ILME) have been developed, outperforming the classic shallow fusion (SF) method. The basic idea behind these methods is that RNN-T posterior should first subtract the implicitly learned internal language model (ILM) prior, in order to integrate the ELM. While recent studies suggest that RNN-T only learns some low-order language model information, the DR method uses a well-trained neural language model with full context, which may be inappropriate for the estimation of ILM and deteriorate the integration performance. Based on the DR method, we propose a low-order density ratio method (LODR) by replacing the estimation with a low-order weak language model. Extensive empirical experiments are conducted on both in-domain and cross-domain scenarios on English LibriSpeech & Tedlium-2 and Chinese WenetSpeech & AISHELL-1 datasets. It is shown that LODR consistently outperforms SF in all tasks, while performing generally close to ILME and better than DR in most tests. Implementation in k2 https://github.com/k2-fsa/icefall/pull/678 decoding method test-clean test-other modified_beam_search 2.73 7.15 modified_beam_search_rnnlm_shallow_fusion 2.42 6.46 modified_beam_search_rnnlm_LODR 2.28 5.94

Kaldi released Gigaspeech models

ATCO2 corpus: A Large-Scale Dataset for Research on Automatic Speech Recognition and Natural Language Understanding of Air Traffic Control Communications https://www.atco2.org/data https://arxiv.org/abs/2211.04054 Juan Zuluaga-Gomez, Karel Veselý, Igor Szöke, Petr Motlicek, Martin Kocour, Mickael Rigault, Khalid Choukri, Amrutha Prasad, Seyyed Saeed Sarfjoo, Iuliia Nigmatulina, Claudia Cevenini, Pavel Kolčárek, Allan Tart, Jan Černocký Personal assistants, automatic speech recognizers and dialogue understanding systems are becoming more critical in our interconnected digital world. A clear example is air traffic control (ATC) communications. ATC aims at guiding aircraft and controlling the airspace in a safe and optimal manner. These voice-based dialogues are carried between an air traffic controller (ATCO) and pilots via very-high frequency radio channels. In order to incorporate these novel technologies into ATC (low-resource domain), large-scale annotated datasets are required to develop the data-driven AI systems. Two examples are automatic speech recognition (ASR) and natural language understanding (NLU). In this paper, we introduce the ATCO2 corpus, a dataset that aims at fostering research on the challenging ATC field, which has lagged behind due to lack of annotated data. The ATCO2 corpus covers 1) data collection and pre-processing, 2) pseudo-annotations of speech data, and 3) extraction of ATC-related named entities. The ATCO2 corpus is split into three subsets. 1) ATCO2-test-set corpus contains 4 hours of ATC speech with manual transcripts and a subset with gold annotations for named-entity recognition (callsign, command, value). 2) The ATCO2-PL-set corpus consists of 5281 hours of unlabeled ATC data enriched with automatic transcripts from an in-domain speech recognizer, contextual information, speaker turn information, signal-to-noise ratio estimate and English language detection score per sample. Both available for purchase through ELDA at this http URL. 3) The ATCO2-test-set-1h corpus is a one-hour subset from the original test set corpus, that we are offering for free at this https URL. We expect the ATCO2 corpus will foster research on robust ASR and NLU not only in the field of ATC communications but also in the general research community.