Speech Technology

https://flashspeech.github.io/ FlashSpeech, extremely efficient zero-shot TTS with 20x inference speedup while maintaining high-quality voice cloning, friendly for online real-time serving. https://arxiv.org/abs/2404.14700

VoiceCraft released 830M model https://huggingface.co/pyp1/VoiceCraft_830M_TTSEnhanced

Unofficial implementation of wavenext neural vocoder(WIP) https://github.com/wetdog/wavenext_pytorch WaveNext: ConvNext-Based fast neural vocoder without ISTFT layer WaveNext proposed to replace the ISTFT final layer of Vocos with a linear layer without bias followed by a reshape op. As this is a slight modification of vocos we're just using the official vocos implementation and adding the WaveNext head in wavenext_pytorch/vocos/heads.py

📣 We are delighted to announce that we will be hosting the Text-dependent Speaker Verification (TdSV) Challenge 2024 in conjunction with SLT 2024. Following the success of two previous Short-duration Speaker Verification Challenges, the TdSV Challenge 2024 aims to focus on the relevance of recent training strategies, such as self-supervised learning. 🏆 The challenge evaluates the TdSV task in two practical scenarios, namely, conventional TdSV using predefined Passphrases (Task 1) and TdSV using user-defined passphrases (Task 2). Three cash prizes will be given away for each task ($7000 in total) based on the results of the evaluation dataset and other qualitative factors. 🌐 Challenge website 🌐 Challenge registration form

A guy recently shared 4-bit versions of Whisper V3 models that made me return to Whisper libraries and retest. https://huggingface.co/Ftfyhh/whisper-ggml-q4_0-models/tree/main Overall state is that there is a lot of work still https://alphacephei.com/nsh/2024/04/20/status-of-whisper.html But 4-bit Whisper is recommended, works pretty well

Train Long and Test Long:Leveraging Full Document Contexts in Speech Processing https://ieeexplore.ieee.org/document/10446727 William Chen; Takatomo Kano; Atsunori Ogawa; Marc Delcroix; Shinji Watanabe The quadratic memory complexity of self-attention has generally restricted Transformer-based models to utterance-based speech processing, preventing models from leveraging long-form contexts. A common solution has been to formulate long-form speech processing into a streaming problem, only using limited prior context. We propose a new and simple paradigm, encoding entire documents at once, which has been unexplored in Automatic Speech Recognition (ASR) and Speech Translation (ST) due to its technical infeasibility. We exploit developments in efficient attention mechanisms, such as Flash Attention, and show that Transformer-based models can be easily adapted to document-level processing. We experiment with methods to address the quadratic complexity of attention by replacing it with simpler alternatives. As such, our models can handle up to 30 minutes of speech during both training and testing. We evaluate our models on ASR, ST, and Speech Summarization (SSUM) using How2, TEDLIUM3, and SLUE-TED. With document-level context, our ASR models achieve 33.3% and 6.5% relative improvements in WER on How2 and TEDLIUM3 over prior work. Finally, we use our findings to propose a new attention-free self-supervised model, LongHuBERT, capable of handling long inputs. In doing so, we achieve state-of-the-art performance on SLUE-TED SSUM, outperforming cascaded systems that have dominated the benchmark.

Big announcement: @pleiasfr releases a massive open corpus of 2 million Youtube videos in Creative Commons (CC-By) on @huggingface . Youtube-Commons features 30 billion words of audio transcriptions in multiple languages, and soon other modalities https://huggingface.co/datasets/PleIAs/YouTube-Commons

Small vocoder only 1.8M parameters https://github.com/signofthefour/fregrad

People say this vocoder has a point by joining signal processing with neural tech https://ast-astrec.nict.go.jp/demo_samples/firnet_icassp2024/ FIRNet: Fast and pitch controllable neural vocoder with trainable finite impulse response filter Some neural vocoders with fundamental frequency (f0) control have succeeded in performing real-time inference on a single CPU while preserving the quality of the synthetic speech. However, compared with legacy vocoders based on signal processing, their inference speeds are still low. This paper proposes a neural vocoder based on the source-filter model with trainable time-variant finite impulse response (FIR) filters, to achieve a similar inference speed to legacy vocoders. In the proposed model, FIRNet, multiple FIR coefficients are predicted using the neural networks, and the speech waveform is then generated by convolving a mixed excitation signal with these FIR coefficients. Experimental results show that FIRNet can achieve an inference speed similar to legacy vocoders while maintaining f0 controllability and natural speech quality. https://ast-astrec.nict.go.jp/release/preprints/preprint_icassp_2024_ohtani.pdf

Assembly.AI paper The technical report about our latest Universal-1 multilingual ASR model is out! Universal-1 is our latest ASR model in production, designed for high-quality, high-throughput, and large-scale operations. Not only does it demonstrate competitive or superior WERs in English, Spanish, German, and French under various conditions, but it also shows advantages in various practically relevant areas, such as accurate timestamp prediction, robustness against hallucinations, and handling code-switching. In this report, we adopt a holistic, system-centric approach to analyzing various aspects of fully-fledged ASR models to draw practically relevant insights that are useful for real-world services operating at scale. We hope that our report will help advance the speech field as it finds more and more applications in the real world. https://arxiv.org/abs/2404.09841

I tested Google ASR recently - Chirp, Conformer (latest version) and Gemini. Conformer is not good. Chirp is ok, somewhat better than Whisper V.

ICASSP starts next week. There will be many cool things if I'd have time to read it all https://research.google/conferences-and-events/google-at-icassp-2024/

https://arxiv.org/abs/2404.06714 Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness Xincan Feng, Akifumi Yoshimoto Recent advancements in Natural Language Processing (NLP) have seen Large-scale Language Models (LLMs) excel at producing high-quality text for various purposes. Notably, in Text-To-Speech (TTS) systems, the integration of BERT for semantic token generation has underscored the importance of semantic content in producing coherent speech outputs. Despite this, the specific utility of LLMs in enhancing TTS synthesis remains considerably limited. This research introduces an innovative approach, Llama-VITS, which enhances TTS synthesis by enriching the semantic content of text using LLM. Llama-VITS integrates semantic embeddings from Llama2 with the VITS model, a leading end-to-end TTS framework. By leveraging Llama2 for the primary speech synthesis process, our experiments demonstrate that Llama-VITS matches the naturalness of the original VITS (ORI-VITS) and those incorporate BERT (BERT-VITS), on the LJSpeech dataset, a substantial collection of neutral, clear speech. Moreover, our method significantly enhances emotive expressiveness on the EmoV_DB_bea_sem dataset, a curated selection of emotionally consistent speech from the EmoV_DB dataset, highlighting its potential to generate emotive speech.

Parler-TTS quality is not something exceptional. But the whole idea working with audio with text prompts is somewhat interesting (audio cleanup, denoising, separation, etc).

https://github.com/speechbrain/benchmarks/tree/main/benchmarks/CL_MASR CL-MASR: A Continual Learning Benchmark for Multilingual ASR This is the official benchmark platform accompanying the paper CL-MASR: A Continual Learning Benchmark for Multilingual ASR. It includes scripts to train Whisper and WavLM-based ASR systems on a subset of 20 languages selected from Common Voice 13 in a continual learning fashion using a handful of methods including rehearsal-based, architecture-based, and regularization-based approaches. The goal is to continually learn new languages while limiting forgetting the previously learned ones. An ideal method should achieve both positive forward transfer (i.e. improve performance on new tasks leveraging shared knowledge from previous tasks) and positive backward transfer (i.e. improve performance on previous tasks leveraging shared knowledge from new tasks).

Multimodal speech LLM work by Google DeepMind Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung, Daniel Cer, Siddharth Dalmia, Gustavo Hernandez Abrego Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn't require speech data during LLM pre-training and can exploit LLM's multilingual text understanding capabilities to match speech and text in languages unseen during retrieval training. Our multi-modal LLM-based retrieval system is capable of matching speech and text in 102 languages despite only training on 21 languages. Our system outperforms previous systems trained explicitly on all 102 languages. We achieve a 10% absolute improvement in Recall@1 averaged across these languages. Additionally, our model demonstrates cross-lingual speech and text matching, which is further enhanced by readily available machine translation data. https://arxiv.org/abs/2404.01616v2

RALL-E with chain-of-thought (CoT) prompting is helpful to improve the robustness of codec LLM for speech synthesis, reducing error rate from 68% to 4% on extremely hard test sentences. https://huggingface.co/papers/2404.03204 https://ralle-demo.github.io/RALL-E/

Hm, 12 million hours and only 13% more accurate than Whisper Key stats: - Trained on 12.5M hours of training data - 13.5% more accurate than models like Whisper and >22% more accurate than APIs from Azure/AWS/Google - Up to 30% fewer hallucinations than seq2seq models like Whisper - 71% better speaker count estimation and 14% better word timestamp estimation compared to our prior models - 38 seconds to process a 60-minute audio file https://twitter.com/AssemblyAI/status/1775527558412460120

The audio-to-audio feature in stableaudio opens up new workflows for rapid sonic exploration in audiovisual production. https://twitter.com/jordiponsdotme/status/1775779209891246560