Speech Technology

The Expresso dataset is a high-quality (48kHz) expressive speech dataset that includes both expressively rendered read speech (8 styles, in mono wav format) and improvised dialogues (26 styles, in stereo wav format). The dataset includes 4 speakers (2 males, 2 females), and totals 40 hours (11h read, 30h improvised). The transcriptions of the read speech are also provided. https://huggingface.co/datasets/ylacombe/expresso

More serious speech LLM https://github.com/ddlBoJack/SLAM-LLM

Another multimodal LLM doing speech https://github.com/OpenMOSS/AnyGPT

There will be CHiME-8 webinar! If you're interested in the CHiME-8 challenge, please join! Date: May 20, 2024 Time: 8:00 AM (US - ET) Place: https://cmu.zoom.us/j/92314209923?pwd=TFpPUm1DTDhUOHJKbDdndFg1QmxPdz09 Likely meeting will be recorded

Valuation 50x revenue. Ok https://github.com/PolyAI-LDN/pheme https://www.linkedin.com/posts/seb-johnson_polyai-secures-near-500mn-valuation-in-boost-activity-7196548712846761984-MOLg/ BREAKING: PolyAI RAISES $50M AT $500M VALUATION On monday I posted about PolyAI and how they were shaping up to become one of the UK's strongest AI companies. Today they announced a huge fundraise from new investors including chipmaking giant NVIDIA. PolyAI is a Conversational AI platform automating customer service. It was founded in 2017 by Nikola Mrkšić, Tsung-Hsien Wen, and Pei-Hao (Eddy) Su who met at the University of Cambridge’s Machine Intelligence Lab. As part of the fundraise the company highlighted some INSANE financial results: * 10m revenue last year * On track to TRIPLE that this year (i.e. $30m) * 90% Gross Margin (FY23) PolyAI grew out of Entrepreneur First before going on to raise several investment rounds, including their $40m Series B which valued the company at nearly $300 million post-money. Two years later this has grown to $500m! A great result for the team and a further boost for the UK’s ambitions of becoming an AI hub.

Whisper.cpp implemented flash attention https://github.com/ggerganov/whisper.cpp/releases/tag/v1.6.0

Apple optimizes Conformer for mobile devices. Some in-depth ideas https://arxiv.org/abs/2312.10359

And another one text-guided editing https://github.com/JinhuaLiang/WavCraft

https://github.com/tincans-ai/gazelle https://tincans.ai/ We're building real-time conversational speech with large language models. Bots should be fun to talk to. Previously, we built large scale machine learning systems at Cash App, Quora, Airbnb, and more.

Its interesting that Cambridge lab is doing a lot of Whisperology recently https://arxiv.org/abs/2405.06134 Muting Whisper: A Universal Acoustic Adversarial Attack on Speech Foundation Models Vyas Raina, Rao Ma, Charles McGhee, Kate Knill, Mark Gales Recent developments in large speech foundation models like Whisper have led to their widespread use in many automatic speech recognition (ASR) applications. These systems incorporate `special tokens' in their vocabulary, such as <endoftext>, to guide their language generation process. However, we demonstrate that these tokens can be exploited by adversarial attacks to manipulate the model's behavior. We propose a simple yet effective method to learn a universal acoustic realization of Whisper's <endoftext> token, which, when prepended to any speech signal, encourages the model to ignore the speech and only transcribe the special token, effectively `muting' the model. Our experiments demonstrate that the same, universal 0.64-second adversarial audio segment can successfully mute a target Whisper ASR model for over 97\% of speech samples. Moreover, we find that this universal adversarial audio segment often transfers to new datasets and tasks. Overall this work demonstrates the vulnerability of Whisper models to `muting' adversarial attacks, where such attacks can pose both risks and potential benefits in real-world settings: for example the attack can be used to bypass speech moderation systems, or conversely the attack can also be used to protect private speech data.

Interesting case of split of the SLU between STM32 microcontroller and the cloud https://arxiv.org/abs/2311.18188 Speech Understanding on Tiny Devices with A Learning Cache Afsara Benazir, Zhiming Xu, Felix Xiaozhu Lin (University of Virginia) This paper addresses spoken language understanding (SLU) on microcontroller-like embedded devices, integrating on-device execution with cloud offloading in a novel fashion. We leverage temporal locality in the speech inputs to a device and reuse recent SLU inferences accordingly. Our idea is simple: let the device match incoming inputs against cached results, and only offload inputs not matched to any cached ones to the cloud for full inference. Realization of this idea, however, is non-trivial: the device needs to compare acoustic features in a robust yet low-cost way. To this end, we present SpeechCache (or SC), a speech cache for tiny devices. It matches speech inputs at two levels of representations: first by sequences of clustered raw sound units, then as sequences of phonemes. Working in tandem, the two representations offer complementary tradeoffs between cost and efficiency. To boost accuracy even further, our cache learns to personalize: with the mismatched and then offloaded inputs, it continuously finetunes the device's feature extractors with the assistance of the cloud. We implement SC on an off-the-shelf STM32 microcontroller. The complete implementation has a small memory footprint of 2MB. Evaluated on challenging speech benchmarks, our system resolves 45%-90% of inputs on device, reducing the average latency by up to 80% compared to offloading to popular cloud speech recognition services. The benefit brought by our proposed SC is notable even in adversarial settings - noisy environments, cold cache, or one device shared by a number of users.

Multi-resolution is always nice Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction https://openreview.net/forum?id=kUuKFW7DIF

Interesting dataset generated with ElevenLabs. Multiple languages in the same phrase https://huggingface.co/datasets/MohamedRashad/multilingual-tts

Diffusion-based vocoder better than Vocos https://github.com/bfs18/rfwave overall, many complains about Vocos quality but I'm not sure why. Technically it is a good architecture. Vocos problems are likely limitation of training setup, not really the architecture itself.

Interesting work to finetune Whisper to handle small context https://github.com/futo-org/whisper-acft

WeNet trained 1B mixture of experts model with good results https://arxiv.org/abs/2404.16407 U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF Xingchen Song, Di Wu, Binbin Zhang, Dinghao Zhou, Zhendong Peng, Bo Dang, Fuping Pan, Chao Yang Scale has opened new frontiers in natural language processing, but at a high cost. In response, by learning to only activate a subset of parameters in training and inference, Mixture-of-Experts (MoE) have been proposed as an energy efficient path to even larger and more capable language models and this shift towards a new generation of foundation models is gaining momentum, particularly within the field of Automatic Speech Recognition (ASR). Recent works that incorporating MoE into ASR models have complex designs such as routing frames via supplementary embedding network, improving multilingual ability for the experts, and utilizing dedicated auxiliary losses for either expert load balancing or specific language handling. We found that delicate designs are not necessary, while an embarrassingly simple substitution of MoE layers for all Feed-Forward Network (FFN) layers is competent for the ASR task. To be more specific, we benchmark our proposed model on a large scale inner-source dataset (160k hours), the results show that we can scale our baseline Conformer (Dense-225M) to its MoE counterparts (MoE-1B) and achieve Dense-1B level Word Error Rate (WER) while maintaining a Dense-225M level Real Time Factor (RTF). Furthermore, by applying Unified 2-pass framework with bidirectional attention decoders (U2++), we achieve the streaming and non-streaming decoding modes in a single MoE based model, which we call U2++ MoE. We hope that our study can facilitate the research on scaling speech foundation models without sacrificing deployment efficiency.

Interesting about Amazon Echo hardware https://www.briandorey.com/post/echo-dot-4th-gen-smart-speaker-teardown