Speech Technology

https://twitter.com/JiliJeanlouis/status/1728149644683673991 At gladia_io we've tackled a major challenge that often goes under-discussed: language detection in the presence of strong accents. A common issue in speech recognition is misinterpretation caused by strong accents. Traditional systems struggle when English (or any language) is spoken with a heavy accent, often confusing it with another language.

🔊 Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models Сhat & pretrained large audio language model proposed by Alibaba Cloud. Qwen-Audio (Qwen Large Audio Language Model) - это мультимодальная версия серии больших моделей Qwen (аббревиатура Tongyi Qianwen), предложенная компанией Alibaba Cloud. Qwen-Audio принимает на вход различные звуки (человеческую речь, естественные звуки, музыку и песни) и текст, а на выходе выдает текст. Функции Qwen-Audio включают в себя: ▪Фундаментальные аудиомодели: Qwen-Audio - это фундаментальная многозадачная аудио-языковая модель, поддерживающая различные задачи, языки и типы аудио, выступающая в качестве универсальной модели понимания аудио. ▪Qwen-Audio-Chat позволяет вести полноценные диалоги . Многозадачная система обучения для всех типов аудиозаписей. Модель включает в себя более 30 задач, и обширные эксперименты показывают, что модель демонстрирует высокую производительность. ▪Результаты экспериментов показывают, что Qwen-Audio достигает впечатляющей производительности в различных эталонных задачах, не требуя тонкой настройки под конкретную задачу, и превосходит свои аналоги. В частности, Qwen-Audio достигает лучших результатов на тестовых наборах Aishell1, cochlscene, ClothoAQA и VocalSound. ▪Гибкий многозадачный чат из аудио- и текстового ввода: Qwen-Audio поддерживает анализ нескольких аудиофайлов, понимание и осмысление звука, восприятие музыки и использование инструментов для редактирования речи. 🐱 Github: https://github.com/qwenlm/qwen-audio 🚀 Demo: https://qwen-audio.github.io/Qwen-Audio/ 📕 Paper: https://arxiv.org/abs/2311.07919v1 ⏩ Dataset: https://paperswithcode.com/dataset/vocalsound @ai_machinelearning_big_data

Whisper V3 comments: https://github.com/openai/whisper/discussions/1762 Hallucinations are much more than those in v2, which was trained solely on weakly supervised original data. The hallucinated data from v2 (coming from difficult audio data) is probably propagated via new training data that is four times larger. It made sense to increase mel filters, given a larger amount of training data to capture the noisy variabilities, but hallucination-heavy parts could have caused this. A solution would have been to use compression ratio as an auxiliary measure to filter out the unwanted ones during training. I would really hope that they fix it. Thoughts @jongwook I have been using V3 for a bit now and can say with certainty that the areas of missing punctuation and capitalization, which did occur in V2 be sure, are much, much more prevalent in V3. I would guess at least 3 to 4 times more prevalent. I will definitely be going back to V2.

https://twitter.com/heiga_zen/status/1724052613132345363 Interestingly in time-domain audio generation (vocoder) the performance of diffusion is not as impressive as that in image generation. Maybe the objective function of the diffusion process is more correlated with the perceptual relevance in vision than hearing. Going by what Sander has said in his blog, and is referencing in the post here, it’s that diffusion is good at low frequency information, but not high. Eyes are sensitive to low frequency, but not high, whereas audio requires high frequencies.

Nice page to compare English TTS services

Flash Attention 2 https://twitter.com/reach_vb/status/1723810943329616007 2x Whisper speedup

This actually aligns very well with our view on speech recognition https://alphacephei.com/nsh/2023/09/22/time-brain-ctc-blank.html. High-level tokens are natural way for speech modeling https://arxiv.org/abs/2311.04753 1-step Speech Processing and Understanding Using CTC Loss Karan Singla, Shahab Jalavand, Yeon-Jun Kim, Antonio Moreno Daniel, Srinivas Bangalore, Andrej Ljolje, Ben Stern Recent studies have made some progress in refining end-to-end (E2E) speech recognition encoders by applying Connectionist Temporal Classification (CTC) loss to enhance named entity recognition within transcriptions. However, these methods have been constrained by their exclusive use of the ASCII character set, allowing only a limited array of semantic labels. Our proposed solution extends the E2E automatic speech recognition (ASR) system's vocabulary by adding a set of unused placeholder symbols, conceptually akin to the <pad> tokens used in sequence modeling. These placeholders are then assigned to represent semantic tags and are integrated into the transcription process as distinct tokens. We demonstrate notable improvements in entity tagging, intent discernment, and transcription accuracy on the SLUE benchmark and yields results that are on par with those for the SLURP dataset. Additionally, we provide a visual analysis of the system's proficiency in accurately pinpointing meaningful tokens over time, illustrating the enhancement in transcription quality through the utilization of supplementary semantic tags.

Hacking the brain wtih fMRI https://arxiv.org/abs/2311.04664 Speech language models lack important brain-relevant semantics Subba Reddy Oota, Emin Çelik, Fatma Deniz, Mariya Toneva Despite known differences between reading and listening in the brain, recent work has shown that text-based language models predict both text-evoked and speech-evoked brain activity to an impressive degree. This poses the question of what types of information language models truly predict in the brain. We investigate this question via a direct approach, in which we eliminate information related to specific low-level stimulus features (textual, speech, and visual) in the language model representations, and observe how this intervention affects the alignment with fMRI brain recordings acquired while participants read versus listened to the same naturalistic stories. We further contrast our findings with speech-based language models, which would be expected to predict speech-evoked brain activity better, provided they model language processing in the brain well. Using our direct approach, we find that both text-based and speech-based language models align well with early sensory regions due to shared low-level features. Text-based models continue to align well with later language regions even after removing these features, while, surprisingly, speech-based models lose most of their alignment. These findings suggest that speech-based models can be further improved to better reflect brain-like language processing.

https://twitter.com/honualx/status/1722250102285406699 Stereo models for all MusicGen variants (+ a new large melody both mono and stereo): 6 new models available on HuggingFace. We show how a simple fine tuning procedure with codebook interleaving takes us from boring mono to immersive stereo

From Facebook FLAP: Fast Language-Audio Pre-training Ching-Feng Yeh, Po-Yao Huang, Vasu Sharma, Shang-Wen Li, Gargi Gosh We propose Fast Language-Audio Pre-training (FLAP), a self-supervised approach that efficiently and effectively learns aligned audio and language representations through masking, contrastive learning and reconstruction. For efficiency, FLAP randomly drops audio spectrogram tokens, focusing solely on the remaining ones for self-supervision. Through inter-modal contrastive learning, FLAP learns to align paired audio and text representations in a shared latent space. Notably, FLAP leverages multiple augmented views via masking for inter-modal contrast and learns to reconstruct the masked portion of audio tokens. Moreover, FLAP leverages large language models (LLMs) to augment the text inputs, contributing to improved performance. These approaches lead to more robust and informative audio-text representations, enabling FLAP to achieve state-of-the-art (SoTA) performance on audio-text retrieval tasks on AudioCaps (achieving 53.0% R@1) and Clotho (achieving 25.5% R@1).

The 2023 SpeechHome speech technology seminar will be held in Beijing from November 18th to November 19th. At the same time, the open source speech technology exchange meeting, the 8th Kaldi Technology Exchange Meeting , will be held . Everyone is welcome to sign up. Mostly Chinese but there will be online translation. https://mp.weixin.qq.com/s/itIxsQZRNh5p2heF-lihmA

Low-latency Real-time Voice Conversion on CPU abs: https://arxiv.org/abs/2311.00873 code: https://github.com/KoeAI/LLVC The paper behind uses a GAN + knowledge distillation, trained on LibreSpeech on a single RTX 3090 GPU. Fastest real-time voice conversion on CPUs.

Scarlett Johansson hits AI app with legal action for cloning her voice in an ad Article, Comments

Overall, Espnet got some traction (and probably funding recently) https://www.wavlab.org/activities/2023/foundations/a

Following modern codec things everyone is doing discrete speech processing. In TTS and in ASR too. It has been tried before, not very successfully. I don't think this fashion will be long since there are many intrinsic reasons speech is continuous. For example, many speech details have 0.01 second scale while people look for 40ms frame rate for discrete codecs. Nvidia P-flow TTS https://pflow-demo.github.io/projects/pflow/ promises to demo that continuous inputs work better than discrete. Voice conversion also moved to something more continuous with soft-vc. Recently Espnet landed discrete ASR recipe https://github.com/espnet/espnet/tree/master/egs2/gigaspeech/asr2 WER is 12.9 which is ok but worse than continuous alternative. Best is like 10.2. Paper https://arxiv.org/abs/2305.18108 Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learning Xuankai Chang, Brian Yan, Yuya Fujita, Takashi Maekaku, Shinji Watanabe Self-supervised learning (SSL) of speech has shown impressive results in speech-related tasks, particularly in automatic speech recognition (ASR). While most methods employ the output of intermediate layers of the SSL model as real-valued features for downstream tasks, there is potential in exploring alternative approaches that use discretized token sequences. This approach offers benefits such as lower storage requirements and the ability to apply techniques from natural language processing. In this paper, we propose a new protocol that utilizes discretized token sequences in ASR tasks, which includes de-duplication and sub-word modeling to enhance the input sequence. It reduces computational cost by decreasing the length of the sequence. Our experiments on the LibriSpeech dataset demonstrate that our proposed protocol performs competitively with conventional ASR systems using continuous input features, while reducing computational and storage costs.

Google lays off employees working on its voice assistant Article, Comments

Announcing Distil-Whisper 🚀 Led by Sanchit Gandhi, we've distilled OpenAI's Whisper on 20,000 hours of open-sourced audio data. Distil-Whisper is up to 6x faster than Whisper while performing within 1% Word-Error-Rate on out-of-distribution eval sets 🎯 Why? OpenAI's Whisper yields astonishing accuracy for most audio, but it's too slow and expensive for most production use cases. In addition, it has a tendency to hallucinate. How? Encoding takes O(1) passes while decoding takes O(N) => Reducing decoder layers is N time more effective. We keep the whole encoder, but only 2 decoder layers. The encoder is frozen during distillation to ensure Whisper's robustness to noise is kept. To make sure Distil-Whisper does not inherit hallucinations, we filtered out all data samples below a certain WER threshold. By doing so, we were able to reduce hallucinations and actually beat the teacher on long-form audio evaluation 🤯 Checkpoints? License? The checkpoints will be released in two days and will be directly available in 🤗 Transformers. All with MIT License. For more information, please see: 👉 Official paper: https://github.com/huggingface/distil-whisper/blob/main/Distil_Whisper.pdf 👉 GitHub page (checkpoints will be published here): https://github.com/huggingface/distil-whisper