Speech Technology

Btw, you can use them from Python I suppose, no API keys required https://pypi.org/project/edge-tts/

New voices from Microsoft powered by LLMs https://twitter.com/shengzhao8/status/1748868829332283795 Honestly, a bit flat voice in my opinion, not really conversational style

Some other popular discord servers, let me know if you know others: Coqui https://discord.gg/5eXr5seRrv Mistral https://discord.gg/mistralai StyleTTS2 https://discord.gg/ha8sxdG2K4 RVC https://discord.gg/HcsmBBGyVk

Discord is getting very popular among speech community recently. Some Vosk links, so I don't forget about them. No need to join, I don't plan it to be very active anyway https://discord.gg/Bpmd2qXp - discord https://matrix.to/#/#alphacep_community:gitter.im - gitter https://twitter.com/alphacep - twitter https://www.reddit.com/r/speechtech - reddit

Some latest features in GAN-based TTS https://alphacephei.com/nsh/2024/01/19/30-tts-details.html

People say https://github.com/RVC-Boss/GPT-SoVITS is very good

And LLM is still relevant for many speech tasks. This paper from Google https://arxiv.org/abs/2401.03506v1 DiarizationLM: Speaker Diarization Post-Processing with Large Language Models Quan Wang, Yiling Huang, Guanlong Zhao, Evan Clark, Wei Xia, Hank Liao In this paper, we introduce DiarizationLM, a framework to leverage large language models (LLM) to post-process the outputs from a speaker diarization system. Various goals can be achieved with the proposed framework, such as improving the readability of the diarized transcript, or reducing the word diarization error rate (WDER). In this framework, the outputs of the automatic speech recognition (ASR) and speaker diarization systems are represented as a compact textual format, which is included in the prompt to an optionally finetuned LLM. The outputs of the LLM can be used as the refined diarization results with the desired enhancement. As a post-processing step, this framework can be easily applied to any off-the-shelf ASR and speaker diarization systems without retraining existing components. Our experiments show that a finetuned PaLM 2-S model can reduce the WDER by rel. 25.9% on the Fisher telephone conversation dataset, and rel. 31% on the Callhome English dataset.

Interesting overview of the tech, not much new here though. GSS, conformers, hubert https://arxiv.org/abs/2401.03473 ICMC-ASR: The ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition Challenge He Wang, Pengcheng Guo, Yue Li, Ao Zhang, Jiayao Sun, Lei Xie, Wei Chen, Pan Zhou, Hui Bu, Xin Xu, Binbin Zhang, Zhuo Chen, Jian Wu, Longbiao Wang, Eng Siong Chng, Sun Li To promote speech processing and recognition research in driving scenarios, we build on the success of the Intelligent Cockpit Speech Recognition Challenge (ICSRC) held at ISCSLP 2022 and launch the ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition (ICMC-ASR) Challenge. This challenge collects over 100 hours of multi-channel speech data recorded inside a new energy vehicle and 40 hours of noise for data augmentation. Two tracks, including automatic speech recognition (ASR) and automatic speech diarization and recognition (ASDR) are set up, using character error rate (CER) and concatenated minimum permutation character error rate (cpCER) as evaluation metrics, respectively. Overall, the ICMC-ASR Challenge attracts 98 participating teams and receives 53 valid results in both tracks. In the end, first-place team USTCiflytek achieves a CER of 13.16% in the ASR track and a cpCER of 21.48% in the ASDR track, showing an absolute improvement of 13.08% and 51.4% compared to our challenge baseline, respectively.

On continuation of the above a paper which also explores local and global convolutions https://arxiv.org/abs/2401.08342 ECAPA2: A Hybrid Neural Network Architecture and Training Strategy for Robust Speaker Embeddings Jenthe Thienpondt, Kris Demuynck In this paper, we present ECAPA2, a novel hybrid neural network architecture and training strategy to produce robust speaker embeddings. Most speaker verification models are based on either the 1D- or 2D-convolutional operation, often manifested as Time Delay Neural Networks or ResNets, respectively. Hybrid models are relatively unexplored without an intuitive explanation what constitutes best practices in regard to its architectural choices. We motivate the proposed ECAPA2 model in this paper with an analysis of current speaker verification architectures. In addition, we propose a training strategy which makes the speaker embeddings more robust against overlapping speech and short utterance lengths. The presented ECAPA2 architecture and training strategy attains state-of-the-art performance on the VoxCeleb1 test sets with significantly less parameters than current models. Finally, we make a pre-trained model publicly available to promote research on downstream tasks.

https://twitter.com/chimechallenge/status/1731345235744088352 It gives us great pleasure to pre-announce the 8th CHiME Speech Separation and Recognition Challenge (CHiME-8) that will launch in February 2024. CHiME-8 TASKS includes: Task 1 - DASR Task 2 - NOTSOFAR-1 Task 3 - MMCSG Please check https://www.chimechallenge.org/#current

I don't really believe in simplified attention, one can easily prove it doesn't work with multispeaker inputs. But still somewhat interesting https://github.com/SamsungLabs/SummaryMixing

Merging Whisper and Mistral This is the first part of many posts I am writing to consolidate learnings on how to finetune Large Language Models (LLMs) to process audio, with the eventual goal of being able to build and host a LLM able to describe human voices. https://paul.mou.dev/posts/2023-12-31-listening-with-llm/

https://twitter.com/slseanwu/status/1745441453201879154 (1/n) We improve audio captioning performance by leveraging pretrained models & LLMs everywhere in our system, including (i) audio encoder, (ii) auxiliary supervision, and (iii) mix-up data augmentation. https://github.com/slSeanWU/beats-conformer-bart-audio-captioner

From Alibaba 10k speakers https://github.com/alibaba-damo-academy/3D-Speaker https://3dspeaker.github.io/

From Alibaba 10k speakers https://3dspeaker.github.io/

100% truth https://www.reddit.com/r/speechtech/comments/18y708e/comment/kg9w084/ Sorry to hear and I wish everyone the best. I have stopped working on TTS recently, after a decade in the field. I still struggle to let go but it's nice to work on other things again. It's a technically rewarding field but surviving is hard. Customer expectations rise all the time and no matter how good the current state is, there is always a word somewhere where they don't like the pronunciation, the prosody or the noise. And you always have to blindly play them recordings as well, otherwise they always find something. And then are embarrassed when they find out they actually preferred a synthetic sample ;). And they flock from one hype company to the next, I remember when I heard Lyrebird all day long from clients and investors. Last months I felt so tired of hearing Elevenlabs all the time. Seems to also slowly die down again and I heard the first "not so good" statements again. "Robotic" - our all favorite word ;), applied to everything that's a few months old. And eventually for each company there is enough legacy to maintain... and then the next startup will come, again act as if they were the first to do voice cloning and hide the "for an authentic clone you need more than 3 minutes of data though" behind some footnote or in the FAQ ;). And people flock over till they are unhappy again. But the worst thing is what too many people do with our technology. I did my PhD so we can give the blind better tech, to give the voiceless a voice. Not to have Emma Watson say naughty things. I see why James Betker got sick of it when Tortoise became 4chan's darling. With audio foundation models I also feel the time of the small tinkerers with one GPU at home :) will gradually come to an end. Similarly to LLMs we will probably see a lot of work on inference, all kinds of adaptation mechanisms but just kicking off a training from scratch on a single GPU and be done in a day... I'm not sure but "more data, more parameters" just sounds natural for speech as well. It's been a ride, I've started out with unit selection but soon HMM-based synthesis when there was HTK/HTS and the Festival/Flite tracks and for vocoders you chose between STRAIGHT or hts_engine (thanks to the person who wrote WORLD later on btw). The time of 120 steps perl scripts calling HTS cli tools, with festival Scheme files mixed in... and Tcl... and weird shell scripts calling C libraries. To Theano, Keras, Tensorflow, finally Pytorch. I remember writing my first LSTMs manually in C++ and my own model format to run synthesis on mobile before we had Tensorflow Lite/Mobile, ONNX, Pytorch etc. I've also ported some of my stuff to Rust at some point ;). Well, I did not mean to hijack this topic. Again, I wish you all the best. We all get thrown into a competition in a very "winner-takes-all" market that got a new winner every few years. While in the end we all just want to build cool technology that people like to use.