Skip to content
The methodOpen data
Back to the collection

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Chengyi Wang, Sanyuan Chen, Yu Wu

Show all 13 authors

Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei

2023 · arXiv

Language modelsStatistics & probability
Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

alephnerd
Neutral mention · rule
> Care to give some examples In the 2010s they were the leader in NLP and the precursor of LLMs like GPT3/3.5/4/4o Machine Translation with Human Parity (2018) - https://arxiv.org/abs/1803.05567 MT-DNN (2019) - https://arxiv.org/abs/1901.11504 MASS (2019) - https://arxiv.org/abs/1905.02450 VALL-E (2023) - https://arxiv.org/abs/2301.02111 VALL-E 2 (2024) - https://arxiv.org/abs/2406.05370 While OpenAI was the first to monetize an LLM at scale via ChatGPT, it's still the early stages of this field, and there is a lot of innovation that can still be leveraged, especially in non-English language modeling, machine translation, text-to-speech, etc. It's in this segment that…
Read the full comment
Microsoft Research shines moreso than even Google Research let alone other organizations because of their strong NLP background in Chinese (Microsoft Research Asia), South Asian languages (Microsoft Research India), Arabic (Microsoft Research's older work during the Iraq War), etc.
Original comment Amazon's Secret Weapon in Chip Design Is Amazon
CSSer
Neutral mention · rule
I think your last point may hold for this genre of music but not necessarily for pop. Pop musicians are extremely versatile. They sort of have to be to stay relevant. Take Ariana Grande for example. She could sing basically anything in any style you want[0], but you may not even have to ask her because we have efficient zero-shot voice cloning now[1]. Before you wonder if this will work for music, check out this video of AI generated Eminem track[3]. [0] https://www.youtube.com/watch?v=ss9ygQqqL2Q [1] https://arxiv.org/pdf/2301.02111.pdf [2] https://aka.ms/valle (demos from one) [3] https://www.youtube.com/watch?v=WtFNOSTTPYg [4] https://github.com/NVIDIA/tacotron2 (used to create audio for 3)
Original comment I had ChatGPT write a Decemberists song
syntaxers
Neutral mention · rule
On your first point, there's some really good results from this paper: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers https://arxiv.org/abs/2301.02111 Website with examples: https://valle-demo.github.io/ For your second question, Apple is already rolling out AI-narrated audiobooks. See: https://arstechnica.com/gadgets/2023/01/apple-rolls-out-ai-n...
Original comment Will TTS and STT benefit from the AI boom?

Keep following the thread.

More in Language models

More from the same topics.