Coqui TTS

What's Changed

• fix: Few typos in Tortoise docs. by @VladCuciureanu in https://github.com/coqui-ai/TTS/pull/3352
• fix pause problem of Chinese speech by @aaron-lii in https://github.com/coqui-ai/TTS/pull/3351
• Fix typos by @omahs in https://github.com/coqui-ai/TTS/pull/3368
• Print message for either commercial license or CPML by @JRMeyer in https://github.com/coqui-ai/TTS/pull/3381
• Add inference parameters by @WeberJulian in https://github.com/coqui-ai/TTS/pull/3373
• Training fastspeech2 with External Speaker Embeddings by @freds0 in https://github.com/coqui-ai/TTS/pull/3404
• fixes a typo by @joelhoward0 in https://github.com/coqui-ai/TTS/pull/3392
• support multiple GPU training for XTTS by @aaron-lii in https://github.com/coqui-ai/TTS/pull/3391
• Add studio speakers to open source XTTS! by @WeberJulian in https://github.com/coqui-ai/TTS/pull/3405

New Contributors

• @VladCuciureanu made their first contribution in https://github.com/coqui-ai/TTS/pull/3352
• @aaron-lii made their first contribution in https://github.com/coqui-ai/TTS/pull/3351
• @omahs made their first contribution in https://github.com/coqui-ai/TTS/pull/3368
• @JRMeyer made their first contribution in https://github.com/coqui-ai/TTS/pull/3381
• @freds0 made their first contribution in https://github.com/coqui-ai/TTS/pull/3404
• @joelhoward0 made their first contribution in https://github.com/coqui-ai/TTS/pull/3392

Full Changelog: https://github.com/coqui-ai/TTS/compare/v0.21.3...v0.22.0

🐸Coqui.ai News

• 📣 ⓍTTSv2 is here with 16 languages and better performance across the board.
• 📣 ⓍTTS fine-tuning code is out. Check the example recipes.
• 📣 ⓍTTS can now stream with <200ms latency.
• 📣 ⓍTTS, our production TTS model that can speak 13 languages, is released Blog Post, Demo, Docs
• 📣 🐶Bark is now available for inference with unconstrained voice cloning. Docs
• 📣 You can use ~1100 Fairseq models with 🐸TTS.
• 📣 🐸TTS now supports 🐢Tortoise with faster inference. Docs

<div align="center">

<img src="https://static.scarf.sh/a.png?x-pxid=cf317fe7-2188-4721-bc01-124bb5d5dbb2" />

<img src="https://raw.githubusercontent.com/coqui-ai/TTS/main/images/coqui-log-green-TTS.png" height="56"/>

🐸TTS is a library for advanced Text-to-Speech generation.

🚀 Pretrained models in +1100 languages.

🛠️ Tools for training new models and fine-tuning existing models in any language.

📚 Utilities for dataset analysis and curation.

---

Discord

License

PyPI version

Covenant

Downloads

DOI

GithubActions

GithubActions

GithubActions

GithubActions

GithubActions

GithubActions

GithubActions

GithubActions

GithubActions

GithubActions

GithubActions

Docs

</div>

---

💬 Where to ask questions

Please use our dedicated channels for questions and discussion. Help is much more valuable if it's shared publicly so that more people can benefit from it.

| Type | Platforms |

| ------------------------------- | --------------------------------------- |

| 🚨 Bug Reports | [GitHub Issue Tracker] |

| 🎁 Feature Requests & Ideas | [GitHub Issue Tracker] |

| 👩‍💻 Usage Questions | [GitHub Discussions] |

| 🗯 General Discussion | [GitHub Discussions] or [Discord] |

[github issue tracker]: https://github.com/coqui-ai/tts/issues

[github discussions]: https://github.com/coqui-ai/TTS/discussions

[discord]: https://discord.gg/5eXr5seRrv

[Tutorials and Examples]: https://github.com/coqui-ai/TTS/wiki/TTS-Notebooks-and-Tutorials

🔗 Links and Resources

| Type | Links |

| ------------------------------- | --------------------------------------- |

| 💼 Documentation | ReadTheDocs

| 💾 Installation | TTS/README.md|

| 👩‍💻 Contributing | CONTRIBUTING.md|

| 📌 Road Map | Main Development Plans

| 🚀 Released Models | TTS Releases and Experimental Models|

| 📰 Papers | TTS Papers|

🥇 TTS Performance

<p align="center"><img src="https://raw.githubusercontent.com/coqui-ai/TTS/main/images/TTS-performance.png" width="800" /></p>

Underlined "TTS" and "Judy" are internal 🐸TTS models that are not released open-source. They are here to show the potential. Models prefixed with a dot (.Jofish .Abe and .Janice) are real human voices.

Features

• High-performance Deep Learning models for Text2Speech tasks.
• Text2Spec models (Tacotron, Tacotron2, Glow-TTS, SpeedySpeech).
• Speaker Encoder to compute speaker embeddings efficiently.
• Vocoder models (MelGAN, Multiband-MelGAN, GAN-TTS, ParallelWaveGAN, WaveGrad, WaveRNN)
• Fast and efficient model training.
• Detailed training logs on the terminal and Tensorboard.
• Support for Multi-speaker TTS.
• Efficient, flexible, lightweight but feature complete Trainer API.
• Released and ready-to-use models.
• Tools to curate Text2Speech datasets under``dataset_analysis``.
• Utilities to use and test your models.
• Modular (but not too much) code base enabling easy implementation of new ideas.

Model Implementations

Spectrogram models

• Tacotron: paper
• Tacotron2: paper
• Glow-TTS: paper
• Speedy-Speech: paper
• Align-TTS: paper
• FastPitch: paper
• FastSpeech: paper
• FastSpeech2: paper
• SC-GlowTTS: paper
• Capacitron: paper
• OverFlow: paper
• Neural HMM TTS: paper
• Delightful TTS: paper

End-to-End Models

• ⓍTTS: blog
• VITS: paper
• 🐸 YourTTS: paper
• 🐢 Tortoise: orig. repo
• 🐶 Bark: orig. repo

Attention Methods

• Guided Attention: paper
• Forward Backward Decoding: paper
• Graves Attention: paper
• Double Decoder Consistency: blog
• Dynamic Convolutional Attention: paper
• Alignment Network: paper

Speaker Encoder

• GE2E: paper
• Angular Loss: paper

Vocoders

• MelGAN: paper
• MultiBandMelGAN: paper
• ParallelWaveGAN: paper
• GAN-TTS discriminators: paper
• WaveRNN: origin
• WaveGrad: paper
• HiFiGAN: paper
• UnivNet: paper

Voice Conversion

• FreeVC: paper

You can also help us implement more models.

Installation

🐸TTS is tested on Ubuntu 18.04 with python >= 3.9, < 3.12..

If you are only interested in synthesizing speech with the released 🐸TTS models, installing from PyPI is the easiest option.

pip install TTS

If you plan to code or train models, clone 🐸TTS and install it locally.

git clone https://github.com/coqui-ai/TTS
pip install -e .[all,dev,notebooks]  # Select the relevant extras

If you are on Ubuntu (Debian), you can also run following commands for installation.

$ make system-deps  # intended to be used on Ubuntu (Debian). Let us know if you have a different OS.
$ make install

If you are on Windows, 👑@GuyPaddock wrote installation instructions here.

Docker Image

You can also try TTS without install with the docker image.

Simply run the following command and you will be able to run TTS without installing it.

docker run --rm -it -p 5002:5002 --entrypoint /bin/bash ghcr.io/coqui-ai/tts-cpu
python3 TTS/server/server.py --list_models #To get the list of available models
python3 TTS/server/server.py --model_name tts_models/en/vctk/vits # To start a server

You can then enjoy the TTS server here

More details about the docker images (like GPU support) can be found here

Synthesizing speech by 🐸TTS

🐍 Python API

Running a multi-speaker and multi-lingual model

import torch
from TTS.api import TTS

# Get device
device = "cuda" if torch.cuda.is_available() else "cpu"

# List available 🐸TTS models
print(TTS().list_models())

# Init TTS
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)

# Run TTS
# ❗ Since this model is multi-lingual voice cloning model, we must set the target speaker_wav and language
# Text to speech list of amplitude values as output
wav = tts.tts(text="Hello world!", speaker_wav="my/cloning/audio.wav", language="en")
# Text to speech to a file
tts.tts_to_file(text="Hello world!", speaker_wav="my/cloning/audio.wav", language="en", file_path="output.wav")

Running a single speaker model

# Init TTS with the target model name
tts = TTS(model_name="tts_models/de/thorsten/tacotron2-DDC", progress_bar=False).to(device)

# Run TTS
tts.tts_to_file(text="Ich bin eine Testnachricht.", file_path=OUTPUT_PATH)

# Example voice cloning with YourTTS in English, French and Portuguese
tts = TTS(model_name="tts_models/multilingual/multi-dataset/your_tts", progress_bar=False).to(device)
tts.tts_to_file("This is voice cloning.", speaker_wav="my/cloning/audio.wav", language="en", file_path="output.wav")
tts.tts_to_file("C'est le clonage de la voix.", speaker_wav="my/cloning/audio.wav", language="fr-fr", file_path="output.wav")
tts.tts_to_file("Isso é clonagem de voz.", speaker_wav="my/cloning/audio.wav", language="pt-br", file_path="output.wav")

Example voice conversion

Converting the voice in source_wav to the voice of target_wav

tts = TTS(model_name="voice_conversion_models/multilingual/vctk/freevc24", progress_bar=False).to("cuda")
tts.voice_conversion_to_file(source_wav="my/source.wav", target_wav="my/target.wav", file_path="output.wav")

Example voice cloning together with the voice conversion model.

This way, you can clone voices by using any model in 🐸TTS.

tts = TTS("tts_models/de/thorsten/tacotron2-DDC")
tts.tts_with_vc_to_file(
    "Wie sage ich auf Italienisch, dass ich dich liebe?",
    speaker_wav="target/speaker.wav",
    file_path="output.wav"
)

Example text to speech using Fairseq models in ~1100 languages 🤯.

For Fairseq models, use the following name format: tts_models/<lang-iso_code>/fairseq/vits.

You can find the language ISO codes here

and learn about the Fairseq models here.

# TTS with on the fly voice conversion
api = TTS("tts_models/deu/fairseq/vits")
api.tts_with_vc_to_file(
    "Wie sage ich auf Italienisch, dass ich dich liebe?",
    speaker_wav="target/speaker.wav",
    file_path="output.wav"
)

Command-line tts

<!-- begin-tts-readme -->

Synthesize speech on command line.

You can either use your trained model or choose a model from the provided list.

If you don't specify any models, then it uses LJSpeech based English model.

Single Speaker Models

• List provided models:

  $ tts --list_models

• Get model info (for both tts_models and vocoder_models):

• Query by type/name:

The model_info_by_name uses the name as it from the --list_models.

    $ tts --model_info_by_name "<model_type>/<language>/<dataset>/<model_name>"

For example:

    $ tts --model_info_by_name tts_models/tr/common-voice/glow-tts
    $ tts --model_info_by_name vocoder_models/en/ljspeech/hifigan_v2

• Query by type/idx:

The model_query_idx uses the corresponding idx from --list_models.

    $ tts --m