Skip to content

sh-lee-prml/HierSpeechpp

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation by Hierarchical Variational Inference for Zero-shot Speech Synthesis
The official implementation of HierSpeech++

Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Kim, Seong-Whan Lee

Department of Artificial Intelligence, Korea University, Seoul, Korea

Abstract

Large language models (LLM)-based speech synthesis has been widely adopted in zero-shot speech synthesis. However, they require a large-scale data and possess the same limitations as previous autoregressive speech models, including slow inference speed and lack of robustness. This paper proposes HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech (TTS) and voice conversion (VC). We verified that hierarchical speech synthesis frameworks could significantly improve the robustness and expressiveness of the synthetic speech. Furthermore, we significantly improve the naturalness and speaker similarity of synthetic speech even in zero-shot speech synthesis scenarios. For text-to-speech, we adopt the text-to-vec framework, which generates a self-supervised speech representation and an F0 representation based on text representations and prosody prompts. Then, HierSpeech++ generates speech from the generated vector, F0, and voice prompt. We further introduce a high-efficient speech super-resolution framework from 16 kHz to 48 kHz. The experimental results demonstrated that the hierarchical variational autoencoder could be a strong zero-shot speech synthesizer given that it outperforms LLM-based and diffusion-based models. Moreover, we achieved the first human-level quality zero-shot speech synthesis.

Fig1_pipeline

This repository contains:

  • 🪐 A PyTorch implementation of HierSpeech++ (TTV, Hierarchical Speech Synthesizer, SpeechSR)
  • ⚡️ Pre-trained HierSpeech++ models trained on LibriTTS (Train-460, Train-960, and more dataset)
  • Hugging Face Spaces Gradio Demo on HuggingFace. HuggingFace provides us with a community GPU grant. Thanks 😊

Previous Our Works

  • [NeurIPS2022] HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis
  • [Interspeech2023] HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer

This paper is an extension version of above papers.

Update

24.02.20

  • We get back the reconstruction loss for ttv. Adding the loss masking for zero-padding decrease the tts performance by generating a random long pause in generated speech and repeated sound(It may affect the loss balance). Sorry for the confusion. I revised it as a paper version.

24.01.19

  • We have released the TTV_v1 training code. Regardless of the language, you can train TTV using personal dataset, and perform speech synthesis using the pre-trained Hierarchical Speech Synthesizer model.

Todo

Hierarchical Speech Synthesizer

  • HierSpeechpp-Backbone (LibriTTS-train-460)
  • HierSpeechpp-Backbone (LibriTTS-train-960)
  • HierSpeechpp-Backbone-60epoch (LibriTTS-train-960, Libri-light (Medium), Expresso, MSSS(Kor), NIKL(Kor))
  • HierSpeechpp-Backbone-200epoch (LibriTTS-train-960, Libri-light (Medium), Expresso, MSSS(Kor), NIKL(Kor))

Text-to-Vec (TTV)

  • TTV-v1 (LibriTTS-train-960)
  • TTV-v2 (Multi-lingual TTV)

Speech Super-resolution (16k --> 24k or 48k)

  • SpeechSR-24k
  • SpeechSR-48k

Cleaning Up the Source Code

  • Clean Code

Training code (Will be released after paper acceptance)

  • TTV
  • Hierarchical Speech Synthesizer
  • SpeechSR

Getting Started

Pre-requisites

  1. Pytorch >=1.13 and torchaudio >= 0.13
  2. Install requirements
pip install -r requirements.txt
  1. Install Phonemizer
pip install phonemizer
sudo apt-get install espeak-ng

Checkpoint [Download]

Hierarchical Speech Synthesizer

Model Sampling Rate Params Dataset Hour Speaker Checkpoint
HierSpeech2 16 kHz 97M LibriTTS (train-460) 245 1,151 [Download]
HierSpeech2 16 kHz 97M LibriTTS (train-960) 555 2,311 [Download]
HierSpeech2 16 kHz 97M LibriTTS (train-960), Libri-light (Small, Medium), Expresso, MSSS(Kor), NIKL(Kor) 2,796 7,299 [Download]

TTV

Model Language Params Dataset Hour Speaker Checkpoint
TTV Eng 107M LibriTTS (train-960) 555 2,311 [Download]

SpeechSR

Model Sampling Rate Params Dataset Checkpoint
SpeechSR-24k 16kHz --> 24 kHz 0.13M LibriTTS (train-960), MSSS (Kor) speechsr24k
SpeechSR-48k 16kHz --> 48 kHz 0.13M MSSS (Kor), Expresso (Eng), VCTK (Eng) speechsr48k

Text-to-Speech

sh inference.sh

# --ckpt "logs/hierspeechpp_libritts460/hierspeechpp_lt460_ckpt.pth" \ LibriTTS-460
# --ckpt "logs/hierspeechpp_libritts960/hierspeechpp_lt960_ckpt.pth" \ LibriTTS-960
# --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v1_ckpt.pth" \ Large_v1 epoch 60 (paper version)
# --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v1.1_ckpt.pth" \ Large_v1.1 epoch 200 (20. Nov. 2023)

CUDA_VISIBLE_DEVICES=0 python3 inference.py \
                --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v1.1_ckpt.pth" \
                --ckpt_text2w2v "logs/ttv_libritts_v1/ttv_lt960_ckpt.pth" \
                --output_dir "tts_results_eng_kor_v2" \
                --noise_scale_vc "0.333" \
                --noise_scale_ttv "0.333" \
                --denoise_ratio "0"

  • For better robustness, we recommend a noise_scale of 0.333
  • For better expressiveness, we recommend a noise_scale of 0.667
  • Find your best parameters for your style prompt

Noise Control

# without denoiser
--denoise_ratio "0"
# with denoiser
--denoise_ratio "1"
# Mixup (Recommend 0.6~0.8)
--denoise_rate "0.8" 

Voice Conversion

  • This method only utilize a hierarchical speech synthesizer for voice conversion.
sh inference_vc.sh

# --ckpt "logs/hierspeechpp_libritts460/hierspeechpp_lt460_ckpt.pth" \ LibriTTS-460
# --ckpt "logs/hierspeechpp_libritts960/hierspeechpp_lt960_ckpt.pth" \ LibriTTS-960
# --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v1_ckpt.pth" \ Large_v1 epoch 60 (paper version)
# --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v1.1_ckpt.pth" \ Large_v1.1 epoch 200 (20. Nov. 2023)

CUDA_VISIBLE_DEVICES=0 python3 inference_vc.py \
                --ckpt "logs/hierspeechpp_eng_kor/hierspeechpp_v1.1_ckpt.pth" \
                --output_dir "vc_results_eng_kor_v2" \
                --noise_scale_vc "0.333" \
                --noise_scale_ttv "0.333" \
                --denoise_ratio "0"
  • For better robustness, we recommend a noise_scale of 0.333
  • For better expressiveness, we recommend a noise_scale of 0.667
  • Find your best parameters for your style prompt
  • Voice Conversion is vulnerable to noisy target prompt so we recommend to utilize a denoiser with noisy prompt
  • For noisy source speech, a wrong F0 may be extracted by YAPPT resulting in a quality degradation.

Speech Super-resolution

  • SpeechSR-24k and SpeechSR-48 are provided in TTS pipeline. If you want to use SpeechSR only, please refer inference_speechsr.py.
  • If you change the output resolution, add this
--output_sr "48000" # Default
--output_sr "24000" # 
--output_sr "16000" # without super-resolution.

Speech Denoising for Noise-free Speech Synthesis (Only used in Speaker Encoder during Inference)

  • For denoised style prompt, we utilize a denoiser (MP-SENet).
  • When using a long reference audio, there is an out-of-memory issue with this model so we have a plan to learn a memory efficient speech denoiser in the future.
  • If you have a problem, we recommend to use a clean reference audio or denoised audio before TTS pipeline or denoise the audio with cpu (but this will be slow😥).
  • (21, Nov. 2023) Sliced window denoising. This may reduce a burden for denoising a speech.
          if denoise == 0:
              audio = torch.cat([audio.cuda(), audio.cuda()], dim=0)
          else:
              with torch.no_grad():
                  
                  if ori_prompt_len > 80000:
                      denoised_audio = []
                      for i in range((ori_prompt_len//80000)):
                          denoised_audio.append(denoise(audio.squeeze(0).cuda()[i*80000:(i+1)*80000], denoiser, hps_denoiser))
                      
                      denoised_audio.append(denoise(audio.squeeze(0).cuda()[(i+1)*80000:], denoiser, hps_denoiser))
                      denoised_audio = torch.cat(denoised_audio, dim=1)
                  else:
                      denoised_audio = denoise(audio.squeeze(0).cuda(), denoiser, hps_denoiser)
    
              audio = torch.cat([audio.cuda(), denoised_audio[:,:audio.shape[-1]]], dim=0)
    

TTV-v2 (WIP)

  • TTV-v1 is a simple model which is very slightly modified from VITS. Although this simple TTV could synthesize a speech with high-quality and high speaker similarity, we thought that there is room for improvement in terms of expressiveness such as prosody modeling.
  • For TTV-v2, we modify some components and training process (Model size: 107M --> 278M)
    1. Intermediate hidden size: 256 --> 384
    2. Loss masking for wav2vec reconstruction loss (I left out masking the loss for zero-padding sequences😥)
    3. For long sentence generation, we finetune the model with full LibriTTS-train dataset without data filtering (Decrease the learning rate to 2e-5 with batch size of 8 per gpus)
    4. Multi-lingual Dataset (We are training the model with Eng, Indic, and Kor dataset now)

GAN VS Diffusion

[Read More] We think that we could not confirm which is better yet. There are many advatanges for each model so you can utilize each model for your own purposes and each study must be actively conducted simultaneously.

GAN (Specifically, GAN-based End-to-End Speech Synthesis Models)

  • (pros) Fast Inference Speed
  • (pros) High-quality Audio
  • (cons) Slow Training Speed (Over 7~20 Days)
  • (cons) Lower Voice Style Transfer Performance than Diffusion Models
  • (cons) Perceptually High-quality but Over-smoothed Audio because of Information Bottleneck by the sampling from the low-dimensional Latent Variable

Diffusion (Diffusion-based Mel-spectrogram Generation Models)

  • (pros) Fast Training Speed (within 3 Days)
  • (pros) High-quality Voice Style Transfer
  • (cons) Slow Inference Speed
  • (cons) Lower Audio quality than End-to-End Speech Synthesis Models

(In this wors) Our Approaches for GAN-based End-to-End Speech Synthesis Models

  • Improving Voice Style Transfer Performance in End-to-End Speech Synthesis Models for OOD (Zero-shot Voice Style Transfer for Novel Speaker)
  • Improving the Audio Quality beyond Perceptal Quality for Much more High-fidelity Audio Generation

(Our other works) Diffusion-based Mel-spectrogram Generation Models

  • DDDM-VC: Disentangled Denoising Diffusion Models for High-quality and High-diversity Speech Synthesis Models
  • Diff-hierVC: Hierarhical Diffusion-based Speech Synthesis Model with Diffusion-based Pitch Modeling

Our Goals

  • Integrating each model for High-quality, High-diversity and High-fidelity Speech Synthesis Models

LLM-based Models

We hope to compare LLM-based models for zero-shot TTS baselines. However, there is no public-available official implementation of LLM-based TTS models. Unfortunately, unofficial models have a poor performance in zero-shot TTS so we hope they will release their model for a fair comparison and reproducibility and for our speech community. THB I could not stand the inference speed almost 1,000 times slower than e2e models It takes 5 days to synthesize the full sentences of LibriTTS-test subsets. Even, the audio quality is so bad. I hope they will release their official source code soon.

In my very personal opinion, VITS is still the best TTS model I have ever seen. But, I acknowledge that LLM-based models have much powerful potential for their creative generative performance from the large-scale dataset but not now.

Limitation of our work

  • Slow training speed and Relatively large model size (Compared with VITS) --> Future work: Light-weight and Fast training pipeline and much larger model...
  • Could not generate realistic background sound --> Future work: adding audio generation part by disentangling speech and sound.
  • Could not generate a speech from a too long sentence becauase of our training setting. We see increasing max length could improve the model performance. I hope to use GPUs with 80 GB 😢
     # Data Filtering for limited computation resource. 
      wav_min = 32
      wav_max = 600 # 12s 
      text_min = 1
      text_max = 200
    

TTV v2 may reduce this issue significantly...!

Results [Download]

We have attached all samples from LibriTTS test-clean and test-other.

Reference

Our repository is heavily based on VITS and BigVGAN.

[Read More]

Our Previous Works

Baseline Model

Waveform Generator for High-quality Audio Generation

Self-supervised Speech Model

Other Large Language Model based Speech Synthesis Model

  • VALL-E & VALL-E-X
  • SPEAR-TTS
  • Make-a-Voice
  • MEGA-TTS & MEGA-TTS 2
  • UniAudio

Diffusion-based Model

  • NaturalSpeech 2

AdaLN-zero

Thanks for all nice works.

About

The official implementation of HierSpeech++

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages