Audio synthesis with Forward Transformer TTS and WaveRNN Vocoder

Installing dependencies and pre-trained models

# Clone the Transformer TTS and WaveRNN repos
!git clone https://github.com/as-ideas/TransformerTTS.git
!cd TransformerTTS && git checkout 1c1cb03 && cd ..
!git clone https://github.com/fatchord/WaveRNN

# Install requirements
!apt-get install -y espeak
!pip install -r TransformerTTS/requirements.txt

# Download the transformer pre-trained weights
! wget https://public-asai-dl-models.s3.eu-central-1.amazonaws.com/TransformerTTS/ljspeech_wavernn_forward_transformer.zip
! unzip -o ljspeech_wavernn_forward_transformer.zip

# Unzip the wave pretrained model
!unzip -o WaveRNN/pretrained/ljspeech.wavernn.mol.800k.zip -d WaveRNN/pretrained/

Cloning into 'TransformerTTS'...
remote: Enumerating objects: 4107, done.
remote: Counting objects: 100% (646/646), done.
remote: Compressing objects: 100% (214/214), done.
remote: Total 4107 (delta 456), reused 611 (delta 431), pack-reused 3461
Receiving objects: 100% (4107/4107), 26.00 MiB | 25.43 MiB/s, done.
Resolving deltas: 100% (2826/2826), done.
Note: checking out '1c1cb03'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 1c1cb03 Add Audio class. WaveRNN and MelGAN compatible normalizations.
Cloning into 'WaveRNN'...
remote: Enumerating objects: 928, done.
remote: Total 928 (delta 0), reused 0 (delta 0), pack-reused 928
Receiving objects: 100% (928/928), 242.13 MiB | 34.46 MiB/s, done.
Resolving deltas: 100% (525/525), done.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  espeak-data libespeak1 libportaudio2 libsonic0
The following NEW packages will be installed:
  espeak espeak-data libespeak1 libportaudio2 libsonic0
0 upgraded, 5 newly installed, 0 to remove and 39 not upgraded.
Need to get 1,219 kB of archives.
After this operation, 3,031 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libportaudio2 amd64 19.6.0-1 [64.6 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsonic0 amd64 0.2.0-6 [13.4 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 espeak-data amd64 1.48.04+dfsg-5 [934 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libespeak1 amd64 1.48.04+dfsg-5 [145 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 espeak amd64 1.48.04+dfsg-5 [61.6 kB]
Fetched 1,219 kB in 1s (927 kB/s)
Selecting previously unselected package libportaudio2:amd64.
(Reading database ... 160772 files and directories currently installed.)
Preparing to unpack .../libportaudio2_19.6.0-1_amd64.deb ...
Unpacking libportaudio2:amd64 (19.6.0-1) ...
Selecting previously unselected package libsonic0:amd64.
Preparing to unpack .../libsonic0_0.2.0-6_amd64.deb ...
Unpacking libsonic0:amd64 (0.2.0-6) ...
Selecting previously unselected package espeak-data:amd64.
Preparing to unpack .../espeak-data_1.48.04+dfsg-5_amd64.deb ...
Unpacking espeak-data:amd64 (1.48.04+dfsg-5) ...
Selecting previously unselected package libespeak1:amd64.
Preparing to unpack .../libespeak1_1.48.04+dfsg-5_amd64.deb ...
Unpacking libespeak1:amd64 (1.48.04+dfsg-5) ...
Selecting previously unselected package espeak.
Preparing to unpack .../espeak_1.48.04+dfsg-5_amd64.deb ...
Unpacking espeak (1.48.04+dfsg-5) ...
Setting up libportaudio2:amd64 (19.6.0-1) ...
Setting up espeak-data:amd64 (1.48.04+dfsg-5) ...
Setting up libsonic0:amd64 (0.2.0-6) ...
Setting up libespeak1:amd64 (1.48.04+dfsg-5) ...
Setting up espeak (1.48.04+dfsg-5) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.2) ...
/sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from -r TransformerTTS/requirements.txt (line 1)) (3.2.2)
Requirement already satisfied: librosa>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from -r TransformerTTS/requirements.txt (line 2)) (0.8.0)
Requirement already satisfied: numpy>=1.17.4 in /usr/local/lib/python3.7/dist-packages (from -r TransformerTTS/requirements.txt (line 3)) (1.19.5)
Collecting phonemizer==2.1
  Downloading https://files.pythonhosted.org/packages/d3/82/666045375029df9c2f274923539f43346a7b7abc349b02e33dff585da56f/phonemizer-2.1-py3-none-any.whl (47kB)
     |████████████████████████████████| 51kB 4.6MB/s 
Collecting ruamel.yaml>=0.16.6
  Downloading https://files.pythonhosted.org/packages/9e/00/1ba32614cc9572fd6e98dbfdf642f55f9c5ed8a89ab9328d2ce6f39e6fb3/ruamel.yaml-0.17.7-py3-none-any.whl (108kB)
     |████████████████████████████████| 112kB 14.5MB/s 
Requirement already satisfied: tensorflow>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from -r TransformerTTS/requirements.txt (line 6)) (2.5.0)
Requirement already satisfied: tqdm>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from -r TransformerTTS/requirements.txt (line 7)) (4.41.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->-r TransformerTTS/requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->-r TransformerTTS/requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->-r TransformerTTS/requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->-r TransformerTTS/requirements.txt (line 1)) (1.3.1)
Requirement already satisfied: scikit-learn!=0.19.0,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (0.22.2.post1)
Requirement already satisfied: audioread>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (2.1.9)
Requirement already satisfied: resampy>=0.2.2 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (0.2.2)
Requirement already satisfied: soundfile>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (0.10.3.post1)
Requirement already satisfied: pooch>=1.0 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (1.3.0)
Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (1.4.1)
Requirement already satisfied: numba>=0.43.0 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (0.51.2)
Requirement already satisfied: decorator>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (4.4.2)
Requirement already satisfied: joblib>=0.14 in /usr/local/lib/python3.7/dist-packages (from librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (1.0.1)
Collecting segments
  Downloading https://files.pythonhosted.org/packages/1e/ae/02d31d73cfc3fa1dc74b7b7f14820fadc287e74406583d7af7b80fcaac41/segments-2.2.0-py2.py3-none-any.whl
Requirement already satisfied: attrs>=18.1 in /usr/local/lib/python3.7/dist-packages (from phonemizer==2.1->-r TransformerTTS/requirements.txt (line 4)) (21.2.0)
Collecting ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.10"
  Downloading https://files.pythonhosted.org/packages/5e/6e/f652c56bbb2c3d3fca252ffc7c0358597f57a1bbdf484dac683054950c63/ruamel.yaml.clib-0.2.2-cp37-cp37m-manylinux1_x86_64.whl (547kB)
     |████████████████████████████████| 552kB 15.7MB/s 
Requirement already satisfied: gast==0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (0.4.0)
Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (0.12.0)
Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.12)
Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (0.2.0)
Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (3.12.4)
Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (0.36.2)
Requirement already satisfied: grpcio~=1.34.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.34.1)
Requirement already satisfied: tensorflow-estimator<2.6.0,>=2.5.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (2.5.0)
Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.1.2)
Requirement already satisfied: six~=1.15.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.15.0)
Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.12.1)
Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.6.3)
Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.1.0)
Requirement already satisfied: tensorboard~=2.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (2.5.0)
Requirement already satisfied: keras-nightly~=2.5.0.dev in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (2.5.0.dev2021032900)
Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (3.7.4.3)
Requirement already satisfied: h5py~=3.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (3.1.0)
Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (3.3.0)
Requirement already satisfied: cffi>=1.0 in /usr/local/lib/python3.7/dist-packages (from soundfile>=0.9.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (1.14.5)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from pooch>=1.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (2.23.0)
Requirement already satisfied: appdirs in /usr/local/lib/python3.7/dist-packages (from pooch>=1.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (1.4.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from pooch>=1.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (20.9)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.43.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (57.0.0)
Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.43.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (0.34.0)
Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from segments->phonemizer==2.1->-r TransformerTTS/requirements.txt (line 4)) (2019.12.20)
Collecting clldutils>=1.7.3
  Downloading https://files.pythonhosted.org/packages/f7/99/3ea7e3595e730332c2799938e2dad456916772e571fa0cd8dcdfb9d5780a/clldutils-3.9.0-py2.py3-none-any.whl (195kB)
     |████████████████████████████████| 204kB 18.3MB/s 
Collecting csvw>=1.5.6
  Downloading https://files.pythonhosted.org/packages/55/ae/afb43a6b88c4202d29e4ec7aca76633d8c530140f4f5a32ee762d07c4607/csvw-1.11.0-py2.py3-none-any.whl
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.0.1)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (0.6.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (0.4.4)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.8.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.30.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (3.3.4)
Requirement already satisfied: cached-property; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from h5py~=3.1.0->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.5.2)
Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.0->soundfile>=0.9.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (2.20)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->pooch>=1.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->pooch>=1.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->pooch>=1.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (2020.12.5)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->pooch>=1.0->librosa>=0.7.1->-r TransformerTTS/requirements.txt (line 2)) (2.10)
Collecting colorlog
  Downloading https://files.pythonhosted.org/packages/32/e6/e9ddc6fa1104fda718338b341e4b3dc31cd8039ab29e52fc73b508515361/colorlog-5.0.1-py2.py3-none-any.whl
Requirement already satisfied: tabulate>=0.7.7 in /usr/local/lib/python3.7/dist-packages (from clldutils>=1.7.3->segments->phonemizer==2.1->-r TransformerTTS/requirements.txt (line 4)) (0.8.9)
Requirement already satisfied: uritemplate>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from csvw>=1.5.6->segments->phonemizer==2.1->-r TransformerTTS/requirements.txt (line 4)) (3.0.1)
Collecting rfc3986
  Downloading https://files.pythonhosted.org/packages/c4/e5/63ca2c4edf4e00657584608bee1001302bbf8c5f569340b78304f2f446cb/rfc3986-1.5.0-py2.py3-none-any.whl
Collecting isodate
  Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB)
     |████████████████████████████████| 51kB 7.9MB/s 
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (1.3.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (4.2.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.6" in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (4.7.2)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (4.0.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (3.1.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (0.4.8)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard~=2.5->tensorflow>=2.2.0->-r TransformerTTS/requirements.txt (line 6)) (3.4.1)
Installing collected packages: colorlog, rfc3986, isodate, csvw, clldutils, segments, phonemizer, ruamel.yaml.clib, ruamel.yaml
Successfully installed clldutils-3.9.0 colorlog-5.0.1 csvw-1.11.0 isodate-0.6.0 phonemizer-2.1 rfc3986-1.5.0 ruamel.yaml-0.17.7 ruamel.yaml.clib-0.2.2 segments-2.2.0
--2021-06-07 01:11:12--  https://public-asai-dl-models.s3.eu-central-1.amazonaws.com/TransformerTTS/ljspeech_wavernn_forward_transformer.zip
Resolving public-asai-dl-models.s3.eu-central-1.amazonaws.com (public-asai-dl-models.s3.eu-central-1.amazonaws.com)... 52.219.168.9
Connecting to public-asai-dl-models.s3.eu-central-1.amazonaws.com (public-asai-dl-models.s3.eu-central-1.amazonaws.com)|52.219.168.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 210039489 (200M) [application/zip]
Saving to: ‘ljspeech_wavernn_forward_transformer.zip’

ljspeech_wavernn_fo 100%[===================>] 200.31M  21.1MB/s    in 11s     

2021-06-07 01:11:23 (19.0 MB/s) - ‘ljspeech_wavernn_forward_transformer.zip’ saved [210039489/210039489]

Archive:  ljspeech_wavernn_forward_transformer.zip
   creating: ljspeech_wavernn_forward_transformer/
  inflating: __MACOSX/._ljspeech_wavernn_forward_transformer  
  inflating: ljspeech_wavernn_forward_transformer/.DS_Store  
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/._.DS_Store  
   creating: ljspeech_wavernn_forward_transformer/wavernn/
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/._wavernn  
  inflating: ljspeech_wavernn_forward_transformer/wavernn/.DS_Store  
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/wavernn/._.DS_Store  
  inflating: ljspeech_wavernn_forward_transformer/wavernn/forward_config.yaml  
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/wavernn/._forward_config.yaml  
   creating: ljspeech_wavernn_forward_transformer/wavernn/forward_weights/
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/wavernn/._forward_weights  
  inflating: ljspeech_wavernn_forward_transformer/wavernn/data_config.yaml  
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/wavernn/._data_config.yaml  
  inflating: ljspeech_wavernn_forward_transformer/wavernn/forward_weights/checkpoint  
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/wavernn/forward_weights/._checkpoint  
  inflating: ljspeech_wavernn_forward_transformer/wavernn/forward_weights/ckpt-133.index  
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/wavernn/forward_weights/._ckpt-133.index  
  inflating: ljspeech_wavernn_forward_transformer/wavernn/forward_weights/ckpt-133.data-00001-of-00002  
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/wavernn/forward_weights/._ckpt-133.data-00001-of-00002  
  inflating: ljspeech_wavernn_forward_transformer/wavernn/forward_weights/ckpt-133.data-00000-of-00002  
  inflating: __MACOSX/ljspeech_wavernn_forward_transformer/wavernn/forward_weights/._ckpt-133.data-00000-of-00002  
Archive:  WaveRNN/pretrained/ljspeech.wavernn.mol.800k.zip
  inflating: WaveRNN/pretrained/latest_weights.pyt  

Load pre trained models

from pathlib import Path
WaveRNN_path = 'WaveRNN/'
TTS_path = 'TransformerTTS/'
config_path = Path('ljspeech_wavernn_forward_transformer/wavernn')

# wavernn model
import sys
sys.path.append(WaveRNN_path)

from utils.dsp import hp
from models.fatchord_version import WaveRNN
import torch
import numpy as np
WaveRNN_path = Path(WaveRNN_path)

# Load pretrained model
try:
    hp.configure(WaveRNN_path / 'hparams.py')  # Load hparams from file
except:
    # cant reconfigure, bypass to avoid restart runtime
    pass

if torch.cuda.is_available():
    device = torch.device('cuda')
else:
    device = torch.device('cpu')
wave_model = WaveRNN(rnn_dims=hp.voc_rnn_dims,
                fc_dims=hp.voc_fc_dims,
                bits=hp.bits,
                pad=hp.voc_pad,
                upsample_factors=hp.voc_upsample_factors,
                feat_dims=hp.num_mels,
                compute_dims=hp.voc_compute_dims,
                res_out_dims=hp.voc_res_out_dims,
                res_blocks=hp.voc_res_blocks,
                hop_length=hp.hop_length,
                sample_rate=hp.sample_rate,
                mode=hp.voc_mode).to(device)

wave_model.load(str(WaveRNN_path / 'pretrained/latest_weights.pyt'))

# Ignore some TF warnings
import tensorflow as tf
tf.get_logger().setLevel('ERROR')

# fix deprecated module on librosa
import soundfile as sf
import librosa

class output:
    write_wav = lambda path, data, sr: sf.write(path, data, samplerate=sr, subtype='PCM_24')

librosa.output = output

# Generate sample with pre-trained WaveRNN vocoder
hp_data = hp
def generate(mel, file_name="sample.wav", batch_pred=False, batch_size=5000, hp=hp_data):
    _ = wave_model.generate(mel.clip(0,1)[np.newaxis,:,:], file_name, batch_pred, batch_size, hp.voc_overlap, hp.mu_law)

    # Load wav file
    ipd.display(ipd.Audio(file_name))


# ljspeech_wavernn_forward_model
sys.path.remove('WaveRNN/')
sys.modules.pop('utils')
sys.path.append(TTS_path)

# Load pretrained models
from utils.config_manager import ConfigManager
from utils.audio import Audio

import IPython.display as ipd

config_loader = ConfigManager(str(config_path), model_kind='forward')
audio = Audio(config_loader.config)
model = config_loader.load_model(str(config_path / 'forward_weights/ckpt-133'))

Trainable Parameters: 4.234M
WARNING: could not retrieve git hash. Command '['git', 'describe', '--always']' returned non-zero exit status 128.
WARNING: could not check git hash. Command '['git', 'describe', '--always']' returned non-zero exit status 128.
restored weights from ljspeech_wavernn_forward_transformer/wavernn/forward_weights/ckpt-133 at step 665000

sentence = 'Transformer TTS: A Text-to-Speech Transformer in TensorFlow 2, Audio synthesis with Forward Transformer TTS and WaveRNN Vocoder' #@param {type:"string"}
speed_regulator = 1 #@param {type:"slider", min:0, max:2, step:0.1}
batch_pred = True #@param {type:"boolean"}
batch_size = 1 #@param
out_normal = model.predict(sentence, speed_regulator=speed_regulator)

# Normalize for WaveRNN
mel = (out_normal['mel'].numpy().T+4.)/8.
generate(mel, batch_pred=batch_pred, batch_size=batch_size)

| ████████████████ 445500/445905 | Batch Size: 405 | Gen Rate: 233.7kHz |