Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise Distillation

Audio Demo for 2022 IEEE Spoken Language Technology Workshop (SLT 2022)

Abstract: Several solutions for lightweight TTS have shown promising results. Still, they either rely on a hand-crafted design that reaches non-optimum size or use a neural architecture search but often suffer training costs. We present Nix-TTS, a lightweight TTS achieved via knowledge distillation to a high-quality yet large-sized, non-autoregressive, and end-to-end (vocoder-free) TTS teacher model. Specifically, we offer module-wise distillation, enabling flexible and independent distillation to the encoder and decoder module. The resulting Nix-TTS inherited the advantageous properties of being non-autoregressive and end-to-end from the teacher, yet significantly smaller in size, with only 5.23M parameters or up to 89.34% reduction of the teacher model; it also achieves over 3.04x and 8.36x inference speedup on Intel-i7 CPU and Raspberry Pi 3B respectively and still retains a fair voice naturalness and intelligibility compared to the teacher model. We provide audio samples of Nix-TTS.

Abstract Read with Nix-TTS:

LJ Speech Samples

Input Text

 VITS (Teacher Model) (29.08M Params) Nix-TTS (Proposed Distilled Model) (5.23M Params) BVAE-TTS + HifiGAN (Lightweight TTS Baseline) (15.99M + 13.93M Params) SpeedySpeech + HifiGAN (Lightweight TTS Baseline) (4.52M + 13.93M Params)
"And there may be only nine."
"After a moment's hesitation, she told me that he worked at the Texas School Book Depository near the downtown area of Dallas."
"In the face of impediments confessedly discouraging."
"Such of the Babylonians as witnessed the treachery took refuge in the temple of Jupiter Belus."
"Meanwhile, Chief Curry ordered the police base station to notify Parkland Hospital that the wounded President was en route."
"His letter to the Corporation, under date fourth June."
"And he could disassemble it more rapidly."
"And beyond it was a room called the "wine room," because formerly used for the sale of wine, but"
"From defection to return to Fort Worth."
"The Ishtar gateway leading to the palace was encased with beautiful blue glazed bricks."