Learning the Beauty in Songs: Neural Singing Voice Beautifier
Jinglin Liu, Chengxi Li, Yi Ren, Zhiying Zhu, Zhou Zhao
Zhejiang University
ACL 2022 Main conference
Code project: NeuralSVB
Related project: DiffSinger 
Abstract
We are interested in a novel task, singing voice beautifying (SVB). Given the singing voice of an amateur singer, SVB aims to improve the intonation and vocal tone of the voice, while keeping the content and vocal timbre. Current automatic pitch correction techniques are immature, and most of them are restricted to intonation but ignore the overall aesthetic quality. Hence, we introduce Neural Singing Voice Beautifier (NSVB), the first generative model to solve the SVB task, which adopts a conditional variational autoencoder as the backbone and learns the latent representations of vocal tone. In NSVB, we propose a novel time-warping approach for pitch correction: Shape-Aware Dynamic Time Warping (SADTW), which ameliorates the robustness of existing time-warping approaches, to synchronize the amateur recording with the template pitch curve. Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one. Extensive experiments on both Chinese and English songs demonstrate the effectiveness of our methods in terms of both objective and subjective metrics.
Singing Audio Samples
Note that the singer in the testing data could not be found in the training data.
Chinese
- 世界比你想象中朦胧, shì jiè bǐ nǐ xiǎng xiàng zhōng méng lóng
GT Professional GT Amateur baseline NSVB wav - 不会一场空, bú huì yī cháng kōng
GT Professional GT Amateur baseline NSVB wav - 不是天晴就会有彩虹, bú shì tiān qíng jiù huì yǒu cǎi hóng
GT Professional GT Amateur baseline NSVB wav - 要如何再搜索, yào rú hé zài sōu suǒ
GT Professional GT Amateur baseline NSVB wav - 也许未来遥远在光年之外, yě xǔ wèi lái yáo yuǎn zài guāng nián zhī wài
GT Professional GT Amateur baseline NSVB wav - 足够抵挡天旋地转, zú gòu dǐ dǎng tiān xuán dì zhuàn
GT Professional GT Amateur baseline NSVB wav - 虽然一刹花火, suī rán yī shā huā huǒ
GT Professional GT Amateur baseline NSVB wav - 从来也不觉得错, cóng lái yě bù jué dé cuò
GT Professional GT Amateur baseline NSVB wav English
- I’m not angry anymore
GT Professional GT Amateur baseline NSVB wav - and the band won’t play
GT Professional GT Amateur baseline NSVB wav - it’s love
GT Professional GT Amateur baseline NSVB wav - the days grow long
GT Professional GT Amateur baseline NSVB wav - were beautiful like diamonds in the sky
GT Professional GT Amateur baseline NSVB wav - cause I wanna be better than I was before
GT Professional GT Amateur baseline NSVB wav - I’ll fix you with my love
GT Professional GT Amateur baseline NSVB wav - we will glow in the dark turning dust to gold
GT Professional GT Amateur baseline NSVB wav Special cases on dialect
- 我身骑白马, 走三关 gua sin khia peh be, tsau sam kuan
GT Professional GT Amateur baseline NSVB wav - 我改换素衣呦,回中原 gua kai uann soo i, hue tiong guan
GT Professional GT Amateur baseline NSVB wav