NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations

Abstract

Paralinguistic vocalizations—including non-verbal sounds like laughter and breathing, as well as lexicalized interjections such as “uhm” and “oh”—are integral to natural spoken communication. Despite their importance in conveying affect, intent, and interactional cues, such cues remain largely overlooked in conventional automatic speech recognition (ASR) and text-to-speech (TTS) systems. We present NVSpeech, an integrated and scalable pipeline that bridges the recognition and synthesis of paralinguistic vocalizations, encompassing dataset construction, ASR modeling, and controllable TTS. (1) We introduce a manually annotated dataset of 48,430 human-spoken utterances with 18 word-level paralinguistic categories. (2) We develop the paralinguistic-aware ASR model, which treats paralinguistic cues as inline decodable tokens (e.g., “You’re so funny [Laughter]”), enabling joint lexical and non-verbal transcription. This model is then used to automatically annotate a large corpus, the first large-scale Chinese dataset of 174,179 utterances (573 hours) with word-level alignment and paralingustic cues. (3) We finetune zero-shot TTS models on both human- and auto-labeled data to enable explicit control over paralinguistic vocalizations, allowing context-aware insertion at arbitrary token positions for human-like speech synthesis. By unifying the recognition and generation of paralinguistic vocalizations, NVSpeech offers the first open, large-scale, word-level annotated pipeline for expressive speech modeling in Mandarin, integrating recognition and synthesis in a scalable and controllable manner.

Motivation

Figure 1. The gap between conventional speech processing systems and paralinguistic-aware modeling. (a) Common speech datasets omit paralinguistic vocalizations, while NVSpeech provides word-level, explicit annotations. (b) Conventional ASR ignores such cues; our paralinguistic-aware ASR jointly transcribes lexical and non-lexical content. (c) Standard TTS generates only text-based speech, while our TTS supports explicit insertion of paralinguistic vocalizations for human-like synthesis.

Pipeline

Figure 2. Overview of our paralinguistic-aware speech recognition and generation pipeline. (1) A word-level human annotated dataset of verbal and non-verbal vocalizations is first constructed. (2) A paralinguistic-aware ASR model is trained to jointly transcribe verbal and non-verbal content. (3) This model is used to automatically annotate large-scale unlabeled speech. (4) The expanded dataset enables training a controllable and expressive TTS system that explicitly renders paralinguistic cues.