Is Dynamic Time Warping of speech signals suitable for articulatory signal comparison using ultrasound tongue images?

—In speech technology, the examination of speaker dependency is vital – that is, whether methods developed for one speaker can be adapted to another speaker or not. In the case of text-to-speech synthesis, well-usable speaker adaptation methods are already available, but they cannot be used directly for articulatory data (movement of the tongue, lips, etc, during speech production). In this research, we investigate the above question and analyze the speaker dependency of the articulatory movement, using audio signal and ultrasound tongue imaging (UTI) recorded in parallel during speech production. For the comparison, we use the well-known Dynamic Time Warping (DTW) procedure of speech technology. DTW of the speech signal has already been successfully applied 1) with UTI, for within-speaker comparisons, 2) with electromagnetic articulography (EMA), for the analysis of inter-speaker differences, 3) with EMA and electrocortocography (ECoG), also for inter-speaker comparisons. However, there has been no previous research yet on the application of DTW on speech signals with ultrasound tongue images for different speakers. In the present research, we examine the applicability of DTW for comparing speakers’ speech and articulatory data on a few Hungarian and English examples, and visually analyze them. In the long term, we plan to use the results for speech-based brain-computer interfaces, so that we can supplement the brain signal with ultrasound-based articulation information.

• Publication year

  2023

• Venue

  1st Workshop on Intelligent Infocommunication Networks, Systems and Services

• Publication date

  Unknown publication date

• Fields of study

  Not labeled

• Identifiers
  DOI 10.3311/wins2023-012
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Towards Ultrasound Tongue Image prediction from EEG during speech production

  2023cited by this paper
• Optimizing the Ultrasound Tongue Image Representation for Residual Network-Based Articulatory-to-Acoustic Mapping

  2022cited by this paper
• Speaker Adaptation Experiments with Limited Data for End-to-End Text-To-Speech Synthesis using Tacotron2

  2022cited by this paper
• Decoding speech from brain activity using linear methods. (Décodage de la parole à partir de l'activité cérébrale à l'aide de méthodes linéaires)

  2022cited by this paper
• Investigations on speaker adaptation using a continuous vocoder within recurrent neural network based text-to-speech synthesis

  2022cited by this paper
• Speaker Adaptation on Articulation and Acoustics for Articulation-to-Speech Synthesis

  2022cited by this paper
• Speaker dependent articulatory-to-acoustic mapping using real-time MRI of the vocal tract

  2020cited by this paper
• Speaker dependent acoustic-to-articulatory inversion using real-time MRI of the vocal tract

  2020cited by this paper
• Tal: A Synchronised Multi-Speaker Corpus of Ultrasound Tongue Imaging, Audio, and Lip Videos

  2020cited by this paper
• ACCURACY ASSESSMENTS OF HAND AND AUTOMATIC MEASUREMENTS OF ULTRASOUND IMAGES OF THE TONGUE

  2019cited by this paper
• DNN-based Acoustic-to-Articulatory Inversion using Ultrasound Tongue Imaging

  2019cited by this paper
• SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks

  2019cited by this paper
• Ultrasound-based Silent Speech Interface Built on a Continuous Vocoder

  2019cited by this paper
• Acquiring and visualizing 3D/4D ultrasound recordings of tongue motion

  2018cited by this paper
• Divide-and-warp temporal alignment of speech signals between speakers: Validation using articulatory data

  2017influential reference
• Statistical Mapping Between Articulatory and Acoustic Data for an Ultrasound-Based Silent Speech Interface

  2011cited by this paper
• Silent speech interfaces

  2010cited by this paper
• The acoustic to articulation mapping: non-linear or non-unique?

  2008cited by this paper
• A guide to analysing tongue motion from ultrasound images

  2005cited by this paper
• Dynamic programming method for temporal registration of three-dimensional tongue surface motion from multiple utterances

  2002cited by this paper
• Electromagnetic articulography: use of alternating magnetic fields for tracking movements of multiple points inside and outside the vocal tract.

  1987cited by this paper
• Analysis of real-time ultrasound images of tongue configuration using a grid-digitizing system

  1983cited by this paper
• Dynamic programming algorithm optimization for spoken word recognition

  1978cited by this paper

• No citing papers are available for this paper.