Multilingual Customized Keyword Spotting Using Similar-Pair Contrastive Learning

The goal of keyword spotting (KWS) is to find words and voice commands. KWS research has received a lot of attention recently. However, the majority of them emphasize predefined KWS. More often than not, customers wish to choose individualized unique keywords. In this work, we present a brand-new Similar-Pair Contrastive Learning (SPCL) training method called MCKWS-SPCL to address Multilingual Customized Keyword Spotting. The three key processes in our MCKWS-SPCL are acoustic feature extraction, embedding model, and similarity computation. By pushing similar-pair samples closer together in projection space, the SPCL training approach is specifically created to obtain similar features with noise existing. Additionally, multiple embedding neural network structures are carefully constructed to examine the trade-off between a limited number of parameters and excellent performance in order to reduce power consumption. For the first time, we complete an all-around multi-dimensional comparison to demonstrate the superior performance of our SPCL over conventional Contrastive Learning in KWS. Results from all trials, including metrics for average accuracy(96.84%), the detection error tradeoff curve, the false reject rate under 5% false alarm rate per hour(1.901%), and visualization using t-Stochastic Neighbor Embedding(same class is closer than CL), conclusively demonstrate our framework's effectiveness and superiority to Contrastive Learning-based approaches. Additionally, the analysis of four distinct language datasets and implementation on the NVIDIA JETSON TX2 hardware platform demonstrates the versatility, affordability, and hardware-friendliness of our new approach.

• Publication year

  2023

• Venue

  IEEE/ACM Transactions on Audio Speech and Language Processing

• Publication date

  Unknown publication date

• Fields of study

  Linguistics, Computer Science

• Identifiers
  DOI 10.1109/TASLP.2023.3284523
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Query-by-Example on-Device Keyword Spotting using Convolutional Recurrent Neural Network and Connectionist Temporal Classification

  2021cited by this paper
• CvT: Introducing Convolutions to Vision Transformers

  2021cited by this paper
• BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

  2021influential reference
• Query-By-Example Keyword Spotting System Using Multi-Head Attention and Soft-triple Loss

  2021cited by this paper
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

  2020cited by this paper
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

  2020cited by this paper
• Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

  2020cited by this paper
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations

  2020cited by this paper
• CERT: Contrastive Self-supervised Learning for Language Understanding

  2020cited by this paper
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

  2020cited by this paper
• Conformer: Convolution-augmented Transformer for Speech Recognition

  2020cited by this paper
• Towards Learning a Universal Non-Semantic Representation of Speech

  2020influential reference
• A Simple Framework for Contrastive Learning of Visual Representations

  2020cited by this paper
• Small-Footprint Open-Vocabulary Keyword Spotting with Quantized LSTM Networks

  2020cited by this paper
• A High-speed Low-cost CNN Inference Accelerator for Depthwise Separable Convolution

  2020cited by this paper
• Contrastive Learning of General-Purpose Audio Representations

  2020cited by this paper
• Rethinking Attention with Performers

  2020cited by this paper
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

  2019cited by this paper
• Temporal Convolution for Real-time Keyword Spotting on Mobile Devices

  2019cited by this paper
• Self-supervised audio representation learning for mobile devices

  2019cited by this paper
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

  2019cited by this paper
• Open-Vocabulary Keyword Spotting with Audio and Text Embeddings

  2019cited by this paper
• Momentum Contrast for Unsupervised Visual Representation Learning

  2019cited by this paper
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

  2018cited by this paper
• DONUT: CTC-based Query-by-Example Keyword Spotting

  2018cited by this paper
• MobileNetV2: Inverted Residuals and Linear Bottlenecks

  2018cited by this paper
• Unsupervised Learning of Semantic Audio Representations

  2017cited by this paper
• End-to-end ASR-free keyword search from speech

  2017cited by this paper
• Squeeze-and-Excitation Networks

  2017cited by this paper
• Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification

  2016cited by this paper
• Unsupervised neural network based feature extraction using weak top-down constraints

  2015cited by this paper
• Librispeech: An ASR corpus based on public domain audio books

  2015cited by this paper
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  2015cited by this paper
• Query-by-example keyword spotting using long short-term memory networks

  2015cited by this paper
• Rigid-Motion Scattering for Texture Classification

  2014influential reference
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Distributed Representations of Words and Phrases and their Compositionality

  2013cited by this paper
• Unsupervised spoken keyword spotting via segmental DTW on Gaussian posteriorgrams

  2009cited by this paper
• Query-by-example Spoken Term Detection For OOV terms

  2009cited by this paper
• Query-by-example spoken term detection using phonetic posteriorgram templates

  2009cited by this paper
• Rapid and accurate spoken term detection

  2007influential reference
• Mel Frequency Cepstral Coefficients for Music Modeling

  2000influential reference
• The DET curve in assessment of detection task performance

  1997cited by this paper

• TCLP-KWS: A Triplet Contrastive Learning Framework for Efficient and Customizable Keyword Spotting

  2026cites this paper
• DeepFlaky: Deep hybrid representation learning for flaky test prediction

  2026cites this paper
• ASDNet: An Efficient Self-Supervised Convolutional Network for Anomalous Sound Detection

  2025cites this paper
• Deep Enhancement Spotting Network for Low-complexity Keyword Spotting in Noisy Environments

  2025cites this paper
• Dynamic convolution models for cross-frontend keyword spotting

  2025cites this paper
• A Small-Footprint Keyword Spotting Modeling with Novel Data Augmentation Methods

  2024cites this paper
• Whisper-based spoken term detection systems for search on speech ALBAYZIN evaluation challenge

  2024cites this paper
• Phoneme-Level Contrastive Learning for User-Defined Keyword Spotting with Flexible Enrollment

  2024cites this paper
• The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language

  2023cites this paper