How to Train Your MAML to Excel in Few-Shot Classification

Model-agnostic meta-learning (MAML) is arguably one of the most popular meta-learning algorithms nowadays. Nevertheless, its performance on few-shot classification is far behind many recent algorithms dedicated to the problem. In this paper, we point out several key facets of how to train MAML to excel in few-shot classification. First, we find that MAML needs a large number of gradient steps in its inner loop update, which contradicts its common usage in few-shot classification. Second, we find that MAML is sensitive to the class label assignments during meta-testing. Concretely, MAML meta-trains the initialization of an $N$-way classifier. These $N$ ways, during meta-testing, then have"$N!$"different permutations to be paired with a few-shot task of $N$ novel classes. We find that these permutations lead to a huge variance of accuracy, making MAML unstable in few-shot classification. Third, we investigate several approaches to make MAML permutation-invariant, among which meta-training a single vector to initialize all the $N$ weight vectors in the classification head performs the best. On benchmark datasets like MiniImageNet and TieredImageNet, our approach, which we name UNICORN-MAML, performs on a par with or even outperforms many recent few-shot classification algorithms, without sacrificing MAML's simplicity.

• Publication year

  2021

• Venue

  International Conference on Learning Representations

• Publication date

  2021-06-30

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 2106.16245
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Embedding Adaptation is Still Needed for Few-Shot Learning

  2021influential reference
• MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-Shot Learning

  2021cited by this paper
• Large-Scale Meta-Learning with Continual Trajectory Shifting

  2021cited by this paper
• Few-Shot Learning With a Strong Teacher

  2021cited by this paper
• Meta-learning with negative learning rates

  2021cited by this paper
• Negative Margin Matters: Understanding Margin in Few-shot Classification

  2020cited by this paper
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?

  2020cited by this paper
• DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover’s Distance and Structured Classifiers

  2020cited by this paper
• Revisiting Meta-Learning as Supervised Learning

  2020cited by this paper
• Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation

  2020cited by this paper
• MATE: Plugging in Model Awareness to Task Embedding for Meta Learning

  2020cited by this paper
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

  2020cited by this paper
• Meta-Learning Requires Meta-Augmentation

  2020cited by this paper
• Improving Generalization in Meta-learning via Task Augmentation

  2020cited by this paper
• Adaptive Subspaces for Few-Shot Learning

  2020cited by this paper
• Boosting Few-Shot Learning With Adaptive Margin Loss

  2020cited by this paper
• Meta-Learning in Neural Networks: A Survey

  2020cited by this paper
• Adversarial Feature Hallucination Networks for Few-Shot Learning

  2020cited by this paper
• Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes

  2019cited by this paper
• PARN: Position-Aware Relation Networks for Few-Shot Learning

  2019cited by this paper
• A Baseline for Few-Shot Image Classification

  2019cited by this paper
• Generalizing from a Few Examples

  2019cited by this paper
• Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation

  2019cited by this paper
• Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

  2019influential reference
• Learning how to learn

  2019cited by this paper
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML

  2019cited by this paper
• Meta-Learning with Implicit Gradients

  2019cited by this paper
• Feature-Critic Networks for Heterogeneous Domain Generalization

  2019cited by this paper
• Meta-Transfer Learning Through Hard Tasks

  2019cited by this paper
• Associative Alignment for Few-shot Image Classification

  2019cited by this paper
• Meta-Learning without Memorization

  2019influential reference
• Few-Shot Image Recognition With Knowledge Transfer

  2019cited by this paper
• SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning

  2019cited by this paper
• Few-shot learning with adaptively initialized task optimizer: a practical meta-learning approach

  2019cited by this paper
• Meta-Learning With Differentiable Convex Optimization

  2019influential reference
• A Closer Look at Few-shot Classification

  2019cited by this paper
• Hierarchically Structured Meta-learning

  2019cited by this paper
• Meta-Learning for Low-Resource Neural Machine Translation

  2018cited by this paper
• Meta-Learning: A Survey

  2018cited by this paper
• How to train your MAML

  2018cited by this paper
• Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference

  2018cited by this paper
• DropBlock: A regularization method for convolutional networks

  2018cited by this paper
• MetaReg: Towards Domain Generalization using Meta-Regularization

  2018cited by this paper
• The Importance of Sampling in Meta-Reinforcement Learning

  2018cited by this paper
• MetaGAN: An Adversarial Approach to Few-Shot Learning

  2018cited by this paper
• Learning To Learn Around A Common Mean

  2018cited by this paper
• The Importance of Sampling inMeta-Reinforcement Learning

  2018cited by this paper
• Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning

  2018cited by this paper
• Learning to Learn with Gradients

  2018cited by this paper
• Meta-Learning for Batch Mode Active Learning

  2018cited by this paper
• Meta-Learning Update Rules for Unsupervised Representation Learning

  2018cited by this paper
• Tunability: Importance of Hyperparameters of Machine Learning Algorithms

  2018cited by this paper
• Meta-Learning for Semi-Supervised Few-Shot Classification

  2018influential reference
• Few-Shot Learning via Embedding Adaptation With Set-to-Set Functions

  2018influential reference
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

  2018cited by this paper
• Natural Language to Structured Query Generation via Meta-Learning

  2018cited by this paper
• Places: A 10 Million Image Database for Scene Recognition

  2018cited by this paper
• Exploring the Limits of Weakly Supervised Pretraining

  2018cited by this paper
• Learning to Reweight Examples for Robust Deep Learning

  2018cited by this paper
• On First-Order Meta-Learning Algorithms

  2018cited by this paper
• Low-Shot Learning from Imaginary Data

  2018cited by this paper
• Meta-learning with differentiable closed-form solvers

  2018cited by this paper
• Learning to Multitask

  2018cited by this paper
• Learning to Adapt: Meta-Learning for Model-Based Control

  2018cited by this paper
• Been There, Done That: Meta-Learning with Episodic Recall

  2018cited by this paper
• Probabilistic Model-Agnostic Meta-Learning

  2018cited by this paper
• Meta-Learning Transferable Active Learning Policies by Deep Reinforcement Learning

  2018cited by this paper
• Meta-Learning with Latent Embedding Optimization

  2018influential reference
• Transfer Learning via Learning to Transfer

  2018cited by this paper
• Unsupervised Learning via Meta-Learning

  2018cited by this paper
• Multi-attention Network for One Shot Learning

  2017cited by this paper
• Prototypical Networks for Few-shot Learning

  2017influential reference
• Learning to Compose Domain-Specific Transformations for Data Augmentation

  2017cited by this paper
• Meta Learning Shared Hierarchies

  2017cited by this paper
• Learning to Generalize: Meta-Learning for Domain Generalization

  2017cited by this paper
• Supervising Unsupervised Learning

  2017cited by this paper
• Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions

  2017cited by this paper
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments

  2017cited by this paper
• A Simple Neural Attentive Meta-Learner

  2017cited by this paper
• Logical Vision: One-Shot Meta-Interpretive Learning from Real Images

  2017cited by this paper
• Learning to Compare: Relation Network for Few-Shot Learning

  2017cited by this paper
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

  2017influential reference
• One-Shot Imitation Learning

  2017cited by this paper
• Few-Shot Adversarial Domain Adaptation

  2017cited by this paper
• Attentive Recurrent Comparators

  2017cited by this paper
• Learning to Remember Rare Events

  2017cited by this paper
• Learned Optimizers that Scale and Generalize

  2017cited by this paper
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm

  2017cited by this paper
• One-Shot Visual Imitation Learning via Meta-Learning

  2017cited by this paper
• Learning to Model the Tail

  2017cited by this paper
• A Bridge Between Hyperparameter Optimization and Larning-to-learn

  2017cited by this paper
• Neural Optimizer Search with Reinforcement Learning

  2017cited by this paper
• Learning Algorithms for Active Learning

  2017cited by this paper
• Few-Shot Image Recognition by Predicting Parameters from Activations

  2017cited by this paper
• Learning to Multi-Task by Active Sampling

  2017cited by this paper
• A Meta-Learning Perspective on Cold-Start Recommendations for Items

  2017cited by this paper
• Low-Shot Visual Recognition by Shrinking and Hallucinating Features

  2016cited by this paper
• Learning to learn by gradient descent by gradient descent

  2016cited by this paper
• Towards a Neural Statistician

  2016cited by this paper
• Optimization as a Model for Few-Shot Learning

  2016influential reference

• Meta-learning for few-shot open task recognition

  2026cites this paper
• Improving meta-learned loss function accuracy and convergence speed through normalisation for image classification tasks

  2026cites this paper
• Recursive Confidence Training for Pseudo-Labeling Calibration in Semi-Supervised Few-Shot Learning

  2025cites this paper
• Benchmarking for practice: Few-shot time-series crop-type classification on the EuroCropsML dataset

  2025cites this paper
• Eternal-MAML: a meta-learning framework for cross-domain defect recognition

  2025cites this paper
• Attraction Diminishing and Distributing for Few-Shot Class-Incremental Learning

  2025cites this paper
• PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning

  2025cites this paper
• Meta-Reinforcement Learning With Evolving Gradient Regularization

  2025cites this paper
• Self-randomized focuses effectively boost metric-based few-shot classifiers

  2025cites this paper
• Adaptive Task-Aware Refining Network for Few-Shot Fine-Grained Image Classification

  2025cites this paper
• Distributional Metrics for Reliable Few Shot Image Classification

  2025cites this paper
• A query-driven twin network framework with optimization-based meta-learning for few-shot hyperspectral image classification

  2025cites this paper
• Region-aware metric learning for few-shot detection of counterfeit cigarettes from packaging images

  2025cites this paper
• A Course Recommendation Model Rooted in Heterogeneous Information Networks for Meta-Learning

  2024cites this paper
• Decomposed Meta-Learning for Few-Shot Relational Triple Extraction

  2024cites this paper
• Cross-System Software Log-based Anomaly Detection Using Meta-Learning

  2024cites this paper
• From laboratory to field: cross-domain few-shot learning for crop disease identification in the field

  2024cites this paper
• Key-based data augmentation with curriculum learning for few-shot code search

  2024cites this paper
• Few-Shot Cross-System Anomaly Trace Classification for Microservice-based systems

  2024cites this paper
• Few Shot Contrastive Spectral Clustering with Meta Learning and Neighbor Mining

  2024cites this paper
• Learn To Learn More Precisely

  2024cites this paper
• Bi-Directional Ensemble Feature Reconstruction Network for Few-Shot Fine-Grained Classification

  2024cites this paper
• Cross-System Categorization of Abnormal Traces in Microservice-Based Systems via Meta-Learning

  2024cites this paper
• Leveraging Bilateral Correlations for Multi-Label Few-Shot Learning

  2024cites this paper
• Meta-Learning Neural Procedural Biases

  2024influential citation
• Meta Learning Approach Based on Episodic Learning for Few-Shot Image Classification

  2024cites this paper
• “Define Your Terms” : Enhancing Efficient Offensive Speech Classification with Definition

  2024cites this paper
• Dual adversarial network with meta-learning for domain-generalized few-shot text classification

  2023cites this paper
• Meta Attention-Generation Network for Cross-Granularity Few-Shot Learning

  2023cites this paper
• From Instance to Metric Calibration: A Unified Framework for Open-World Few-Shot Learning

  2023cites this paper
• Learning to Learn Task-Adaptive Hyperparameters for Few-Shot Learning

  2023cites this paper
• The general framework for few-shot learning by kernel HyperNetworks

  2023cites this paper
• Elastically-Constrained Meta-Learner for Federated Learning

  2023cites this paper
• FILM: How can Few-Shot Image Classification Benefit from Pre-Trained Language Models?

  2023cites this paper
• Multipretext-Task Prototypes Guided Dynamic Contrastive Learning Network for Few-Shot Remote Sensing Scene Classification

  2023cites this paper
• Photo Pre-Training, But for Sketch

  2023cites this paper
• Bridging the Reality Gap of Reinforcement Learning based Traffic Signal Control using Domain Randomization and Meta Learning

  2023cites this paper
• Permute-MAML: exploring industrial surface defect detection algorithms for few-shot learning

  2023influential citation
• Few-Shot Modulation Recognition with Feature Transformation and Meta-Learning

  2023cites this paper
• Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks

  2022cites this paper
• StratDef: Strategic defense against adversarial attacks in ML-based malware detection

  2022cites this paper
• Bi-directional Feature Reconstruction Network for Fine-Grained Few-Shot Image Classification

  2022cites this paper
• RankDNN: Learning to Rank for Few-shot Learning

  2022cites this paper
• Multimodal Meta-Learning for Cold-Start Sequential Recommendation

  2022cites this paper
• Task-specific method-agnostic metric for few-shot learning

  2022cites this paper
• Hypernetwork approach to Bayesian MAML

  2022cites this paper
• StratDef: a strategic defense against adversarial attacks in malware detection

  2022cites this paper
• Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning

  2022cites this paper
• Channel Importance Matters in Few-Shot Image Classification

  2022cites this paper
• Meta-SysId: A Meta-Learning Approach for Simultaneous Identification and Prediction

  2022cites this paper
• HyperShot: Few-Shot Learning by Kernel HyperNetworks

  2022cites this paper
• Strategies for Meta-Learning with Diverse Tasks

  2022cites this paper
• HyperMAML: Few-Shot Adaptation of Deep Models with Hypernetworks

  2022cites this paper
• A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities

  2022cites this paper
• Evaluating few-shot and contrastive learning methods for code clone detection

  2022cites this paper
• Building a Robust Question Answering System with Meta-Learning

  2022cites this paper
• Property-Aware Relation Networks for Few-Shot Molecular Property Prediction

  2021cites this paper
• Few-Shot Learning With a Strong Teacher

  2021cites this paper
• Improving Few-Shot Learning Through Multi-task Representation Learning Theory

  2020cites this paper
• Revisiting Unsupervised Meta-Learning via the Characteristics of Few-Shot Tasks

  2020cites this paper