Processing math: 100%

Towards Enabling Meta-Learning from Target Models

Part of Advances in Neural Information Processing Systems 34 (NeurIPS 2021)

Bibtex Paper Reviews And Public Comment » Supplemental

Authors

Su Lu, Han-Jia Ye, Le Gan, De-Chuan Zhan

Abstract

Meta-learning can extract an inductive bias from previous learning experience and assist the training of new tasks. It is often realized through optimizing a meta-model with the evaluation loss of task-specific solvers. Most existing algorithms sample non-overlapping support sets and query sets to train and evaluate the solvers respectively due to simplicity (S/Q protocol). Different from S/Q protocol, we can also evaluate a task-specific solver by comparing it to a target model T, which is the optimal model for this task or a model that behaves well enough on this task (S/T protocol). Although being short of research, S/T protocol has unique advantages such as offering more informative supervision, but it is computationally expensive. This paper looks into this special evaluation method and takes a step towards putting it into practice. We find that with a small ratio of tasks armed with target models, classic meta-learning algorithms can be improved a lot without consuming many resources. We empirically verify the effectiveness of S/T protocol in a typical application of meta-learning, i.e., few-shot learning. In detail, after constructing target models by fine-tuning the pre-trained network on those hard tasks, we match the task-specific solvers and target models via knowledge distillation.