FewNLU: Benchmarking Few-Shot Natural Language Understanding

We open-source FewNLU, an integrated toolkit designed for few-shot natural language understanding. It contains implementation of several state-of-the-art methods, data processing utilities, a standardized few-shot training framework, and most importantly, the proposed evaluation framework. FewNLU also allows customizing new tasks and methods, and performing training and evaluation over them. The goal of FewNLU is to facilitate benchmarking few-shot NLU methods and to facilitate future research in related field. Key features and capabilities of FewNLU include:

• An Evaluation Framework with Recommended Data-split Strategy

  We propose an evaluation framework for few-shot NLU. The newly-formulated framework consists of a repeated procedure -- selecting a hyperparameter, selecting a data split, training and evaluating the model.

• A Collection of State-of-the-Art Methods for Few-Shot NLU

  The FewNLU toolkit contains a number of state-of-the-art few-shot methods. We take a further step to re-evaluate them under the newly-proposed evaluation framework and report the results in Leaderboard.

• Easy-to-Use Customization of Tasks and Methods

  FewNLU allows customizing new tasks or methods with easy-to-use interfaces. Customization enables NLU to easily scale to a diverse range of future works.

Report your results: If you have new results experimented with FewNLU, please send emails to fewnlu@gmail.com or zyanan93@gmail.com or zhouj18@mails.tsinghua.edu .cn. The goal of this leaderboard is to collect research works under the evaluation framework and to measure the true progress of the field. So It is encouraged that you attach a link to the reproducible source codes. Thank you!

Notes:
1. Unlabeled data are used.
2. The ensemble technique is used.
3. Using the data setting with 64 training examples. (Results under the 32-labeled data setting are to be released soon).